XMPP as VOEvent transport

[Rob Seaman]

> On Sep 24, 2015, at 5:24 AM, John Swinbank <swinbank at transientskp.org> wrote:
> 
> the intent here is ... to standardize a protocol that has been informally defined since ~2008, and is currently actively in use by several projects.

The initial implementations of VTP were in 2005 between the first VOEvent workshop and the second, at which time iamalive and other features were added. The original concept of operations of VTP was roughly “GCN transport in reverse”. Others who bathed in that primordial ooze may comment.

> On Sep 24, 2015, at 5:35 AM, John Swinbank <swinbank at transientskp.org> wrote:
> 
> A new thread on this, since I don’t want the discussion of VTP to turn into a more general-purpose discussion of how one could or should transport VOEvents. However, I was interested to hear that Svom is pressing forwards with XMPP as a distribution mechanism: I’m aware of that being discussed several years ago, but hadn’t seen any news in some time (I’m out of the loop; sorry).
> 
> I’d be curious to know what the latest status of this is. Is there code available? A “getting started” document describing how it works? Any thoughts on scaling to large numbers of events, or large event packet sizes, or bulk distribution? Does it help address issues regarding event authenticity?
> 
> VTP is useful as a minimal baseline, but I have reservations about it, particularly in terms of scalability, and I’d love an update on other work in this area.

The VOEventNet project piloted transport via XMPP / Jabber; I’m not sure how much of this persisted into SkyAlert. Other options were also investigated and sometimes prototyped (including Java messaging at NOAO, twitter implementations at various places, etc). The notion was (and should remain) to operate bridges between the different protocols. This is implicit in any web-based client-server architecture like SkyAlert where alerts arrive in some mix of transport technologies and are passed to end-users via more general-purpose channels.

We used a quirky notification protocol (from some company in Australia) for the original VOEvent demo at the NVO summer school in early 2005. Heck, the Lego robots use bluetooth messaging. One strong motivation for the VTP reference document is to emphasize how straightforward the requirements are and to exhort the troops to keep the communications channels simple and robust. Something like Jabber has lots of extra features that celestial transient event alerting may not need. This isn’t necessarily a problem, but restraint must be shown to avoid introducing excessive bells and whistles.

On the other hand, very early on – perhaps at the first VOEvent workshop in Pasadena, but certainly by the Aspen NVO summer school a few months later – we were discussing using OAI protocols (associated with Dublin Core) for a coordinating layer above simple transport. No matter how scalable the transport, something needs to act to organize the network of brokers into a coherent whole. The idea was that brokers would exchange VOEvent packets on one channel and could talk amongst themselves using OAI on a separate command channel. (Actual implementations could be multiplexed.) More recent Hotwired and SPIE discussions of dynamic coalition management (section 3 of http://arxiv.org/pdf/1407.7552.pdf) rely on a similar notion: that’s the “dynamic” part.

For simple single-institution projects this can default to manually managed host tables (of whatever format) – that is, the network can be plugged together like tinker toys. Or for a single centralized clearing house like GCN or MPC, hundreds of authors/subscribers can individually be granted firewall privileges, etc., and can keep separate sockets open with only a few tenths of an FTE operator overhead (except when the servers fail and the lack of autonomous provisioning becomes obvious ;-)

However, as the number of surveys publishing alerts increases, and as the consumers of these alerts (follow-up facilities, archives, etc) ramifies, a tipping point will be reached beyond which ad hoc plumbing no longer suffices. An analogy (and perhaps techical inspiration) is the internet itself: a network of networks mediated by sometimes arcane heuristics instantiated in self-organizing switches. Those switches often also serve the purpose of bridging different transport protocols.

The other half of this is that no matter what architectural paradigm is used to build the alert network, there will also come another tipping point where resource discovery is no longer scalable via the word of mouth that we have used so far. Requirements include a scalable and flexible registry. For instance, see page 3 of:

	http://www.ivoa.net/documents/VOEvent/20110711/REC-VOEvent-2.0.pdf

One way to look at the transport discussion is that we are filling in the blank areas at the top and bottom of this diagram with additional standards, whether native to IVOA or adopted from outside.

It is hard to say how far off the tipping point is, but with each ad hoc email cc-ed willy-nilly to painstakingly negotiate target-of-opportunity telescope access, observing scripts, finder coordinates, pipeline recipes, proprietary data rights, etc, it becomes more evident that it ain’t that far off.

Rob