VOEvent at Heidelberg InterOp

[John Swinbank]

Dear all,

I'm currently on the train home from the (ongoing) InterOp in Heidelberg. I hope that Matthew (or possibly his successor as TDIG chair, depending on the outcome of today's Exec meeting) will provide some commentary on the general TDIG relevant discussion at the meeting, but there were a few VOEvent-specific items which I thought might be worthy of further discussion. I apologize in advance for the somewhat rambling nature of the following.

First (and with apologies to Bob, in particular, that this has been on my back burner for so long), the latest draft of the VOEvent Transport Protocol document is available from <http://tinyurl.com/20130513vtp>. It's my hope that we can agree upon a version of this that we're happy to take forward to the standardization process soon. Your comments are very much welcome.

Secondly, Mario Juric of LSST raised a couple of interesting points about future VOEvent development in today's time domain discussion session. I guess I should emphasize that VOEvent 2.0 seems "good enough" for the time being at least: I see no harm in musing over future extensions, and have no doubt they'll be necessary some day, but evolving too rapidly will simply hurt adoption rates.

With that out of the way: firstly, Mario worried about the relatively heavyweight nature of the XML serialization, pointing to a particular example from the IVOA website where a several-hundred-character VOEvent provides about 40 characters of actually useful information. In the LSST world of 2e6 events/night, that's obviously a substantial overhead. This naturally makes me recall previous discussions of alternative VOEvent serializations (JSON, anybody…?).

Of course, by ~2020 (indeed, by 2013) shifting a couple of million messages a night, even if they contain kilobytes of XML overhead, doesn't seem prohibitively expensive – and even if that were an issue within some part of the LSST system, they could presumably define their own internal event representation, and only reserialize it to XML for broadcast to the rest of the world.

[On the other hand, defining an alternative (easily canonicalizable...) VOEvent representations might have other advantages regarding event signatures. But I think that's out of scope for the time being.]

Another issue Mario raised was that of embedding richer content, such as thumbnail images, into VOEvent packets. The argument here is that the existing reference mechanism isn't necessarily scalable to the volumes LSST needs: they don't want to have to field 2e6 * n_subscribers * n_references call-back requests every night, and, further, worry about the additional latency for event consumers (who, rather than immediately making a decision on whether to perform follow-up, now have to request additional information and wait for that to be delivered before they can proceed).

While I see the argument, my gut is rather sceptical of the above: it seems to me that, rather than event authors fielding millions of callbacks, they can easily and (relatively…) cheaply make their content available on distribution networks to which the number of requests is fairly trivial (let S3 take the strain!), and rather focus on driving down latency by keeping the VOEvent packets themselves small and distribution networks fast.

Indeed, the above makes me wonder if there should be a standardized upper limit on the size of VOEvent messages being distributed over VTP. Of course, such a limit already exists in that we use a 32 bit integer to specify the size of the packet being transmitted. But should we mandate that any brokers signing up to the "VOEvent backbone" are obliged to carry messages up to that size? Or up to some other limit?

Thoughts on any of the above?

John