VOEvent at Heidelberg InterOp

[John Swinbank]

Hi Rob, all,

On 17 May 2013, at 14:35, Rob Seaman <seaman at noao.edu> wrote:

>> I'm currently on the train home from the (ongoing) InterOp in Heidelberg. I hope that Matthew (or possibly his successor as TDIG chair, depending on the outcome of today's Exec meeting) will provide some commentary on the general TDIG relevant discussion at the meeting, but there were a few VOEvent-specific items which I thought might be worthy of further discussion.
> 
> The word was that you were to be TDIG chair.  You might want to contact the Exec :-)

Thanks for the heads-up – I've now heard this reported in a couple of places. More news when I have it… :-)

[…]

>> Of course, by ~2020 (indeed, by 2013) shifting a couple of million messages a night, even if they contain kilobytes of XML overhead, doesn't seem prohibitively expensive – and even if that were an issue within some part of the LSST system, they could presumably define their own internal event representation, and only reserialize it to XML for broadcast to the rest of the world.
> 
> Rather, VOEvent should seek to remain the standard for community-wide event representation, both for internal and external purposes.  Evolution is inevitable whether or not VOEvent remains the go-to format.

While I don't necessarily, I worry that it could lead to complexity through over-generality.

How any given project handles event data is obviously entirely up to them. I imagine there are a bunch of use cases for which XML isn't an appropriate representation – maybe some minimal representation is required to save space, or the data has to be dumped in a relational database, or whatever. So long as VOEvent provides an appropriate representation for those projects to exchange event data with their peers, I don't think it need go out of its way to accommodate every particular project-specific use case.

Of course, if sensible evolution of the VOEvent standard enables it to address as many of those use cases as possible without going overboard on complexity, that's all the better.

[…]

>> Indeed, the above makes me wonder if there should be a standardized upper limit on the size of VOEvent messages being distributed over VTP.
> 
> Rather, perhaps recommendations for best practices.  We should be skeptical of hard limits.
> 
>> Of course, such a limit already exists in that we use a 32 bit integer to specify the size of the packet being transmitted.
> 
> That should be generous enough :-)  I would say a reasonable goal should be that VOEvent software and formats should only require 32-bit hosts.  (All those legacy facilities on mountaintops.
> 
>> But should we mandate that any brokers signing up to the "VOEvent backbone" are obliged to carry messages up to that size? Or up to some other limit?
> 
> We need to give thought to the general issue of service requirements.

Agreed. My (service-oriented!) thought is not so much that we should mandate a maximum size beyond which no VOEvent may grow, but rather we ask backbone-brokers to provide a guarantee that they will carry (or, at least, not drop for size reasons) events up to some plausible limit. We thereby provide some reassurance as to the service level provided by the backbone, but also protect brokers from being overwhelmed by a stream of massive events.

>> Thoughts on any of the above?
> 
> LSST as an operational facility is several years away.  LSST as a project has been ongoing for years, and will kick into high gear very soon.  The pertinent deadline for addressing some of these issues to a solid conceptual stage (and soon after, to a proof of concept) is more like months than years.  They are preparing for design reviews right now.
> 
> Addressing the concerns expressed by Mario would be a good topic for Hotwired3 (http://hotwireduniverse.org).

Definitely.

Cheers,

John