0MQ: The Intelligent Transport Layer

[Rob Seaman]

On Dec 27, 2011, at 1:22 PM, Mike Fitzpatrick wrote:

> Remember too, for some events to be useful (e.g. GRB followup) they cannot be delivered 12 hours later, so the problem (at least a major use-case) has to also solve the question of how to deliver 10K (or is it 50K, or 100K?) events in a matter of *minutes*.

Fractions of a second, actually.  The challenge for VOEvent is to provide a single system suitable for diverse use cases.  Not all phenomena or their contingent science requires such short latency, but the alternative is to have separate networks for separate purposes.  This was historically the case and we've seen how far the community infrastructure can splinter as a result.  It will likely always be the case that purpose-specific publishers will prosper, but that doesn't mean they should duplicate efforts toward formats, transport, tools, brokers, repositories, etc.

The LSST baseline is something like 1000 visits (paired exposures) per night, each averaging 2000 events.  (By adjusting thresholds that can grow by an order of magnitude, but this is the baseline as I last heard it.)  The survey's latency requirement is to publish each visit's events within 60s of the shutter closing.  The corresponding requirement on the transport will be shorter than this for the simple reason that network services will have to keep up.  The science requirements will vary between solar system and stellar and extragalactic projects.

> I don't think the bar napkin on which vTCP was designed was big enough to consider that case thoroughly 8-)

vTCP was elaborated from GCN at a session at VOEvent II in 2005.  It was sketched out on the whiteboard in NOAO's La Quinta conference room and Alasdair took a picture on his cell phone.  The napkin in question rather contained the outline of SimpleTimeSeries and was sketched out at the Arizona Inn during the banquet for Hotwired I (aka VOEvent III) in 2007.

Both should be regarded as successful design efforts that have seeded subsequent work.  That both whiteboard and napkin are finite in size provided encouragement to keep the designs pragmatic in scope.  I would hope that the IVOA time series efforts remain so.

From the joie de vivre with which the transport discussion has been revisited recently and in the past, it seems ripe for a renewed effort.  Part of that effort should be directed toward enhancing the robustness and usability of current protocols, libraries, servers and clients - and part toward characterizing the efforts needed for future work.

One realization of the latter has been planned as a US VAO activity (http://arxiv.org/pdf/1111.2356 and related efforts).  Given funding challenges this is likely to remain pending for the near term, but other efforts can continue depending on their own project priorities and resources.

Rob