0MQ: The Intelligent Transport Layer

[John Swinbank]

Hello,

Completely agree with this: given that the TCP-based protocol is already documented & in use, and I'm not hearing anybody pointing out major deficiencies with it, I don't think there's any serious basis for suggesting that it would be replaced as the "backbone" protocol. I'd certainly be surprised if scalability were an issue: 2e6 events at – say – 10 kB event is only 20 GB, or about 0.5 MB/s over 12 hours. That sort of rate does't sound worrying at all. 

The above said: I can circumstances in which bespoke protocols might be more convenient than vanilla TCP – either for use within some particular project infrastructure, or just because surfing to SkyAlert and downloading events by HTTP in a browser is pretty user friendly! I, for one, would be interested in discussion within this WG on the systems people are developing, at least insofar as they might be relevant to the wider community – and would not infer from such discussion an attempt to undermine or replace existing infrastructure.

I don't really buy the theory that uncertainty over the transport mechanism is a barrier to adoption: returning to my theme of the other day, I suspect that a lack of documented/visible/extant infrastructure is a much bigger issue. I don't really care what protocol I use to collect events, but when I'm left scrabbling around looking at semi-abandoned web pages and vague, non-specific mentions of backbones and XMPP and pubsub and suchlike, without a definitive, up-to-date summary of the options for actually subscribing to a stream of events, it doesn't really inspire confidence. (The exception being the DC3 Dakota stuff, which is great – much credit to Bob for this!)

Per my recent mail, in the new year I'll attempt to partially address the problem by documenting as much infrastructure as I can discover on the web somewhere.

[As an aside, 2e6 events is certainly scientifically interesting: obviously, nobody is going to follow up on all of them "by hand", but, for example, automatically cross-correlating them with events from LOFAR (or SKA, or whatever) could well be an illuminating exercise. At any rate, I'd rather throw many events away than have something interesting go unreported.]

Cheers,

John



On 27 Dec 2011, at 13:07, Frederic V. Hessman wrote:

> Here, here!  We should provide a vanilla solution and let people who want to use something else simply do it - if it turns out to be so much better in the mid-term, then there's more than enough time to switch.  The only thing we could do as a "value-added" feature is maintain a loose collection of solutions, particularly those whose fancy features are only open-ended (i.e. can immediately connect to, e.g., vTCP).   Things like 0MQ and ICE are wonderful in a closed solution space, but it's nicer to have things easy on your end even if the other end isn't.
> 
> Rick
> 
> On 27 Dec 2011, at 07:17, Bob wrote:
> 
>> May write more soon (or maybe I'm done with this), but:
>> 
>> (1) Is "transport neutrality" like "gauge neutrality" for a 
>> railroad? What is the benefit? The meat is in the messages not the 
>> plumbing.
>> 
>> (2) Is "undefined and ever expanding incompatible transport" 
>> standing in the way of adoption? I suspect the uncertainty is a 
>> huge barrier to adoption.
>> 
>> (3) Ah, the LSST spectre... I say again "Who is going to USE (or 
>> even LOOK AT) 2 million alerts per night? What's the point? It 
>> seems like the stick that the sandboxers use to beat the realists 
>> over the head with. That head again ;-) And a sentence ending in a 
>> preposition. Sorry.
>> 
>> (4) Is transport research and development for VOEvent a vehicle 
>> for garnering additional grant (and therefore 
>> grocery/housepayment) money? If so, stability and adoption are 
>> hopeless.
>> 
>> -- Bob
>> 
>> On Mon Dec 26 23:02:29 CST 2011, Rob Seaman <seaman at noao.edu> 
>> wrote:
>> 
>>> Hi Bob,
>>> 
>>> I can't comment on either your head or wall :-) but nothing is 
>>> wrong with either the transport note or Dakota.  And however 
>>> simple zeromg is, it certainly can't be simpler than vanilla TCP. 
>>> My suspicion is that vTCP and thus Dakota will be supported 
>>> indefinitely, but that more boutique protocols like zeromg, 
>>> elvin, Java messaging - and perhaps jabber - will come and go.  
>>> As has been said before, this implies the creation and 
>>> maintenance of bridges in the deployed VOEventNet for any future 
>>> network.  VOEvent itself is transport neutral by design.
>>> 
>>> Will vTCP scale to the current LSST baseline of two million 
>>> alerts per night?  Will other technologies?  In the near term, 
>>> projects now nearing completion or in commissioning will deliver 
>>> many thousands of events nightly in the next few years.  These 
>>> several surveys are each likely to produce numerous streams of 
>>> events, filtered or correlated in scientifically interesting 
>>> ways.  Registries - either explicitly through the VO or 
>>> implicitly through the diverse stakeholders for each survey - 
>>> will be used to discover streams and associated metadata and 
>>> per-project schemata (again, either explicit or implicit).
>>> 
>>> All of these issues and the related science requirements from the 
>>> diverse event-producers and consumers will have a hand in 
>>> evolving the deployed infrastructure needed to scale to the 
>>> future event rates and large numbers of future subscribers of 
>>> different sorts.  There are a lot of different options for how 
>>> that infrastructure will be funded and operated.  It is quite 
>>> likely that different parties will provide access to different 
>>> classes of subscribers as well as classes of event authors (both 
>>> humans and autonomous agents).  For instance, I would think it an 
>>> almost foregone conclusion that your customers would most 
>>> robustly and efficiently subscribe through your software, 
>>> whatever the network topology of future event brokers.  Whether 
>>> one or more of those brokers includes bridging capabilities from 
>>> some yet-to-be-specified transport protocol to vTCP seems (at 
>>> this end) to be an implementation detail.
>>> 
>>> Or alternately, perhaps the native protocol of the event backbone 
>>> will remain vTCP and no bridging will be needed by Dakota 
>>> clients, but other subscribers will choose some trendy boutique 
>>> notification service using either a push or pull paradigm.  
>>> Whichever it is, I would be flabbergasted if the astronomical 
>>> community in general or IVOA in particular sought to forbid 
>>> boundary components of the ramifying VOEventNet from including 
>>> bridging technology to zeromg or any other transport protocol.
>>> 
>>> Rob
>