0MQ: The Intelligent Transport Layer

[Frederic V. Hessman]

Here, here!  We should provide a vanilla solution and let people who want to use something else simply do it - if it turns out to be so much better in the mid-term, then there's more than enough time to switch.  The only thing we could do as a "value-added" feature is maintain a loose collection of solutions, particularly those whose fancy features are only open-ended (i.e. can immediately connect to, e.g., vTCP).   Things like 0MQ and ICE are wonderful in a closed solution space, but it's nicer to have things easy on your end even if the other end isn't.

Rick

On 27 Dec 2011, at 07:17, Bob wrote:

> May write more soon (or maybe I'm done with this), but:
> 
> (1) Is "transport neutrality" like "gauge neutrality" for a 
> railroad? What is the benefit? The meat is in the messages not the 
> plumbing.
> 
> (2) Is "undefined and ever expanding incompatible transport" 
> standing in the way of adoption? I suspect the uncertainty is a 
> huge barrier to adoption.
> 
> (3) Ah, the LSST spectre... I say again "Who is going to USE (or 
> even LOOK AT) 2 million alerts per night? What's the point? It 
> seems like the stick that the sandboxers use to beat the realists 
> over the head with. That head again ;-) And a sentence ending in a 
> preposition. Sorry.
> 
> (4) Is transport research and development for VOEvent a vehicle 
> for garnering additional grant (and therefore 
> grocery/housepayment) money? If so, stability and adoption are 
> hopeless.
> 
>  -- Bob
> 
> On Mon Dec 26 23:02:29 CST 2011, Rob Seaman <seaman at noao.edu> 
> wrote:
> 
>> Hi Bob,
>> 
>> I can't comment on either your head or wall :-) but nothing is 
>> wrong with either the transport note or Dakota.  And however 
>> simple zeromg is, it certainly can't be simpler than vanilla TCP. 
>> My suspicion is that vTCP and thus Dakota will be supported 
>> indefinitely, but that more boutique protocols like zeromg, 
>> elvin, Java messaging - and perhaps jabber - will come and go.  
>> As has been said before, this implies the creation and 
>> maintenance of bridges in the deployed VOEventNet for any future 
>> network.  VOEvent itself is transport neutral by design.
>> 
>> Will vTCP scale to the current LSST baseline of two million 
>> alerts per night?  Will other technologies?  In the near term, 
>> projects now nearing completion or in commissioning will deliver 
>> many thousands of events nightly in the next few years.  These 
>> several surveys are each likely to produce numerous streams of 
>> events, filtered or correlated in scientifically interesting 
>> ways.  Registries - either explicitly through the VO or 
>> implicitly through the diverse stakeholders for each survey - 
>> will be used to discover streams and associated metadata and 
>> per-project schemata (again, either explicit or implicit).
>> 
>> All of these issues and the related science requirements from the 
>> diverse event-producers and consumers will have a hand in 
>> evolving the deployed infrastructure needed to scale to the 
>> future event rates and large numbers of future subscribers of 
>> different sorts.  There are a lot of different options for how 
>> that infrastructure will be funded and operated.  It is quite 
>> likely that different parties will provide access to different 
>> classes of subscribers as well as classes of event authors (both 
>> humans and autonomous agents).  For instance, I would think it an 
>> almost foregone conclusion that your customers would most 
>> robustly and efficiently subscribe through your software, 
>> whatever the network topology of future event brokers.  Whether 
>> one or more of those brokers includes bridging capabilities from 
>> some yet-to-be-specified transport protocol to vTCP seems (at 
>> this end) to be an implementation detail.
>> 
>> Or alternately, perhaps the native protocol of the event backbone 
>> will remain vTCP and no bridging will be needed by Dakota 
>> clients, but other subscribers will choose some trendy boutique 
>> notification service using either a push or pull paradigm.  
>> Whichever it is, I would be flabbergasted if the astronomical 
>> community in general or IVOA in particular sought to forbid 
>> boundary components of the ramifying VOEventNet from including 
>> bridging technology to zeromg or any other transport protocol.
>> 
>> Rob