0MQ: The Intelligent Transport Layer

[John Swinbank]

Hi Roy,

I think there are (at least) two issues here, both of which are relevant.

Of course you're correct that given some millions of events (or letters), extracting the relevant information is challenging – indeed, I'm sure that's far more challenging than simply transporting the envelopes about. It's surely something that this WG will have to devote some considerable thought to.

However, I don't think we can avoid the issue of bulk transport. I assume that the LSST folks (for example) have done their homework, and really will produce a couple of million events which are worthy of distribution per night. They will then face the problem of shifting them down a high-latency, low-bandwidth line. They still need a "super truck" capable of that, even if it's only to carry letters to a sorting office.

Cheers,

John

On 5 Jan 2012, at 18:28, Roy Williams wrote:

> Thanks for this work, John, I think after all the talking, you are the first to actually run some tests! But ......
> 
> When I fetch the paper mail from the mailbox in front of my house, I have no difficulty carrying the quantity of paper. Not so easy is the task of separating out unwanted mail, connecting messages with their previous context, and deciding how to respond to what is left. On the publishing side, I have no problem carrying 100 blank envelopes (or even 1000!), but it would take me a long time to write 100 letters. Therefore let us not buy super trucks that can carry up to 1000 kg of envelopes! Because the rate-limiting step is not the bulk of the messages, but rather the evaluation of those messages.
> 
> We could investigate ways to farm out event annotation and decision into remote ('cloud') servers. We could bundle events to get more efficiency in the evaluation. We could split streams to multiple servers. We could have a sequence of decision filters of increasing sophistication. There are a lot of dimensions to this problem.
> 
> Roy
> 
> 
> On 1/5/12 9:11 AM, John Swinbank wrote:
>> Hello,
>> 
>> On 27 Dec 2011, at 21:22, Mike Fitzpatrick wrote:
>> 
>>> The prospect of many thousands of parallel vTCP connections is very
>>> appealing either ...
>>> 
>>> Remember too, for some events to be useful (e.g. GRB followup)
>>> they cannot be delivered 12 hours later, so the problem (at least a
>>> major use-case) has to also solve the question of how to deliver
>>> 10K (or is it 50K, or 100K?) events in a matter of *minutes*.  I
>>> don't think the bar napkin on which vTCP was designed was big
>>> enough to consider that case thoroughly 8-)
>> 
>> Just for amusement, I wrote a simple test to see how this might work.
>> My results&  the code that generated them are
>> at<https://github.com/jdswinbank/VOEvent-Transport>.
>> 
>> Broadly speaking, maintaining a rate of 2e6 events in 12 hours (~50
>> events/second), or even 50K events in 5 minutes (167 events/second),
>> was pretty doable. My simple VOEvent generator ran out of steam
>> trying to send 100K events in 5 minutes (333 events/second), though.
>> 
>> I certainly don't suggest that's a comprehensive test of the protocol
>> – the aim of the exercise was simply my personal education – but
>> perhaps the code is of interest to others.
>> 
>> Cheers,
>> 
>> John
> 
> -- 
> ---
> Caltech LIGO
> roy at caltech.edu
> 626 395 3670