0MQ: The Intelligent Transport Layer

[Roy Williams]

Thanks for this work, John, I think after all the talking, you are the 
first to actually run some tests! But ......

When I fetch the paper mail from the mailbox in front of my house, I 
have no difficulty carrying the quantity of paper. Not so easy is the 
task of separating out unwanted mail, connecting messages with their 
previous context, and deciding how to respond to what is left. On the 
publishing side, I have no problem carrying 100 blank envelopes (or even 
1000!), but it would take me a long time to write 100 letters. Therefore 
let us not buy super trucks that can carry up to 1000 kg of envelopes! 
Because the rate-limiting step is not the bulk of the messages, but 
rather the evaluation of those messages.

We could investigate ways to farm out event annotation and decision into 
remote ('cloud') servers. We could bundle events to get more efficiency 
in the evaluation. We could split streams to multiple servers. We could 
have a sequence of decision filters of increasing sophistication. There 
are a lot of dimensions to this problem.

Roy


On 1/5/12 9:11 AM, John Swinbank wrote:
> Hello,
>
> On 27 Dec 2011, at 21:22, Mike Fitzpatrick wrote:
>
>> The prospect of many thousands of parallel vTCP connections is very
>> appealing either ...
>>
>> Remember too, for some events to be useful (e.g. GRB followup)
>> they cannot be delivered 12 hours later, so the problem (at least a
>> major use-case) has to also solve the question of how to deliver
>> 10K (or is it 50K, or 100K?) events in a matter of *minutes*.  I
>> don't think the bar napkin on which vTCP was designed was big
>> enough to consider that case thoroughly 8-)
>
> Just for amusement, I wrote a simple test to see how this might work.
> My results&  the code that generated them are
> at<https://github.com/jdswinbank/VOEvent-Transport>.
>
> Broadly speaking, maintaining a rate of 2e6 events in 12 hours (~50
> events/second), or even 50K events in 5 minutes (167 events/second),
> was pretty doable. My simple VOEvent generator ran out of steam
> trying to send 100K events in 5 minutes (333 events/second), though.
>
> I certainly don't suggest that's a comprehensive test of the protocol
> – the aim of the exercise was simply my personal education – but
> perhaps the code is of interest to others.
>
> Cheers,
>
> John

-- 
---
Caltech LIGO
roy at caltech.edu
626 395 3670