VOResource 1.1: Mirrors?

[Patrick Dowler]

I have also been thinking about this a bit as we embark on running
more things on remote systems with this kind of redundancy. A few
things came to mind:

1. if I had mirrors deployed in different locations, the hostnames
would probably all be in the same domain as I'd be unlikely to
register domains in multiple TLDs and I'd probably not get the one I
want across very many TLDs anyway... so while I could name hosts with
something people might grok, machines won't

2. the whole concept is network-closeness which clients can quickly
assess with ping. Although low ping doesn't always equate to high
bandwidth it is something everyone can do

3. in vospace and in one custom-ish CADC service, we give out multiple
transfer endpoints (URLs to files) and in our case storage is
geographically distributed and we try to optimise that. What we do is
give out the URLs in order or preference, so we see where the client
is calling from and sort appropriately. That leaves the server side
free to do geo-location, random, or just primary + spare. We discussed
at a whiteboard an idea that our VOSI-capabilities could assign
ordering to the multiple accessURLs... and then on to an idea that in
the registry we would want to be able to indicate that capabilities
was dynamic/volatile and clients could get a more appropriate/better
result by calling it rather than using the static content in the
registry... just whiteboard-level concepts at this point

With 3, it would still be nice to have mirrorURL: with just a
"dynamic-capabilties" indicator the primary would need to be up. Also
with 3, I would efinitely not embark on this without implementing it
and seeing how it wored out operationally, but the strategy does work
very well for vospace transfers so we also implemented that same
transfer negotiation for archive downloads... so letting the server
indicate order to try for each client is sane... basically CDN. And
no, we can't do it with redirects alone because some APIs (UWS) rely
on post-redirect-get semantics.

Also, we will be implementing much more service mirroring than we have
now in the next 3-9 months, but probably not soon enough to say
anything definitive by the next interop.

my 2c,

Pat

On 14 June 2016 at 03:25, Mark Taylor <M.B.Taylor at bristol.ac.uk> wrote:
> Markus,
>
> On Tue, 14 Jun 2016, Markus Demleitner wrote:
>
>> I took away from our mirrors discussion that
>>
>> (a) people want the concept of mirrors in the registry,
>>
>> (b) modelling each mirror as a separate resource is not appropriate
>> for all the cases (in particular not for VizieR)
>>
>> (c) there is no obvious candidate for metadata beyond the plain URL
>> of the mirror.
>
> Sorry I haven't engaged with this so far, but briefly: addition
> of mirrorURL sounds OK, it's probably an improvement on not
> having them.  However, presented with a bag of apparently
> undifferentiated mirrors, it's hard to imagine how a
> non-non-sophisticated client is going to make an intelligent
> choice between them, unless it already has out-of-registry
> information about the service.
>
> The obvious thing is to present the list to the user and get them
> to decide.  Thinking about the UI for that, some terse human-readable
> information would be useful, e.g.:
>
>    <accessURL use="base">http://example.edu/svc/stars?</accessURL>
>    <mirrorURL use="base" description="Primary backup" >http://example2.edu/svc/stars?</accessURL>
>    <mirrorURL use="base" description="European mirror">http://example.de/svc/stars?</accessURL>
>    <mirrorURL use="base" description="Italian mirror" >http://example.it/svc/stars?</accessURL>
>
> I don't think we know enough about machine use of these things
> yet to codify any machine readable vocabulary for mirror function.
> I am tempted to suggest another attribute that could serve as a
> placeholder for such machine readable information, without specifying
> any content for it now, in case we think of a way to do that in the
> future and don't want to update VOResource, e.g.
>
>    <mirrorURL use="base"
>               description="Primary backup"
>               mirrortype="ivo://ivoa.net/std/VOMirror#backup1"
>               >http://example2.edu/svc/stars?</accessURL>
>    <mirrorURL use="base"
>               description="European mirror"
>               mirrortype="ivo://ivoa.net/std/VOMirror#geo-by-continent"
>               >http://example.de/svc/stars?</accessURL>
>    <mirrorURL use="base"
>               description="Italian mirror"
>               mirrortype="ivo://ivoa.net/std/VOMirror#geo-by-tld"
>               >http://example.it/svc/stars?</accessURL>
>
> but that may be getting a bit adventurous.
>
>> I feel a bit bad for doing this, as I don't see that any software
>> might use this by the time I'd like VOResource 1.1 go to REC, and
>> nothing to go to REC that's not been tested in use.  So, if a client
>> author volunteers, I'd be happy to work with you.
>
> I'm not promising, but: before clients can use it, there would
> need to be some of this content in the registry to test it on.
> Is that likely on the time scale you've got in mind, and if so for
> what types of service?
>
> Mark
>
> --
> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/



-- 
Patrick Dowler
Canadian Astronomy Data Centre
Victoria, BC, Canada