HiPS IVOID issue => proposed solution

[Pierre Fernique]

Dear Registry members,

May I suggest a constructive solution for the IVOID issue that I tried 
to expose in my two last mails, and raised by the current HiPS 
standardization process.

Why not continue to authorize any *ivo://authority.id/A/B/C/etc* at the 
condition that the full id is VO resolvable or, /at least, a left prefix 
of it/.

This identification method is comparable to the DNS resolver mechanism. 
You can query it by a full hostname (host1.network1.gouv.fr), or by 
network (network1.gouv.fr) even by sub-domain (gouv.fr) or domain (fr). 
The number of members is not fixed (generally 3 or 4), and it uses an 
unique separator '.' If the DNS resolver has no information about the 
host, it can provide at least the network information, and so on.

For VO resource, a simple recursive resolution could be used for a 
metadata query. For instance, imagine that a client tries to retrieve 
meta data on "*ivo://CDS.VizieR/I/221/smc*". It queries the VO registry 
with this full id. The registry returns the information concerning the 
longer prefix found in the VO registry. In this case 
"*ivo://CDS.VizieR/I/221*" with a dedicated flag alerting the client 
that this sub-resource is unknown, but its "parents" is described.


Thanks to this more flexible constraint we are sure that any resource 
can continue to be identified with a simple, stable, evolutive and 
canonical (only one way to write the id) method. We avoid to introduce 
the articificial separator "?" for delimiting "fragment" or "query 
syntax" (strange for an identifier and not necessary canonical). And we 
will be able to manage the possible evolutions of the VO registry 
content without having to potentially modify the identifiers (for 
instance if the SMC table is, at the end, described itself in the VO 
registry. And we insure that any existing IVOID has, at least, a 
declared authority id.

How does it sounds ?
Pierre

Le 18/01/2016 20:09, Pierre Fernique a écrit :
> Hi Markus and others,
>
> For helping to understand the problem at the good level, you can 
> consider that the HiPS challenge is comparable to the TAP challenge.
> As TAP, we can imagine that some VO resources will provide directly a 
> dedicated HiPS capability in their corresponding VO registry records. 
> For these resources, there is no problem. But as the VO registry is 
> not really designed for hierarchical resources (catalog/table, 
> survey/band, ...), this approach will be unfortunatelly limited.
>
> Like for TAP table descriptions, it is presently difficult/impossible, 
> to describe all HiPS resources in the VO registry and we use a 2 step 
> mecanisms to discover and manipulate the HiPS descriptions. As your 
> own TAP global schema developement (GLoTS), we deployed a HiPS Global 
> List server harvesting each HiPS servers (called HiPS nodes, 
> comparable to TAP services) and used finally by the HiPS clients.
> So concretly, in the VO registry, we plan to register the HiPS servers 
> (6 presently), rather than all HiPS resources provided by these HiPS 
> servers (about 20 000 HiPS in a few months, probably mirrored in 
> several places).
>
> But even if these HiPS resources are not present individually in the 
> VO registry we need to identify them (in the client interfaces, for 
> script commands, for internal HiPS mirroring management...). And I 
> arrive to my identifier issue : logically we chose IVOA IVORN 
> identification scheme, now called IVOID in your last doc.  But as I 
> said in my previous mail, the last Identifier 2.0 doc does no longer 
> allow the usage of IVOID without a VO real registration (section 2.4). 
> For instance, the way that we identified the "smc" table of the I/221 
> catalog is now invalid: the identificationivo://CDS.VizieR/I/221/smc 
> is no longer a valid IVOID because the "smc" table is not described in 
> the VO registry. Only the catalog I/221 catalog is declared=> 
> http://registry.euro-vo.org/GetResource.jsp?identifier=ivo://CDS.VizieR/I/221.
>
> So we can decide, as you did in your last note, to have a "cavalier 
> approach" (section 3 - 
> http://ivoa.net/documents/Notes/DataCollect/20160108/NOTE-discovercollections-1.0-20160108.pdf) 
> and just ignore the constraint. But I would like to use a more 
> coherent approach. May be, we have just to change the "MUST" word by 
> "SHOULD" in the IVOA PR-identifier 2.0 document - section 2.4 ?
>
> I am also wondering if we should start to discuss how the VO registry 
> could also manage natively complex resources, notably hierarchical 
> resources and mirrored resources. In fact most of the resources that 
> we manipulate are hierarchical and mirrored. Perhaps it is a too big 
> challenge and the usage of specifical additional registry mecanisms 
> such as GLoTS and Global HiPS list is fine.
>
> I hope this HiPS identifier problem related to the VO registry is a 
> little bit more clear.
> Cheers
> Pierre
>
> PS. Also - and that's why I still publish in the Registry mailing list 
> -  it seems that the EURO-VO registry is presently out of date, at 
> least for the example cited below, and since several months, I'm 
> afraid. The up-to-dated record from VizieR is this one: 
> http://cdsweb.u-strasbg.fr/reg-bin/vizier/oai_CS.pl?verb=GetRecord&metadataPrefix=ivo_vor&identifier=ivo%3A%2F%2FCDS.VizieR%2FI%2F221 
> notably by providing two capabilities for CONE SEARCH, one for each 
> individual table (not really easy to use, but there is presently no 
> other good VO registry solution for that). EURO-VO still have the old 
> record version providing only one CONE SEARCH capability declaration 
> (only for the first table I am afraid).
>
>
>
> Le 15/01/2016 13:31, Markus Demleitner a écrit :
>> [Suggestion: Let's continue this on apps; I've set a corresponding
>> reply-to]
>>
>> Dear colleagues,
>>
>> On Fri, Jan 15, 2016 at 09:23:08AM +0100, Pierre Fernique wrote:
>>> Last week, we started the process of the IVOA HiPS standardization, notably
>>> by homogeneizing their identifiers already used by the HiPS providers.
>>> In the existing implementation, HiPS are already identified by an IVORN
>> Although Pierre mentioned it in his mail, just as a reminder IVORN is
>> deprecated (just as URN is deprecated).  Be hip, say IVOID.
>>
>>> following the last IVOA recommendation, normally it should have been easy to
>>> adjust the authority-id (first word of the IVORN) which the corresponding
>>> one in the VO registry or by creating the missing authority-id in the VO
>>> registry and that's all.
>>>
>>>
>>> /HiPS IVORN examples with their associated properties/
>>>
>>> 1. ivo://ov-gso/P/DIRBE/ZSMA8
>>>     <http://cade.irap.omp.eu/documents/Ancillary/4Aladin/DIRBE_ZSMA_8/properties>
>>>     : Band8 of COBE HiPS provided by IRAP
>> [...]
>>
>> The central question is: What should a client get when it resolves
>> the IVOID?  Is it meant to be resolved at all?  If not, what's the
>> function of the identifier?
>>
>>> In a few months, we will have about 20000 HiPS available (tables with
>>> coordinates [15000?] + survey bands [500] + specifical HiPS images [>4000])
>> So, I assume you want to make them discoverable through the Registry?
>> If so, then (something in) the HiPS' IVOIDs will have to resolve anyway.
>>
>>> name for IVORN). Simply because there is presently no HIPS described in the
>>> VO registry, so all these IVOIDs are invalid. But more, *most of the HiPS
>>> IVOIDs will never be valid as they will never be described individually in
>>> the VO registry *(catalog tables ? outreach individual images ? HiPS
>> What is the plan for how people will discover them?
>>
>>> associated to an article ?) Additionnally,  during the last IVOA meeting, we
>>> were thinking to add HiPS in the VO registry as a new capability, and not
>>> necessary as a new separated VO entry (comparable to a cone search or a TAP
>>> access point for a dedicated resource)
>> I would definitely recommend this, in particular since it seems the
>> overwhelming majority of HiPSes are now for resources that already
>> are in the registry (VizieR tables).
>>
>>> *
>>> Three solutions *
>>>
>>> 1.
>>>
>>>     Declare all resources having a HiPS access in the VO registry
>>>     notably all VizieR tables (and not only catalogs), plus all outreach
>>>     images, ... Does it technically and politically possible ? May be by
>> If these things are intended to be discoverable, then yes, that is
>> what needs to be done.  But as already mentioned that doesn't mean
>> they all need new resource records.  Where the underlying data
>> already has an RR, just adding an additional capability will do (see
>> below).
>>
>>>     probably introduce structural incoherences (HiPS table level on one
>>>     side, and catalog description on other side, HiPS per band survey,
>>>     and not HiPS per survey... this point is clearly related to the last
>>>     Markus's discovery resource note)
>> I'm not sure I understand what you're saying here, but what would
>> really help is a clear picture of how clients in the future are
>> expected to discover HiPSes.
>>
>> Gut feeling: if we could construct things such that, say, SDSS has a
>> resource record but not the individual bands, that'd be great.  There
>> are various mechanisms conceivable, e.g., a discovery service for
>> HiPSes, presumably simply SIA. Alternatively, you could have multiple
>> capabilities in a registry record, though I don't really like the
>> idea of regularly distinguishing capabilities by waveband.  Also, all
>> capabilities of a service right now share the same IVOID, and there
>> is no portable way to reference them from the outside yet -- although
>> id/fragment could easily work
>>
>>> 2. Modify the HiPS identification by trying to use the new "fragment"
>>>     syntax introduced in the new PR-Identifiers2.0. For instance,
>>>     replaceivo://CDS.VizieR/I/284/out by
>>>     ivo://CDS.VizieR/I/284?/some_keyword/=out. Unfortunately, this
>> This would make sense if you went for the intermediate discovery
>> service.  This is one of the recommended patterns for SIA and SSA.
>>
>>>     solution will also mean to adapt the internal service nomenclature
>>>     if they already use the previous IVORN "/" separator and not the
>>>     "?something" new syntax. For instance, the notation for tables in
>>>     VizieR and all VizieR clients (Aladin, TOPcat, DS9, Web sites...)
>>>     used for years is I/284/out and not I/284?/some_keyword/=out. Does
>>>     it means that CDS VizieR and associated clients have to support
>>>     both, or remove the usage of previous one ? I suppose no.
>> Again, I suspect I don't quite understand the problem here -- do HiPS
>> clients parse the identifiers?  If so, what do they do with the
>> parsed parts?
>>
>> If they don't parse the identifiers, they shouldn't care if there's a
>> syntax change, should they?
>>
>>> 3. Accept the definition of an IVOID not necessary declared in the VO
>>>     registry.
>> Depending on what clients do with the IVOID, that may be an option,
>> and the Registry WG, perhaps unfortunately, does not control the
>> Death Star and thus has no way to blackmail you into creating
>> registry records.
>>
>> However, again, think of the discoverability of the HiPS, using
>> standard Registry mechanisms.  I suspect you'll find out it's a good
>> idea to play it nice...
>>
>>> 1. Add VizieR tables in the VO Registry for avoiding to break the
>>>     VizieR notation (=> ivo://CDS.VizieR/I/284/out would be valid)
>> But the VizieR tables by and large already are in the Registry -- all
>> you'd need to do is add a capability to them, no?
>>
>>> 2. Use fragment IVOID for really "too small resources" level, notably
>>>     the HiPS outreach images (ivo://eso.org/outreach?product=eso1238a)
>> ...which would mean that an intermediate discovery step would need to
>> be there (or perhaps you require ids on the capabilities and
>> reference those with fragment identifiers, but then a publication of
>> a new image would require an update of the Registry record, which is
>> possible, but a little weird).
>>
>>> 3. Tolerate the existence of IVOIDs not yet defined in the registry
>>>     notably for new HiPS providers (=>ex: ivo://JAXA/xxx), and/or accept
>>>     the creation of VO registry entries by delegation (the ivo:JAXA/xxx
>>>     VO registry would be generated from its properties HiPS definitions)
>> But how would clients find out about their existence?
>>
>>> In the mean time, and for avoiding to jam the HiPS IVOA standardization
>>> process during this discussionI suggest to stop the usage of IVOID in the
>>> "properties" HiPS file (/publisher_did/ field),  and use instead two new
>>> fields splitting the authority-id and the hips-id in two separated fields
>>> (/p//ublisher_id/ and /hips_id/). This solution would allow to continue to
>>> have a usable and homogeneous identification HiPS mechanism without clashing
>>> with the 2.4 constraint, but allowing to generate the associated IVOID
>>> (ivo:// prefix + concatenation with "/" separator) if the associated HiPS is
>>> really defined in the VO registry.
>> Well, my advice is:  Work out how discovery of HiPSes should work.
>> After what I understood of the above, I suspect you might get by
>> defining one standard ID:
>>
>> ivo://ivoa.net/std/hips#hips
>>
>> Such capabilties would have one ParamHTTP interface, the access URL
>> of which directly points to a HiPS.  You'd probably also set the
>> resultType of that interface to your HiPS media type for consistency
>> with case 2:
>>
>> When there are several HiPSes in one resource, you'll have to put in
>> a discovery service.  Let's say it's a SIA service (you could do
>> something simpler, too, perhaps just an enumeration on an HTML page
>> or whatever).
>>
>> Let's say you're using SIA.  You'd then define the SIA service in a
>> normal Registry record.  Clients only looking for HiPSes could figure
>> out which SIA services they should look at by looking at the
>> interface; a HiPS service's interface would give whatever media type
>> you reserve for HiPS as the interface's resultType.  It's a fairly
>> easy registry query pattern, I'd say.
>>
>> In both cases, I don't think a specialized registry extension would
>> be necessary, just use plain vr:Capability (or a SIA capability).
>>
>> In the first case, the HiPS' identifier would be the resource
>> identifier, in the second case, it would be the resource identifier
>> plus "?<local part>".
>>
>>> For the end, I have to apologize for arriving so late in the document
>>> Identifier process (PR state). But I have to say that I had totally missed
>>> the consequence of the 2.4 contraint until this practical case of the HiPS
>>> IVOA standardization raised it.
>> I hope I'm making a convincing point that there is (almost) no
>> problem.
>>
>> Or have I missed something?
>>
>> Cheers,
>>
>>            Markus
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/registry/attachments/20160119/9608bd7b/attachment-0001.html>