HiPS IVOID issue

[Pierre Fernique]

Hi Markus and others,

For helping to understand the problem at the good level, you can 
consider that the HiPS challenge is comparable to the TAP challenge.
As TAP, we can imagine that some VO resources will provide directly a 
dedicated HiPS capability in their corresponding VO registry records. 
For these resources, there is no problem. But as the VO registry is not 
really designed for hierarchical resources (catalog/table, survey/band, 
...), this approach will be unfortunatelly limited.

Like for TAP table descriptions, it is presently difficult/impossible, 
to describe all HiPS resources in the VO registry and we use a 2 step 
mecanisms to discover and manipulate the HiPS descriptions. As your own 
TAP global schema developement (GLoTS), we deployed a HiPS Global List 
server harvesting each HiPS servers (called HiPS nodes, comparable to 
TAP services) and used finally by the HiPS clients.
So concretly, in the VO registry, we plan to register the HiPS servers 
(6 presently), rather than all HiPS resources provided by these HiPS 
servers (about 20 000 HiPS in a few months, probably mirrored in several 
places).

But even if these HiPS resources are not present individually in the VO 
registry we need to identify them (in the client interfaces, for script 
commands, for internal HiPS mirroring management...). And I arrive to my 
identifier issue : logically we chose IVOA IVORN identification scheme, 
now called IVOID in your last doc.  But as I said in my previous mail, 
the last Identifier 2.0 doc does no longer allow the usage of IVOID 
without a VO real registration (section 2.4). For instance, the way that 
we identified the "smc" table of the I/221 catalog is now invalid: the 
identificationivo://CDS.VizieR/I/221/smc is no longer a valid IVOID 
because the "smc" table is not described in the VO registry. Only the 
catalog I/221 catalog is declared=> 
http://registry.euro-vo.org/GetResource.jsp?identifier=ivo://CDS.VizieR/I/221.

So we can decide, as you did in your last note, to have a "cavalier 
approach" (section 3 - 
http://ivoa.net/documents/Notes/DataCollect/20160108/NOTE-discovercollections-1.0-20160108.pdf) 
and just ignore the constraint. But I would like to use a more coherent 
approach. May be, we have just to change the "MUST" word by "SHOULD" in 
the IVOA PR-identifier 2.0 document - section 2.4 ?

I am also wondering if we should start to discuss how the VO registry 
could also manage natively complex resources, notably hierarchical 
resources and mirrored resources. In fact most of the resources that we 
manipulate are hierarchical and mirrored. Perhaps it is a too big 
challenge and the usage of specifical additional registry mecanisms such 
as GLoTS and Global HiPS list is fine.

I hope this HiPS identifier problem related to the VO registry  is a 
little bit more clear.
Cheers
Pierre

PS. Also - and that's why I still publish in the Registry mailing list 
-  it seems that the EURO-VO registry is presently out of date, at least 
for the example cited below, and since several months, I'm afraid. The 
up-to-dated record from VizieR is this one: 
http://cdsweb.u-strasbg.fr/reg-bin/vizier/oai_CS.pl?verb=GetRecord&metadataPrefix=ivo_vor&identifier=ivo%3A%2F%2FCDS.VizieR%2FI%2F221 
notably by providing two capabilities for CONE SEARCH, one for each 
individual table (not really easy to use, but there is presently no 
other good VO registry solution for that). EURO-VO still have the old 
record version providing only one CONE SEARCH capability declaration 
(only for the first table I am afraid).



Le 15/01/2016 13:31, Markus Demleitner a écrit :
> [Suggestion: Let's continue this on apps; I've set a corresponding
> reply-to]
>
> Dear colleagues,
>
> On Fri, Jan 15, 2016 at 09:23:08AM +0100, Pierre Fernique wrote:
>> Last week, we started the process of the IVOA HiPS standardization, notably
>> by homogeneizing their identifiers already used by the HiPS providers.
>> In the existing implementation, HiPS are already identified by an IVORN
> Although Pierre mentioned it in his mail, just as a reminder IVORN is
> deprecated (just as URN is deprecated).  Be hip, say IVOID.
>
>> following the last IVOA recommendation, normally it should have been easy to
>> adjust the authority-id (first word of the IVORN) which the corresponding
>> one in the VO registry or by creating the missing authority-id in the VO
>> registry and that's all.
>>
>>
>> /HiPS IVORN examples with their associated properties/
>>
>> 1. ivo://ov-gso/P/DIRBE/ZSMA8
>>     <http://cade.irap.omp.eu/documents/Ancillary/4Aladin/DIRBE_ZSMA_8/properties>
>>     : Band8 of COBE HiPS provided by IRAP
> [...]
>
> The central question is: What should a client get when it resolves
> the IVOID?  Is it meant to be resolved at all?  If not, what's the
> function of the identifier?
>
>> In a few months, we will have about 20000 HiPS available (tables with
>> coordinates [15000?] + survey bands [500] + specifical HiPS images [>4000])
> So, I assume you want to make them discoverable through the Registry?
> If so, then (something in) the HiPS' IVOIDs will have to resolve anyway.
>
>> name for IVORN). Simply because there is presently no HIPS described in the
>> VO registry, so all these IVOIDs are invalid. But more, *most of the HiPS
>> IVOIDs will never be valid as they will never be described individually in
>> the VO registry *(catalog tables ? outreach individual images ? HiPS
> What is the plan for how people will discover them?
>
>> associated to an article ?) Additionnally,  during the last IVOA meeting, we
>> were thinking to add HiPS in the VO registry as a new capability, and not
>> necessary as a new separated VO entry (comparable to a cone search or a TAP
>> access point for a dedicated resource)
> I would definitely recommend this, in particular since it seems the
> overwhelming majority of HiPSes are now for resources that already
> are in the registry (VizieR tables).
>
>> *
>> Three solutions *
>>
>> 1.
>>
>>     Declare all resources having a HiPS access in the VO registry
>>     notably all VizieR tables (and not only catalogs), plus all outreach
>>     images, ... Does it technically and politically possible ? May be by
> If these things are intended to be discoverable, then yes, that is
> what needs to be done.  But as already mentioned that doesn't mean
> they all need new resource records.  Where the underlying data
> already has an RR, just adding an additional capability will do (see
> below).
>
>>     probably introduce structural incoherences (HiPS table level on one
>>     side, and catalog description on other side, HiPS per band survey,
>>     and not HiPS per survey... this point is clearly related to the last
>>     Markus's discovery resource note)
> I'm not sure I understand what you're saying here, but what would
> really help is a clear picture of how clients in the future are
> expected to discover HiPSes.
>
> Gut feeling: if we could construct things such that, say, SDSS has a
> resource record but not the individual bands, that'd be great.  There
> are various mechanisms conceivable, e.g., a discovery service for
> HiPSes, presumably simply SIA. Alternatively, you could have multiple
> capabilities in a registry record, though I don't really like the
> idea of regularly distinguishing capabilities by waveband.  Also, all
> capabilities of a service right now share the same IVOID, and there
> is no portable way to reference them from the outside yet -- although
> id/fragment could easily work
>
>> 2. Modify the HiPS identification by trying to use the new "fragment"
>>     syntax introduced in the new PR-Identifiers2.0. For instance,
>>     replaceivo://CDS.VizieR/I/284/out by
>>     ivo://CDS.VizieR/I/284?/some_keyword/=out. Unfortunately, this
> This would make sense if you went for the intermediate discovery
> service.  This is one of the recommended patterns for SIA and SSA.
>
>>     solution will also mean to adapt the internal service nomenclature
>>     if they already use the previous IVORN "/" separator and not the
>>     "?something" new syntax. For instance, the notation for tables in
>>     VizieR and all VizieR clients (Aladin, TOPcat, DS9, Web sites...)
>>     used for years is I/284/out and not I/284?/some_keyword/=out. Does
>>     it means that CDS VizieR and associated clients have to support
>>     both, or remove the usage of previous one ? I suppose no.
> Again, I suspect I don't quite understand the problem here -- do HiPS
> clients parse the identifiers?  If so, what do they do with the
> parsed parts?
>
> If they don't parse the identifiers, they shouldn't care if there's a
> syntax change, should they?
>
>> 3. Accept the definition of an IVOID not necessary declared in the VO
>>     registry.
> Depending on what clients do with the IVOID, that may be an option,
> and the Registry WG, perhaps unfortunately, does not control the
> Death Star and thus has no way to blackmail you into creating
> registry records.
>
> However, again, think of the discoverability of the HiPS, using
> standard Registry mechanisms.  I suspect you'll find out it's a good
> idea to play it nice...
>
>> 1. Add VizieR tables in the VO Registry for avoiding to break the
>>     VizieR notation (=> ivo://CDS.VizieR/I/284/out would be valid)
> But the VizieR tables by and large already are in the Registry -- all
> you'd need to do is add a capability to them, no?
>
>> 2. Use fragment IVOID for really "too small resources" level, notably
>>     the HiPS outreach images (ivo://eso.org/outreach?product=eso1238a)
> ...which would mean that an intermediate discovery step would need to
> be there (or perhaps you require ids on the capabilities and
> reference those with fragment identifiers, but then a publication of
> a new image would require an update of the Registry record, which is
> possible, but a little weird).
>
>> 3. Tolerate the existence of IVOIDs not yet defined in the registry
>>     notably for new HiPS providers (=>ex: ivo://JAXA/xxx), and/or accept
>>     the creation of VO registry entries by delegation (the ivo:JAXA/xxx
>>     VO registry would be generated from its properties HiPS definitions)
> But how would clients find out about their existence?
>
>> In the mean time, and for avoiding to jam the HiPS IVOA standardization
>> process during this discussionI suggest to stop the usage of IVOID in the
>> "properties" HiPS file (/publisher_did/ field),  and use instead two new
>> fields splitting the authority-id and the hips-id in two separated fields
>> (/p//ublisher_id/ and /hips_id/). This solution would allow to continue to
>> have a usable and homogeneous identification HiPS mechanism without clashing
>> with the 2.4 constraint, but allowing to generate the associated IVOID
>> (ivo:// prefix + concatenation with "/" separator) if the associated HiPS is
>> really defined in the VO registry.
> Well, my advice is:  Work out how discovery of HiPSes should work.
> After what I understood of the above, I suspect you might get by
> defining one standard ID:
>
> ivo://ivoa.net/std/hips#hips
>
> Such capabilties would have one ParamHTTP interface, the access URL
> of which directly points to a HiPS.  You'd probably also set the
> resultType of that interface to your HiPS media type for consistency
> with case 2:
>
> When there are several HiPSes in one resource, you'll have to put in
> a discovery service.  Let's say it's a SIA service (you could do
> something simpler, too, perhaps just an enumeration on an HTML page
> or whatever).
>
> Let's say you're using SIA.  You'd then define the SIA service in a
> normal Registry record.  Clients only looking for HiPSes could figure
> out which SIA services they should look at by looking at the
> interface; a HiPS service's interface would give whatever media type
> you reserve for HiPS as the interface's resultType.  It's a fairly
> easy registry query pattern, I'd say.
>
> In both cases, I don't think a specialized registry extension would
> be necessary, just use plain vr:Capability (or a SIA capability).
>
> In the first case, the HiPS' identifier would be the resource
> identifier, in the second case, it would be the resource identifier
> plus "?<local part>".
>
>> For the end, I have to apologize for arriving so late in the document
>> Identifier process (PR state). But I have to say that I had totally missed
>> the consequence of the 2.4 contraint until this practical case of the HiPS
>> IVOA standardization raised it.
> I hope I'm making a convincing point that there is (almost) no
> problem.
>
> Or have I missed something?
>
> Cheers,
>
>            Markus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/registry/attachments/20160118/4bc1bdc7/attachment-0001.html>