Obstap / about optional fields

[Douglas Tody]

Optional metadata is metadata which is defined in a standard way by
ObsCore (standard, name, ucd, utype, etc.), but which is not mandatory
for all data providers to provide.

Optional metadata does not interfere with generic queries where the
same query is broadcast to many services; it is just that in a generic
query one cannot reference optional metadata fields in the query.
So, if one is doing generic data discovery the query should only be
composed using the mandatory metadata fields.

However, optional metadata allows a more complete description of
specific data collections or archives.  This metadata can come
back in a query response even though it was not used in the query,
providing more complete information about the data (if multiple
query responses are then aggregated by the client it may or may not
choose to preserve the extra metadata).  Furthermore it can be used
in more focused queries of a specific archive.  This will allow use of
TAP/ObsTAP to drill-down and do more precise queries to access data in
a specific archive.  Again, this will not interfere with global/generic
queries since these are restricted to only the mandatory fields.

 	- Doug


On Thu, 17 Mar 2011, François Bonnarel wrote:

> Hi JuanDe
> I think the simplest for version 1.0 is solution 1 with a minor modification. 
> A standard error message is enough
> for services not supporting the non mandatory fields.
> The idea is to ALLOW the discovery of datasets which will stay headen 
> otherwise....
> The idea is also to minimize the extra work for this version ...
> Cheers
> François
> Le 17/03/2011 10:05, Juande Santander Vela a écrit :
>> Perhaps it has eluded me and is already discussed in the ObsCore document, 
>> but I cannot recall having read it.
>> 
>> I have the following problem with optional fields: when we say that a given 
>> ObsTAP field is optional, what does exactly mean:
>> 
>> a) The field is not mandatory in the ivoa.ObsCore table,  and
>>     therefore no entries exist in TAP_SCHEMA.columns with
>>     table_name = "ivoa.ObsCore".
>> 
>> b) The field is mandatory in the ivoa.ObsCore table, but the
>>     content for all rows is NULL.
>> 
>> The problem with approach a) is that we are defeating the ability of 
>> sending the same ADQL query in all ObsTAP compliant services, unless the 
>> ObsTAP implementations return NULL by default to columns they do not know 
>> about in a given schema.
>> 
>> So, for solving that issue, I see three approaches:
>> 
>> 1) TAP services return NULL for columns they know nothing about.
>>     This seems like a hack, and an extra burden for TAP
>>     implementations.
>> 
>> 2) Take approach b) above. But then we would have to check
>>     that the semantics for NULL do not interfere with anything
>>     we have already in place.
>> 
>> 3) Group optional fields, so that they exist or not together
>>    (even if some of them need to be NULL), and the first discovery
>>     of relevant ObsTAP services is made through capabilities search
>>     in the registry.
>> 
>> I think 3) is more natural: we have more information in the registry for 
>> service discovery, and we ensure that the fields exist for the use case we 
>> are interested about.
>> 
>> What do you think?
>> 
>> Am 17/03/2011 um 00:14 schrieb François Bonnarel:
>> 
>>> It was admitted a long time ago that the obstap parameters should alllow 
>>> the same ADQL
>>> query to be performed on all Obstap/compliant services. Therefore a 
>>> limited set of mandatory
>>> parameters has been defined covering most of the Initially proposed use 
>>> cases.
>>> 
>>> However a few of them (polarization data, Cubes with Radial Velocity, 
>>> specific dataproduct subtypes, etc...) were missing.
>>> A small set of additional optional parameters have been defined in the 
>>> document. (Some new ones have been proposed
>>> in the current discussion)
>>> 
>>> I see two interests in these optional fields:
>>>      * complete the description and help the data discovery phase by 
>>> displaying more complete descriptions of the data sets
>>>
>>>      * extend the Obstap Query mechanism for the benefit of small 
>>> specialized communities...
>>> ADQL query on optional field will make no harm to "minimal" Obstap 
>>> services and will return no
>>> data from them. But data providers of these kind of data will probably 
>>> fill these optional
>>> fields for a better exposition of them, and data consumers may use them in 
>>> their queries if they are interested  and get a chance to discover some 
>>> .....
>>> 
>> --
>> Juande Santander Vela
>> Software Engineer, ALMA Archive Subsystem
>> Data Flow Infrastructure Department, Software Development Division
>> European Southern Observatory (Germany)
>> 
>> George Carlin: Si es cierto que la Humanidad está sola en el Universo, 
>> habría que decir que el Universo no tenía miras muy altas, y se conformó 
>> con bastante poco.
>> 
>> 
>