s_region problem

[Patrick Dowler]

Francois has summaries all the standards that are related in some way
to this issue so I will try not to repeat, but I do need to clarify
 and recall a few things:

1. The STC-S used in TAP-1.0 is *not* itself a standard. Section 6 of
TAP-1.0 is clearly labelled as "informative" and describes a way for
TAP services to outut (in query results) and input (REGION function or
uploaded tables) values that are described with xtype="adql:REGION".
There was not and still is not an STC-S standard. The xtypes in
TAP-1.0 turned out to be pretty ambiguous because ADQL-2.0 doesn't
have a type system and we kind of just made stuff up to fill the gaps.
There is/was quite a lot of inconsistent use even then (to prefix or
not to prefix?, case?)...

2. The extended types defined in DALI-1.1 (interval, point, circle,
polygon) came out of the multi-d requirements and are used in SIA-2.0
and SODA-1.0 (as input params) and the goal from the beginning (Hawaii
Interop) was to standardise a serialistion format that could be used
for input and output. The clear intention was always that these values
would be used in (VO)table output as those defintions in DALI are for
serialisation. I don't want to debate the merits of the choices in
DALI-1.1 but I will note that there are a lot of reasons that the
polymorphic nature of adql:REGION is painful to implement.

3. When ObsCore-1.1 was being worked on, one of the things I tried to
push was that ObsCore should not be to specific in serialisation
because those were not in general within the scope of a data model.
That approach was partially adopted (e.g. for some scalar numeric
types) but when it came to s_region I guess it still looked too far
ahead so nothing was changed.

So, right now there are two ways a service could output polygons:
DALI-1.1 xtype="polygon" or TAP-1.0 xtype="adql:REGION" (and de-facto
STC-S) but only one of those is actually a standard upon which we
should build. TAP-1.1 strongly recommends following DALI but of course
it cannot forbid using both older standards and conventions and/or
custom xtypes/serialisations.

TAP author hat: I think Francois is right that for now,
ObsCore.s_region probably has to follow past convention until the
standard is relaxed, but I haven't read the ObsCore-1.1 spec with my
lawyer's hat on :-). Particularly in SIA-2.0: the input polygons for
queries are DALI-1.1 form and the output would be TAP-1.0/STC-S
form... that needs to be made consistent by allowing DALI polygons in
ObsCore. SODA also uses DALI-1.1 types for input params (output is
"files").

DALI author hat: There are now some calls for something beyond simple
polygons (disjoint areas, holes, etc) that were not considered
important enough to include in DALI-1.1: that is something that can be
pursued in the context of DALI-next.

Personal opinion: In general, clients should be implementing support
for recognising and parsing all the xtypes/serialisations specified in
DALI-1.1 because VOTable(s) can come with those in them. It isn't much
work now (5 simple xtypes) and forms the basis of the type system used
in TAP-1.1

Pat

On 27 November 2017 at 09:32, Theresa Dower <dower at stsci.edu> wrote:
> Hello DAL/Apps,
>
>
> Speaking as a TAP data service provider at MAST, we have put considerable
> effort into implementing footprints based on STC-S.  For CAOM/ObsTAP, even
> our database-internal geometric search functions under the hood use it as
> input. Adding to this would be considerable effort. We're also planning
> replacement of some other standard services with a TAP layer, so my concern
> extends to how those would work as well.
>
>
> I'm confused as to what is being proposed here; am I correct that some data
> providers are currently responding with footprints in a style that is not
> defined by any of our standards? Or is it another known, defined standard
> that can be referenced? Are we suggesting this alternate format should be
> accepted in client queries so that other data providers have to deal with it
> as well, or is the onus entirely on clients to understand either method and
> take the server's lead? I'm not happy with either of these ideas but admit I
> might just not understand the use case.
>
>
> Thanks for your time,
>
> --Theresa Dower
>
>
>
> ________________________________
> From: apps-bounces at ivoa.net <apps-bounces at ivoa.net> on behalf of Felix
> Stoehr <fstoehr at eso.org>
> Sent: Monday, November 27, 2017 10:29 AM
> To: apps at ivoa.net; dal at ivoa.net; François Bonnarel
> Subject: Re: s_region problem
>
> Dear all,
>
> as far as ALMA is concerned, we are using STC-S and have indeed quite
> complex notations with UNION, POLYGON, CIRCLE as well as NOT for holes
> in the footprints. While I can not judge whether or not an
> additional/modified standard of STC-S is really required (for us at
> least STC-S works very well) I can safely say that changing the notation
> would entail non-negligible costs.
>
> In general, of course, the less standards evolve (without backwards
> compatibility), the more stable our systems can be, the less effort it
> is to maintain them and probably also the easier it is for new players
> to take these standards up.
>
> Best regards,
>
> Felix
>
> On 27.11.2017 13:12, François Bonnarel wrote:
>> Dear all,
>>
>> My understanding of this context as a DAL chair involved in several of
>> the specifications as author and (supposed to be) aware of the others.
>>
>> Most of this stuff is extracted from a discussion we had in a more
>> limited group.
>>
>> 1 ) Obscore
>>
>>     The discussion here is actually invloving at least 3 DAL standards +
>> Obscore (which is DM oficially but actually "DM-DAL") : TAP, ADQL, DALI
>> and Obscore. The DAL dillemma is the following : as it encompasses a
>> vast variety of situations and use cases and various interface shapes,
>> DAL technology cannot be defined by a single "SPEC_OF_EVERYTHING". DAL
>> has defined a great deal of different protocols, which of them evolving
>> its own rythm. We have to be carefull all this keep consistent and to be
>> carefull also we do not break everything
>>
>>      Each of the 4 standards involved here have two versions : TAP 1.0,
>> 1.1, ADQL 2.0, 2.1, DALI 1.0, 1.1. Obscore 1.1 and DALI 1.1 are IVOA
>> recommendations since 2017 May the 17th. TAP1.1 and ADQL 1.1 are still
>> in discussion.
>>
>>      1 ) For s_region what does Obscore spec says ? ObsCore is a model
>> (or a view of a larger model) each class/attribute is perfectly defined.
>> To map such a model in a TAP service one has to express it in a TAP
>> schema. The OBscore spec defines precisely this TAP schema. + xtypes for
>> each TAP schema column. In OBscore 1.0 the xtype for s_region is
>> "adql:REGION". Obscore 1.1 maintains the same definitions for s_region
>> (and other ObsCore attributes) than 1.0. But it adds that the definition
>> given in the spec is consistent with TAP 1.0 (the only valid one at the
>> moement it was defined) but that the situation may evolve after the
>> release of TAP 1.1.
>>
>>      2 ) So what does TAP says about xtypes ? 1.0 was defining a couple
>> of them, all bound to adql functions, including adql:Region. adql:REGION
>> WAS AT THE TIME OF WRITING OF 1.0  considered as STC-S. TAP 1.1 gets rid
>> of this and claims that as far as possible xtype should map to the
>> definitions given in DALI. It says "SHOULD" and not MUST of course. You
>> are still allowed to define or use other xtypes if really necessary.
>>
>>      3 ) DALI 1.0 didn't say anything about xtypes. DALI 1.1 defines a
>> couple of them. xtypes CIRCLE and POLYGON are defined as arrays of
>> doubles. This has also been discussed to be xtypes of some of SODA input
>> parameters (together witn interval). These xtypes are those refered by
>> TAP 1.1  as those   that SHOULD be used if no other xtype is required by
>> the use case.
>>
>>      4 ) ADQL 2.0 Defined a couple of functions which TAP was mapping to
>> xtypes: Among them REGION. ADQL 2.1 is considering dropping string-typed
>> REGION and keep more double array oriented CIRCLE and POLYGON. This has
>> consequences for querying. This has no consequence on adql:REGION xtype
>> which can remain defined as in ADQL 2.0 if necessary.
>>
>>
>>      Discussion :
>>
>>                       With the main goal of keeping the defintions and
>> formats stable for a while in the vO in a given context ( the
>> ObsCore/OBstap one here) I think nothing prevent us to keep adql:REGION
>> (equivalent to STC-S string) as an xtype for the s_region attribute in
>> ObsCore.
>>
>>                        I don't know how DACHS is implemented but I
>> assume it generates xtypes in the VOTABLE response of the services from
>> the TAP schemata. So if the the ObsCore TAP schema keeps adql:REGION as
>> the xtype of s_region instead of xtype=polygon, clients like Aladin
>> could still recognize and parse it immediatly.
>>
>>                        Actually it makes more sense than it may appear.
>> The data/metadata (coord system and whatever) distinction is not an
>> issue for Obscore. Everything is ICRS in there. So the STC-S string is
>> unambiguously a polygon, a circle, or any of other stc flavors of space
>> area perfectly understandable. s_region is not only a space area. It's
>> the support of the observation (as defibed by ObsCore and
>> characterisation) expressed as a space area. So it's not that strange to
>> have a spoecific xtype for it, and to keep the historical
>> adql:REGION/STC-S then.
>>
>>                         For POlygon or CIRCLES in general, which have
>> plenty of usages different from "dataset support" (exemply gratia
>> "region of interest", "survey or catalog coverages" , "cross match
>> search regions" ,etc... and which may require different coordinate
>> systems than ICRS going to the POLYGON/CIRCLE xtypes with arrays is the
>> most standard option for the future...
>>
>> Lets' go to Pierre questions in detail
>>
>>
>>
>> Le 24/11/2017 à 12:26, Pierre Fernique a écrit :
>>>
>>>
>>> Dear DAL and Apps members,
>>>
>>> It seems that we have a problem concerning the *s**_region* usage.
>>>
>>> Recently, we discover that some data providers have changed their
>>> syntax, moving from the regular STC-S string to a new one which seems
>>> to be an array of doubles, alternating long and lat. Both Aladin V10
>>> and Aladin Lite fail to manage a such new syntax, and we had
>>> complaints from final users which do no longer see the FoV of their
>>> observations.
>>>
>>> After investigating about this unexpected evolution, it appears that
>>> this new syntax is generated by some versions of DaCHS testing TAP1.1
>>> future possible recommendation.
>>>
>>> So, I have several questions for TAP 1.1 authors and DAL members :
>>>
>>>  1. Is is really true that STC-S s_region should be removed from
>>>     VOTable results ? Personally, I did not see this consequence in
>>>     the TAP 1.1 document. or ?
>>>
>> No. Nothing prevent us to have string field in VOTable
>>>
>>> 1.
>>>
>>>
>>>  2. And if yes : is it only in TAP ? or also in SIA2 or SSA ?
>>>
>> TAP itself doens't prevent to use string-typed FIELDS in the  response.
>> IT only recommands usage of fIELDS with wtype.
>> In the case of s_region the basic thing is not TAP or SIAV2, the
>> discovery protocol, but ObsCOre. ObsCore has to be stabilized.
>> Common discovery metadata in the case of ADQL queries and the case of
>> Parameter query language has been a major progress recently...
>> We should not break it/
>>
>> For SSA si next answer.
>>>
>>>  1. How the client can know the logic (lon/lat or lat/lon,
>>>     counter-clock or anti-counterclock, ...), the frame and other
>>>     coordinate parameters for such array of double ?
>>>
>> I think xtype=POLYGON assumes it's lon/lat. At the moment coordsystem
>> can aonly be given by the de-deprecated COOSYS. So reference to a
>> COOSSYS element seems to be the only solution.
>>>
>>>  1. How providers who already generated complex s_region will do now ?
>>>     (for instance ESO)
>>>
>> I think they should not change that. ObsCore has been designed and
>> discussed durin a long time to be a stable basis.
>>>
>>> 1.
>>>
>>>
>>>  2. How we will manage the transition period ?
>>>
>> In my opinion no transition period is needed for the specific case of
>> s_region which is STCS ICRS.
>>
>> DAL "build up" is always a balance between stability and backward
>> compatibility and controlled progress in the definitions.
>>
>> Cheers
>> François
>>>
>>> Best regards
>>> Pierre Fernique



-- 
Patrick Dowler
Canadian Astronomy Data Centre
Victoria, BC, Canada