Dal architecture concepts

[François Bonnarel]

Dear all,

    Discussion around SODA has some underlying DAL architecture 
conceptions. This a summary of my view of it.

The DAL landscape today is very modular. We have created or are 
developping a pretty large number of different protocols during the 
thirteen last years and the most recent ones work in complementarity on 
some definite aspects and also have to collaborate for higher order and 
more complex fonctionalities.

DALI, for example, is gavering a set of common definitions across 
protocols.

But the main differenciation in the DAL landscape comes with TAP 
emerging beside the so called "Simple Access protocols". TAP and ADQL 
coupled with the simplified observation datamodel ObscOre allow to 
develop the standard generic dataset discovery type of services: the 
so-called "ObsTAP".

SIAV1.0 and SSA have two functionalities : data discovery and  some 
hints of data access (cuting-out, resampling, regridding, mosaicking)

On another side the idea of linking discovered or known datasets to 
additional resources has been  in the air since a long time. Obviously 
there is a need to give access to additional information, other data 
formats, additional processing  without duplicating the lines dedicated 
to a specific dataset in the Query response. The concept of DataLinking 
is born from that need.

For cubes, and later for time series or event list, the Simple protocols 
were insuficient for search and selection on some axes. On the other 
side ObsTAP services are not designed at all for data access and only 
allow full retrieval.

DataLink was the right way to link datasets discovered by ObsTAP to 
various resources and specifically to an AccessData service able to 
provide all kind of cutout and selection facilities (and more access 
functionalities later).

Simple access protocols were long time providing  both discovery and 
accessData methods with a similar interface. Discovery with a simple  
HTTP interface still have advantages  over ObsTAp for some use cases and 
some data providers and was decided at some point to develop SIAV2.0 
version for that.

The current architecture decided at Hawaï interop after year-long 
discussions is the following :
     ObsTAP services and SIAV2 services are parallel discovery (and 
description) pathes for cube datasets The query response is based on 
Obscore datamodel in both cases. Full metadata Resource going beyond 
ObsCore will come later in SIAV2.
      AccessData (now called SODA) is independant and accessible from 
the query responses through one of the DataLink mechanisms. The first 
version is oriented towards Cuting-out and Selection on teh axes. More 
sophisticated access data operations will come later.


Implicit things or still in discussion

      SODA should also be accessible directly and not only from Data 
Discovery services.
       DataLink link resource is a very generic dataset-to-resource 
bounding mechanism. It must stay generic and independant from the 
services it relates which may be operated by various providers.
       SODA services are the best candidates to their auto description.
It is better to rely on them than on DataLink to describe dataset 
specific metadata.
       SODA, ObsTAP and SIAV2 share the same datamodel(s) that should 
reflect in a consistency effort (see my previous email).
       Custom parameters and custom services are interesting features, 
but they are not in our main requirement for this year for SODA at first 
steps. However it is nice and usefull in general  that DataLink is 
providing a way to relate datasets to such services.
        Standard parameters don't really require machine readable 
description because they are defined in the spec and related to the 
underlying model.
     But it can be usefull to give them a description as similar as 
possible to the custom description. This will make easier automatic 
functionality discovery and managment by the clients. However It should 
be done in a way not destroying consistency of the architecture at the 
price of a lack of interoperability.

       Cheers
François