Cube Data Access Layer implementation note

[Markus Demleitner]

Dear Francois,

On Thu, Mar 03, 2016 at 04:19:06PM +0100, François Bonnarel wrote:
> Hi all,
>     A partial answer:
> On 01/03/2016 12:02, Markus Demleitner wrote:
> >>             - Last but not least. SODA is very close to ObsTAP and
> >>             to SIAV2. It should have been a simple methos of SIAV2.
> >No, most definitely not.  All other protocols offering access to
> >datasets require it as well, and historically, the first protocol it
> >(or its predecessor, which still is very close to it) was used
> >together with was SSA.
> SSA has it own embedded AccessData Functionality. And as for the "GetData"

It does not -- is has been mentioned as something specified in the
future, but it fortunately has never been specified.

Fortunately, because we already have too many standards, and it'd be
terrible if there were different VO protocols for slicing and
processing 1D spectra and slicing and processing 2D or 3D spectra.
Can we agree we have to avoid that?

> service you have been developping at GAVO
> (http://docs.g-vo.org/talks/2012-urbana-ssaevo.pdf and
> http://www.g-vo.org/pmwiki/uploads/Documentation/splat_interop_HD.pdf),
> which offer more to the users,  It is allready fully integrable in the VO as

If you look, this stuff is pretty much what is now called SODA.  And
it has worked for slicing-and-dicing cubes back then, too, as it does
now (see the Califa thing I sent around earlier today).  And there's
proven client support over several services (although only for
spectra, but that's not a matter of principle).

> a CUSTOM SERVICE with help of the standard DataLink functionalities. But
> Having GetData (or its successor) as the standard for cubes is another
> story. See below.

I'm getting a bit tired of repeating this, but: I'm not interested in
custom services; I could just as well write web pages for that.  

I'm after clear specs against which authors of clients can work (and
also authors of services, but that's traditionally been less of a
problem within the VO).  It would totally against what I think the VO
should be if clients needed to implement different rules for
different services.

In case you really want to keep up that custom service argument: I
think we agree that custom services are bad in principle.  So, why
should we accept them when it's completely straightforward and free
to define these kinds of services interoperably?  Or in which way is
it not free?  Or not straightforward?

> SODA functionality has been "extracted" from SIAV2 initial project for one
> single reason : The same functionality had to be accessible frm DIAV2  or
> from ObtAP query phases indifferently. But we cannot do as if there is no

But ObsTAP contains spectra, too, no?  And time series?  And lots of
other kinds of data?

> >
> >>             It shares the same syntax as SIAV2 and the same model
> >>             as ObsTAP and SIAV2. We are defining SODA now so we can
> >No, it does not, at least as far as the model is concerned.  SODA, to
> >be useful, needs to be model-free[1],
> 
> Ah !!! This is the point where we really differ, Markus. Currently SODA
> doesn't have to be model-free it has to be adapted to ND-Cubes and we have a
> simple model for that which is Obscore and we will have a full one, CubeDM
> (in a near future because it's allreday in good shape)

Again I'm getting a bit tired of repeating this, but CubeDM doesn't
really enter into it.  We're talking about SODA *services* here, and
going from a data model description to inferring a service interface
is really involved.  I'll believe that it can be done when I see a
good plan how to do that and a client implmenting it working with two
different services.

I don't believe we'll see any of this any time soon.

So -- before we talk on here -- couldn't you just try it?
Service-side and within Aladin?  I'm pretty sure you'll notice that
datalink-embedded per-dataset SODA descriptors are simple,
straightforward and really what you want to have, whereas everything
else is a minefield of complications (unless you put in a lot of
assumptions that will severely restrict the kind of data you can
process, which would be a pity).


> I think it would be a pity not to benefit from the clean description the
> data model provides for cubes. SODA standard parameetrs are really funded on
> the model so far.

What benefits would that be, by the way?  What is it that you're
after that cannot be done by proper parameter description?

You see, I'm getting a bit exasperated because I believe we have a
simple, well-understood, and easy way that I believe does everything
the CSP has asked us to do and more.

On the other side there's a plan building on a data model that's been
in the works for years already. That DM would be used through an as-yet
undefined service interface ("get-gory-details").  To infer the
service interface for those gory details beyond some basics[1]
is too complicated at least for me to imagine.

I am particularly exasperated because we don't have too many chances
to get this right.  I think we only have this one.  Make SODA
incapable or incredibly complicated and the VO is out for something
that could be actual inroads into the community. People are writing
web pages for data access left and right while we discuss here.

> The mandate we have as DAL WG  is to fulfill the CSP priorities for cubes,
> and allow archives who have cubes to provide simple cutout facilities. We
> don't have to disperse into a generic and then more complex thing.

SODA with proper declarations+Datalink satisfy those requirements
*right now*, without anything we don't understand well. Ok, you'll
need to add RA and DEC as parameters, but that's straightforward.  It
exists and works.

Are you sure you can't agree to accepting this as something that
works *right now* and delay all further complications with tech we
don't really have (or fully understand) at this point until further
versions when perhaps there's implementation experience with this?

Cheers,

          Markus

[1] And we already know that data providers will want a lot more and
complain that our protocols (and in particular, clients) can't do it.
And then they'll write web pages instead of interoperable services.