resuming progress on TAP

[Douglas Tody]

Hi All -

We seem to be stuck trying to get to the point where we can all agree
that the core TAP spec should be broad enough to meet the needs of all
the major projects involved.  In the hopes that there is something
we can all agree upon lets look at the following more general point
which Keith raised.  Aurelien also expressed a similar point earlier.
Plus, I think it agrees with a key goal already addressed in the
design of the second generation DAL interfaces.  If so we may have
broad agreement on a critical design concept.

On Thu, 12 Feb 2009, Keith Noddle wrote:
>> On Thu, 12 Feb 2009, Doug Tody wrote:
>> AdqlQuery (AQ) and ParamQuery (PQ) are *service operations*, not query
>> languages, each of which accepts certain parameters and returns
>> certain outputs, provides status returns in a certain way, etc.
>> Altogether this defines the service interface (an abstraction which is
>> not at all specific to HTTP by the way, except for a mapping onto HTTP).
>
> ADQL allows a user to define a query, I hope we are all agreed on that.
> We (DAL) are in the business of describing:
>
> (1) A "Service", which is the *Application* the user wishes to use
> (2) The "Service Interface", which describes *HOW* that application is
> invoked and used
> (3) The "Payload", which defines *WHAT* we want the application to do.

[...]

On Fri, 13 Feb 2009, Keith Noddle wrote:
> To recap, the proposal is that we separate out the "Payload" from the 
> "Service/Service Interface (aka Protocol)" specification(*).

The basic concept here is that we have 1) a "Service" type (TAP, SIA,
SSA, etc.) which is a member of a consistent family of interfaces
(DAL2), 2) a generic "Service Interface" which each member of the
family of services should share, defining how the service is invoked,
and 3) a "Payload" which defines what the individual service does.

There is a critical point here which I have stressed strongly as well,
and it is good to see Keith saying much the same thing, if in slightly
different words.  That is, the DAL2 services should be consistent in
how they are invoked and used, with the basic service interface being
the same for all the services.  All that differs is the "payload",
which is the actual functionality which is unique to each service
type, e.g., what is required to access a table or tableset (TAP),
or a spectrum (SSA), or an image (SIA), and so forth.

Aurelien made this same point in connection with the DALToolkit from
ESAC.  Anyone who has implemented a service framework supporting
several of the DAL services will soon realize that nearly all of
the service interface is the same for each member of a family of
similar services.  This is true for all client side software as well.
We really want to standardize most of this basic service interface
look and feel.  That is, the composition of a service as a set
of operations, definition of common parameters and common dataset
metadata, how input parameters are handled, how parameter values are
composed, how sync/async is handled, how outputs or status/error
conditions are returned, and so on.  All of this is the standard
"Service Interface", defining how the basic syntax and most of the
semantics of the interface for a given service belonging to the
service family.

The basic service interface is very important to provide uniformity
in a family of services, and to enable code sharing and reuse, where
much of the client and server code is the same for all such services.
The "payload" however is where we get to what a given service
actually does.  Given inheritance of a standard service interface,
this is what we add to define what useful work a given type of service
actually does.

The "payload" has two key aspects, 1) the *logical* functionality
defined, and 2) the form in which this is *physically* expressed in the
low level protocol.  The low level protocol, e.g., HTTP in the case of
our current interfaces (sometimes SOAP as well), is important at the
implementation level and in the physical service interface.  By far the
most important part however is the *logical service interface*, which
defines the semantics of the functionality provided by the service.
This is the same at all levels of software, on both the client and
server sides, even when we get far away from a low level protocol
such as HTTP or SOAP.  Ideally this should be defined independently
of the low level protocol or encoding.  Ideally the same logical
service functionality could be implemented with multiple alternative
low level protocols.

By now you probably have an idea where I am headed with this: this
concept or goal, which all of us have promoted at this point, is the
basis of what we already have in the DAL2 service profile introduced
with SSA.  This defines a standard way to compose service interfaces
in terms of logical service operations, each with a defined set of
parameters, with simple standards for parameter usage as well as a
standard mapping onto the HTTP protocol (with other low level mappings
being possible).

What is new for TAP is that thanks to the broader expertise of our
international core design team last fall we were able to expand
this interface pattern to include support for sync/async based
upon UWS, and VOSI standard metadata and service status queries.
The resultant service profile or interface pretty much covers all the
generic functionality required for our production 2ndgen services.
We might have had a cleaner interface pattern had all this been done
from scratch, but nonetheless what we ended up with is quite workable.

To wind up this analysis, the key points I would stress are:

     o	The standard "service interface" or profile for the DAL
 	services defines how the service interface is composed, as
 	well as mapped onto a HTTP protocol.  These are separable
 	concerns although we need to address both each in protocol
 	document.  We want this basic interface pattern to be shared
 	by the entire family of 2ndgen data access protocols (major
 	changes are possible but should be deferred to version 3.0
 	if we want to achieve a useful VO infrastructure before our
 	users abandon us).

     o	The "payload" in Keith's terminology defines the useful
 	functionality performed by each class of service.  This is
 	what differentiates TAP from SIA, and what makes a TAP service
 	useful for table access, or a SIA service useful for image
 	access etc.  In DAL services the "payload" is expressed in
 	terms of the service operations which are unique to each
 	class of service, e.g., AQ and PQ in the case of TAP, QD in
 	the case of SIA/SSA, etc.

     o	It is a mistake to confuse something like optional language
 	dialects in TAP (the LANG parameter in AQ), with the core
 	functionality provided for table access.  TAP is not useful
 	unless it defines the core functionality provided, satisfying
 	the range of primary design use cases.	The purpose of a
 	construct like LANG (providing for alternate query languages)
 	is to provide extensibility, e.g., to support multiple versions
 	of ADQL, or native SQL pass-through.  This extensibility
 	feature is a nice thing to have, but is no substitute for
 	defining the core service functionality so that it addresses
 	all primary use cases.	The same principle applies to adding
 	additional custom service operations or parameters.

The core service functionality ("payload") must be adequately defined
by the service for it to be useful.  It is ok to refer to external
documents defining broader standards, e.g., ADQL, Spectrum data
model, VOTable, DAL2 architecture, etc., but if the given service
specification does not define its unique core functionality it is not
a useful service specification.  We already have standard service
profiles or patterns; what we are after here are services defining
how to do the final, concrete data access.  While a construct like
LANG provides a nice feature supporting arbitrary extension of the
interface (e.g., to native SQL dialects), it is no substitute for
defining the core functionality for access to whatever type of data
the interface supports.

 	- Doug