[SIAv2] query params

[Douglas Tody]

Hi All -

I just read/re-read this whole thread and it appears to me that we are
almost in agreement.

The main issues:

     a)  High level vs low level service API, e.g., the need or otherwise
         for custom/abstract parameter types (more than just int, float,
         etc.; e.g., range and list).

     b)  Mechanism to formally describe the input parameters (input
         parameter set) for a service request/operation.  This includes
         not only the parameters and their types, but could also say
         something about allowable values (min/max, enumerated, list or
         array value permitted, etc.).

     c)  Extension parameters, by which a service can extend the standard
         interface to add custom functionality.  Driving a spectrum
         service that derives spectra from a theoretical model is an
         extreme example of this (image has a directly analogous use case
         BTW), but the mechanism is generally useful for lots of things,
         e.g., custom query parameters for an instrumental data
         collection.

Some discussion follows below.  To summarize, a) a few custom types are
preferred by most of us as the best compromise design and are consistent
with current standards and implementations, b) would be great to have,
should be done, but is not a high priority to support initial
implementations, c) extension is a required capability from the
beginning.

         - Doug


On Tue, 12 Nov 2013, Markus Demleitner wrote:
> I don't believe "comprehensible" parameter syntax ought to be a
> design goal.  "Machine readable", on the other hand, is IMHO of
> utmost importance: We're talking about machine-to-machine protocols
> here.

This gets to the point of issue a).  I agree that we want to do things
like limit the allowable coordinate frames or units in a service
interface or it gets too complicated.  If this were a system interface
like SMTP or maybe VOSpace then being machine readable is probably about
all that really matters.  But an interface to get at science data such
as SIA (or SSA etc.) is different, because these interfaces are often
used directly in science applications.  We often bind these interfaces
directly or with a thin layer above, into end user scripting or
applications development languages (e.g. Python, C/C++).  We have to
provide a client-usable service API and not leave it all to higher level
software as otherwise there will be no uniform standard API.  People
need to do things like refer to the DAL specification for reference
documentation of the service interface, e.g., to document the available
parameters and their allowable values.

A parameterized interface works well for this, as proven by numerous
existing astronomical data analysis systems.  The task-parameter model
used in these systems maps directly onto the request-parameter model
used for data access services (the form-based query interface used in
most archive query interfaces is similar).

I could say more but I'll just state my opinion that for a parameter
interface, adding a few, well-defined complex types such as ranges or
lists is very useful and worthwhile, and in most cases is preferable to
the alternatives of overly complex syntax (a full parsed language) or
using only basic types.


> b)  Mechanism to formally describe the input parameters

Certainly we would like to have this and should do it, but it is not a
high priority to get a usable version of SIAV2 out the door.  The most
important thing to do now is to carefully define the types used by the
input parameters.  A machine-readable mechanism to fully define them can
be added later, although some basic mechanism should probably still be
provided initially, similar to what we have provided in the past (e.g.,
comparable to FORMAT=metadata although I have always disliked abusing
FORMAT in that way).

I agree that some of the types used in the past have probably been too
complicated, e.g., BAND being either a bandpass name or a numerical
range or range list.  But types like string, int, float, range, or list
are all well-defined types (range-list being a type of list).  SSA made
a start at formalizing this but is a bit adhoc, and the list of types
could be improved.

If the Datalink mechanism eventually provides a way to define the
interface to an arbitrary service then this same mechanism could provide
a standard way to describe the interface to a DAL service as well.  The
problem is clearly more general than SIA hence requires a more general
solution.  Nonetheless for a major service like SIA it is still
necessary to define the standard API.


> c)  Extension parameters, by which a service can extend the standard
>     interface to add custom functionality.

Being able to extend the standard API to define custom parameters is
essential, and has always been present in the DAL interfaces.  This is
easy to do; the problem is providing a way for the client to discover
the custom extension parameters.  If we eventually have b) then the
problem may be solved.  Other mechanisms such as FORMAT=metadata or VOSI
getCapabilities address part of the problem.  We should have a basic
mechanism for this from the beginning, e.g., at least a list of
supported parameters, a type and description for each, and a flag
indicating whether they are standard or extension parameters.  A proper
formal mechanism probably needs to be deferred to a later version and
will depend upon more general technology not specific to SIA.

This all reminds me so much of the concept of a parameter set as used in
many of our data analysis systems.  A parameter set defines a list of
parameters, and for each defines the name, type (logical and/or basic
type), description, possibly something about the legal values (min, max,
acceptable enumerated values, list permitted, etc.), and other
attributes such as whether the parameter is standard or a custom
extension.  What we probably need to do is formally define the
abstracted parameter set for each service request, plus the mapping of
logical parameters to the transport protocol, e.g., HTTP (mainly via
DALI).