ADQL-2.1 internal draft

[Dave Morris]

Hi Markus,

On 2015-06-11 08:40, Markus Demleitner wrote:
> 
> I'm not at all enthusiastic about the slew of optional features
> introduced here anyway -- they're a nightmare for clients, as
> figuring out what "effective grammar" a given service supports is
> going to become very hard for them.
> 

Unfortunately, when I went through the notes from the meetings in Madrid 
and Banff it became clear that it would not be possible to accept the 
proposed the features as mandatory in ADQL-2.1. They all have some kind 
of technical or implementation issue with one or more of our target 
platforms.

> I believe we should sit together and identify what we actually can
> and want to require.

I suspect that if we did this we would be in danger of ending up with 
less functionality not more.

The set operators would not make the grade, nor would the bit wise 
operators. Both of them have implementation issues on one or more of our 
target platforms.

The geometrical functions wouldn't make the grade either, which is why 
they were declared as optional in ADQL-2.0.

> .... We cannot (or at least should not) forbid
> services to accept additional constructs, so extra features on
> individual services are always possible; clients and their users,
> however, would know what they can rely upon (or at least, when they
> can complain to the service providers).
> 

This is essentially what we have right now.

1) The minimal mandatory core language (which has not changed since 
2.0).

2) A clearly defined mechanism for adding functionality (established in 
2.0).

3) A set of standard definitions for the optional features that

     a) Declare 'IF you are going to do X, then this is the standard way 
to do it'
     b) Include the name or symbol of the feature in the list of reserved 
terms
     c) Include the corresponding function or operator in the standard 
grammar

All of these were defined in ADQL-2.0 and TAPRegEx.

The only thing that has changed is that we are explicitly making it 
clear which features are optional.

> ... The grammar impact of features like the
> set operators or common table expressions isn't isolated to one
> "standard" rule needing a change and a set of extra rules otherwise
> ignored; fairly typically several standard rules are impacted.  In the
> end, even allowing quite a bit of after-parse logic, the current spec
> induces at least eight nontrivially different grammars.

I agree that dynamically adding and removing bits from the grammar is 
not an option, it would cause way too much complexity and confusion.

ADQL-2.1 should have a single standard grammar, which includes *all* of 
the features, including the optional ones.

So, just as the grammar for ADQL-2.0 includes all of the geometric types 
and functions, the standard grammar for ADQL-2.1 should include all of 
the EXCEPT, INTERSECT and UNION set operators, along with the '&', '|', 
'^' and '~' bit manipulation operators.

However, because these features are optional, a service MAY reject a 
query containing those terms if it does not support them.

This is exactly the same as the way that the optional geometric types 
and functions are handled in ADQL-2.0.

-----------------

Two things that I would like to propose :

The first is that we remove the requirement that ADQL has to be 
implementable on all our target platforms without requiring a grammar 
parser or rewriting parts of the query.

This was supposedly part of the original discussions behind ADQL-1.x. 
However, as far as I know, it not been explicitly specified as a formal 
requirement in any of the documents.

Meeting this requirement has made it difficult to adopt some of the new 
features.

Therefore I propose we formally reject the requirement and clearly state 
it in the specification.

Alternatively, if we want to keep this as a requirement, then again, we 
should explicitly state it in the specification.

-----------------

The second is that we formally define the acceptance criteria for 
mandatory features.

Accepting something as a mandatory part of the specification has a big 
impact on the range of platforms that can support ADQL.

For example, both the set operators and the bit wise operations have 
implementation issues on one or more of the RDBMS platforms currently in 
use by TAP services.

Making these features mandatory could end up excluding one or more of 
the currently available TAP services.

Therefore I would like to propose we define a formal acceptance criteria 
that any new feature has to meet before being accepted as mandatory.

This should go in the standard, probably in an appendix, along with a 
list of the target platforms on which it must be possible to implement a 
feature before it can be accepted as mandatory.

Then, when we want to make a particular feature a mandatory, we have to 
demonstrate that it meets the acceptance criteria and that it is 
possible to implement the feature on all of the required platforms 
before it can be accepted as mandatory.

Cheers,
Dave


--------
Dave Morris
Senior Troll
Wide Field Astronomy Unit
Institute for Astronomy
University of Edinburgh
--------