RegTAP 1.2 IWD, tap_tables proposal

[Markus Demleitner]

Dear Registry WG,

Since VODataService 1.2 is now REC, the extra features there, in
particular the coverage in space, time, and spectrum, need to be made
available through RegTAP.  Mainly with that goal I have worked a bit
on the specification; see https://github.com/ivoa-std/RegTAP for the
sources.

I'd say the document is now ready for the WG-internal review; I've
put up a built document on http://docs.g-vo.org/RegTAP.pdf for your
convenience.

It would be great if you could have a look at the document.  The bulk
of the material is unchanged, so for the purposes of this internal
review I'd ask you to specifically look at sects. 8.15, 8.16, and
8.17 (which are the tables for the STC coverage from VODataService
1.2), sect. 9 (which admittedly requires a few additions in other
IVOA documents that are not yet there), and 10.13 (that illustrates
how the new tables would be queried).  Everything proposed should
already be available on http://reg.g-vo.org/tap. 

I'd be delighted if we had a WD ready for the interop.

There is another feature, unrelated to VODataService 1.2 and not
(yet) in the document, that I would ask you to consider for RegTAP
1.2: rr.tap_tables.

The story: I have been running the Global TAP Schema, GloTS, for
about a decade now; that's a service that goes to all registered TAP
services, fetches their TAP schemas, and republishes them in a
unified way.  That service is being used by, e.g., TOPCAT in its TAP
discovery window.

Since it sucks to have things based on proprietary services, and
because being impacted whenever there is a broken TAP service in the
VO isn't fun, I've been trying to get rid of that service for quite a
while.  It was intended as a temporary stand-in until the Registry
had caught up.  I claim that has, by and large, happened by now.

However, the simple table of service, table names, and descriptions
that glots has does not directly map to RegTAP.  That is because
services can register their tables[1] either directly in the TAP
record, as separate resources using TAP auxiliary capabilities, or
both.  To cope with this, a client could run a query like this to
obtain a glots.tables equivalent:

    SELECT ivoid, table_name, 
      -- tables directly attached to the TAP service
				table_description, table_utype
			FROM rr.res_table
			NATURAL JOIN rr.capability
			WHERE standard_id='ivo://ivoa.net/std/tap'
		UNION 
			-- tables coming in through relationships; only those declaring
			-- an auxiliary capability *and* a relationship will be considered
			SELECT related_id as ivoid, table_name,
				table_description, table_utype
			FROM rr.res_table as tab
			NATURAL JOIN rr.capability as tabcap
			NATURAL JOIN rr.relationship
			JOIN rr.capability AS svccap
				ON (svccap.ivoid=related_id)
			WHERE 
				svccap.standard_id='ivo://ivoa.net/std/tap'
				AND tabcap.standard_id='ivo://ivoa.net/std/tap#aux'
				AND relationship_type='isservedby') AS q)

-- the UNION drops duplicate records that would otherwise show
when services have tables in both the TAP record and in separate
resource(s).

This builds on an optional ADQL feature (UNION), and it's also quite
a mouthful; I suspect Apps folks will not be happy if I present that
to them as a drop-in replacement for glots.tables.

So, I'd like to propose having this mess on the server side in the
form of a view rr.tap_tables.  This view already exists on
http://dc.g-vo.org/tap, with column names a bit different from the
one in GloTS[2].  Before I head over to Apps trying to use that to
wean them off of GloTS: What do people here think about that?  Should
we include more or less information?  Or should we tell Apps to use a
different discovery pattern altogether?

To already address a relatively obvious shortcoming of the proposal:

I've thought about including the id of the metadata record containing
the table definition (i.e., the TAP service's for direct and the
table's for auxiliary registrations).  However, that would spoil my
nice trick of using union, because that only gobbles up identical
rows.  And that would mean that tables defined in both the main TAP
record and in one or more additional records would turn up twice or
even more often in that tap_table.

On the other hand, perhaps just relying on union to remove these
duplicate names perhaps isn't a good idea anyway.  Right now a table
would also occur twice if the table description (say) were different
on the main service and on the auxiliary record, or if the table has
a utype in the auxiliary record and it does not in the main record.
That doesn't feel terribly robust.  

For now, I'd hope all complications are unnecessary and the table is
useful as-is.

Thanks,

                 Markus


[1] Well, they can still opt to not register them at all, i.e., have
records without VODataService tablesets -- but that has become rare
for TAP services, and after all these years I'd be relatively relaxed
about losing a few tables when going from GloTS to rr until the few
places that don't produce tablesets yet have fixed their services.

[2] RegTAP and TAP, somewhat regrettably, don't agree on the column
names of their tables of tables, and I went for RegTAP names here.