thoughts on TAP-1.2

[Markus Demleitner]

Hi Pat,

On Wed, Mar 16, 2022 at 11:11:33AM -0700, Patrick Dowler wrote:
> I wanted to remind people of some work we did at CADC/CANFAR to enable
> users to manage tables in a TAP service. The short version of this is that

That's undoubtedly an important topic.

> it supports create table, create index, drop table, setting permissions
> (private, groups via GMS, or public), and loading data (only adding rows).

The thing that worries me a bit about the current proposal is that
the operations *are* fairly similar to what we offer in VOSpace, and
if we have two rather different APIs for what's straightforwardly
subsumed as remote data management, I think we should have strong
reasons.

Have you considered employing VOSpace for this?  If so, why did you
discard it?  Could it perhaps be fixed to work for this use case?

> The REST API docs for our youcat service are here:
> https://ws-cadc.canfar.net/youcat/

Grumblegrumblefailswithoutjavascriptwithnoobviousreasongrumblegrumble.

> add). I know Adrian already has python tap client support for these
> features somewhere (maybe in cadctap) and they could be moved up to the
> pyvo TAP client pretty easily if that happens.

...which, of course, is probably true for your VOSpace code, too --
which I personally would very much welcome it in pyVO regardless of
what API we'll use for persistent TAP uploads.

> So, if you are interested check out the above links and let's figure out
> how/when to discuss.

I'm in for such a discussion; I think what would interest me most in
addition to the management API is what impact this has on TAP_SCHEMA
and /tables.

Implementation-wise, adapting /tables to show uploaded tables seems
rather straightforward (although: see below).  Reflecting them in
TAP_SCHEMA looks a lot more complex -- what do you do (if you show
them there at all)?  Make it some sort for per-user set of views?
Re-write queries to point them to per-user TAP_SCHEMA-s?

This is even more relevant when you let people set their uploaded
tables to "public" -- I am absolutely sure I wouldn't want
user-uploaded tables to contaminate my official tap_schema, let alone
what I send to the registry (and hence the /tables endpoint -- saying
"/tables may be different from what you send to the Registry is an
alternative I wouldn't like to consider).  User-uploaded tables
aren't registry material their metadata is almost
certain to be sh^Hubstandard , but also because there are
hair-raising problems as to uniqueness of names, persistence, and
basically anything else.

On the other hand, if I don't give that metadata, these tables won't
be in TOPCAT's table browser, and syntax highlighting will pour red
all across queries against such tables.

How do you deal with this?

After these considerations, I start suspecting that we're doing
ourselves a favour if we keep these -- whether public or not -- out
of TAP_SCHEMA and /tables altogether and rather have an
/ephemeral_tables (say; /user_tables would be ok for me, too)
endpoint and perhaps a corresponding ETAP_SCHEMA (or whatever; it'd
need quite a bit of convincing to make me support that latter thing).
That's simpler in implementation and easily prevents bleeding of
unpublishable material into Registry records.

           -- Markus