Persistent TAP uploads

[Patrick Dowler]

I like the ideas (uses case) here and it will be interesting to
compare/contrast
this with the approaches we took in youcat (mainly catalogue publishing).

On the create index, which we discussed a bit already, we did implement
that and

(i) it really needs to be an async (uws) job if the tables might be
large-ish and/or
the service wants to constrain load on the database

(ii) our very simplistic parameter-based solution allows for single column
indices
(unique optional) and while it seems very underwhelming on the surface it
has been
"good enough" for the youcat use cases.

Since TAP (tap_schema) is also  too simple to describe a multi-column index
correctly,
it seems like extending this will grow in scope/complexity.

--
Patrick Dowler
Canadian Astronomy Data Centre
Victoria, BC, Canada


On Mon, 14 Oct 2024 at 07:59, Markus Demleitner via dal <dal at ivoa.net>
wrote:

> Dear Colleagues,
>
> I have lobbied for persistent uploads -- i.e., you upload a table as
> with UPLOAD, but the table remains server-side after the request,
> which it doesn't for DALI UPLOADs -- in TAP for a long while now, but
> always procrastinated on it because it *somehow* needs federated auth
> to be any fun.
>
> Well... by now I figured *something* is better than nothing at all,
> and so I went ahead and put a (functional) sketch of persistent
> uploads into DaCHS.  In that scheme, somewhat inspired by Pat's
> 2018 youcat talk
> (https://wiki.ivoa.net/internal/IVOA/InterOpNov2018DAL/tap-youcat.pdf;
> but that's more a publication interface than the user facility I have
> in mind), there is a sibling endpoint to /capabilities, /tables,
> /sync, and /async, and you can PUT or DELETE tables which then appear
> in or disappear from a magic schema TAP_USER; GET requests there
> return VOSI-style table metadata, and the proposal would be that both
> /tables and TAP_SCHEMA are unaffected by user uploads.  I think I
> have good reasons for that, such as: Clients need *some* changes to
> support persistent uploads anyway; keeping your user tables out of
> the potentially long public table listings is probably a nice
> service; you don't want to make people reload /tables too often,
> because it's a fairly large beast on several interesting services.
>
> The implementation also features an ADQL extension: you can prefix a
> query with CREATE TABLE <name> AS and create a table in TAP_USER,
> too.  Whether that would be a standard facility I can't judge; that
> particular thing was relatively messy in implementation, because
> suddenly the ADQL parser needs to know the requesting user, which in
> DaCHS it so far has not.
>
> Anyway, I have prepared a blog post with a few more details; I might
> build that into an IVOA note after Malta depending on feedback there.
> Meanwhile:
> <https://blog.g-vo.org/a-proposal-for-persistent-tap-uploads.html>;
> there's also a Jupyter notebook in there that you use to play around
> with the existing uploads (note, however, that the server has a
> planned downtime on Wednesday morning CEST).
>
> In the post, I am also giving two points on where I see the most
> pressing open issues:
>
> (a) management interfaces: extending table lifetimes (probably easy);
> letting users create indexes (probably pretty hard).
>
> (b) authz: Should we mandate a facility through which users can share
> their tables with other users?
>
> On (b), my assessment is that this is so much trouble -- including
> GDPR trouble in case we let people discover who else has accounts
> where -- that I'd tell people: "Well, do a select * from tap_user and
> mail your colleague the result; or just send them your credentials."
>
> But then I admit I tend to have little patience for that kind of
> thing, and so I'd be grateful for proposals on authz.  But I'd be
> much more grateful for a good interface to requesting indexes.  Does
> anyone perhaps even have such an interface already?
>
> Thanks,
>
>             Markus
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dal/attachments/20241015/7e6dc362/attachment.htm>