[SIAv2] upload

[Markus Demleitner]

Hi all,

On Fri, Mar 07, 2014 at 01:32:16PM -0800, Patrick Dowler wrote:

> On 22/01/14 11:42 AM, Patrick Dowler wrote:
> >Magically get positional constraints out of the table:
> >       POS=<name>
> >- would work in simple contrived cases
> >- table might not contain any identifiable position data
> >- table might contain multiple/ambiguous position data
> 
> It is really hard to get the magic stuff to work well and certainly
> not currently feasible to resolve ambiguities.

Agreed.

> >Get specific constraints out of a column:
> >       POS=<name>.<field.name>
> >- would resolve the ambiguity, but assumes a single field contains the
> >complete position data (no multiple columns with RA and DEC, for example)
> 
> I think this needs to be supported because a TAP query can have a
> column with a pair of coordinates:
> 
>       select s_region from ivoa.ObsCore ...
>       select CENTROID(s_region) from ivoa.ObsCore ...
> 
> and such output tables should be usable. Several of the original
> ObsTAP use cases revolved around using ObsCore output tables as input
> to another query and the CSP "upload list" feature echoed the same
> kind of uses.

...on the other hand, it's not really hard to

SELECT COORD1(s_region) as ra, COORD2(s_region) as dec 

or similar; I keep being *really* skeptical of having STC-S with its
explicit frames, units and all the mess that comes with it in
VOTables; in particular when you start ingesting that into a DB, this
becomes jolly good fun.  Judging from current experience, I don't
think we'll see robust (as in: remotely correct) support for rocket
science of this sort in many services any time soon.

So: No, let's not allow STC-S columns to specify constraints from
uploaded tables, whatever else we do.

> >Get specific constraints out of multiple columns:
> >       POS=CIRCLE <name>.<ra column> <name>.<dec column> <name>.<radius
> >column>
> >- <name>.<radius column> could be replaced by a numeric constant
> >- typical tables would not be feasibly usable with any other geometric
> >value types, but circle is > 95% of the use cases anyway
> 
> So, with my CADC hat on, I would say that query params need to refer
> to the columns in the uploaded table. We should limit SIA-2.0 to a
> single UPLOAD of a VOTable for simplicity. I would favour a
> shell-script-like syntax to identify columns, eg:
> 
> UPLOAD=foo,http;//example.com/mytable.xml
> 
> POS=CIRCLE $foo.RA $foo.DEC $foo.rad

Ouch.  It's this kind of thing that makes me lobby against complex
service parameters.  Please, let's keep them atomic.  For this kind
of thing, it's perfectly ok, simpler, and equivalent in expressive
power to have

RA_REF=ra DEC_REF=dec SR=0.01

(and probably support SR_REF, too).  This immediately makes it clear
that no support for polygon, box, or ellipse is expected, that we're
talking ICRS (though metadata responses should say so in their STC
metadata, of course), and proper metadata can be declared.

> values, eg:
> 
> POL=$foo.pol_states

While I'm all for dynamic typing, it only works if you can reliably
serialize typed values (e.g., "s/34.1" is a string, "d/34.1" is a
float").  In DAL parameters, we can't, so static typing is what we
have to do if we want typing at all, and I, for one, want typing.  

This means: parameters have one type.  RA is a float, and POL is an
enumeration.  Neither type can be constructed from a string like
"$foo.pol_states", and if we still require services to accept them,
we must make them lie in the PARAM in the metadata response (or
suddenly say RA is a string, and people have every right to delight
us with 15h22m34.55s).  We all know from Asimov's Robot stories what
happens when you make computers lie...

Let's have POL_REF for that and keep POL clean.

> 1. Ignore CSP "upload lists" and say that UPLOAD is not supported in SIA-2.0

Would be my preferred solution.  We have a clean, standard way for
providing this kind of functionality in obscore.  Saying: "If you want
this, use Obscore" would also help Obscore's adoption, for which it's
high time.

Plus, let me quote The Programmer's Scripture a.k.a. the Zen of Python:

msdemlei at victor:/home/msdemlei > python -c "import this" | grep prefer
There should be one-- and preferably only one --obvious way to do it.

(I'd like input from the Netherlands on this, though, as the Zen of
python continues:
msdemlei at victor:/home/msdemlei > python -c "import this" | grep Dutch 
Although that way may not be obvious at first unless you're Dutch.
)

> Plus one option: Do we make support for UPLOAD mandatory or optional?
> Note that if it is optional, that opens up needing a way to declare
> support for it, eg VOSI_capabilities, which implies a RegExt.
> SimpleDALRegExt doesn't cover this...

If we specify it at all, we should make it mandatory.  Optional
features are a pain for clients, require additional service metadata,
possibly handshakes, make all-VO queries a nightmare and generally
instill sighs of "Oh, make your mind up already." in our readers.

Cheers,

        Markus