Fwd: Votable status

[Mark Taylor]

(followups to VOTable list please)

Hi Laurent,

On Tue, 20 Nov 2012, Laurent Bourgès wrote:

> Dear all,
> 
> I would like to give my comments about the VOTable standard (1.2) and its
> new revision 1.3 in order to simplify the standard document (i.e. make
> optional or minor items deprecated) and clarify many topics (explanations,
> use cases).
> 
> Could you remind me links to VOTable 1.3 documentation (wiki page,
> documents) ?

We started discussing this at Pune (Oct 2011), and I attempted to
start a mailing list discussion following that with this message:

   http://www.ivoa.net/pipermail/interop/2011-October/001961.html

We have revisited the issues at subsequent Interop meetings,
most recently at Sao Paulo; see presentations:

   http://wiki.ivoa.net/internal/IVOA/InterOpOct2012Applications/votable.pdf
   http://wiki.ivoa.net/internal/IVOA/PlenarySessionsOct2012/apps-review.pdf

other discussions have taken place on the VOTable mailing list,
and are visible on the archive:

   http://www.ivoa.net/pipermail/votable/

In particular I posted a couple of messages there in October pointing
to the history, recent changes and current version in the Volute
repository.

You will note from these discussions that we are towards the end of
the period for discussions on VOTable 1.3, and are hoping to have a
PR around the time of the next Interop.  As part of the consideration
of the changes, we have agreed that we were going to keep
changes to the document at a minimum.  The upgrade to version 1.3 is
to address some specific issues that have been identified as needing
a fix, and we have explicitly decided not to take this as an
opportunity to rework other parts of the standard.  This decision
was suggested in TCG consideration of the matter, and has not
attracted disagreement during the last year of public discussion.
There are parts of the existing standard that I'm not very keen on,
but given the central place that VOTable occupies within the VO
infrastructure, we wanted as much as possible to avoid introducing
new incompatibilities.

> FYI, I started working with andre schaff on the new Savot implementation
> (4.0) first to fix performance issues and also to add missing features
> (Binary, datatype handling, id / name / reference resolver ...)
> 
> Recently I looked in depth at the standard related to DATA handling,
> encodings (base64, gzip) on BINARY (or TD) and I advocate many points were
> fuzzy to me:
> - which BINARY encoding must be supported by implementation ?
> - which are concrete rules / best practices related to data encoding /
> decoding as VOTable ?

To me the standard doesn't look very well worded/thought out in
this part, but my reading is that inline STREAMs have to be
base-64 encoded (otherwise it will screw up the XML),
so encoding="base64" ought to be explicitly
stated, and therefore gzip is not possible because the encoding
attribute is already used.  From a parser's point of view, if
no encoding is specified, it had better assume base64 anyway.

For external streams any of "gzip", "base64", "dynamic" or null are
possible and should be understood, but base64 would be eccentric
and if you ignore that possibility you're not likely to have trouble
(FWIW my parser ignores the attribute and checks the magic number
to see if it's gzipped).

> - who is using VOTable extensions (XMLDATA or TD encoding) ?

Anything in Appendix A (e.g. XMLDATA) is not part of the VOTable
standard.  I'm not aware of anybody using XMLDATA or TD encoding;
in any case I don't consider a general VOTable parsing library
under obligation to support them or any other Appendix A items
(STIL does not).

I argued strongly against the introduction of the encoding attribute
of TD (March/July 2009 on the votable list), but I was overruled.
It's only there to address some alleged parser deficiencies.
I don't believe parsers need to pay any attention to it.

> - what means gzip encoding in a xml document (gzip + base64) ? why it is
> better than base64 + complete votable document compression (gzip or http
> compression) ?

I don't think there's much point in this, and as I say above I don't
think the standard allows it to happen in any case.

> About compression: very huge TABLEDATA can be compressed (gzip) much more
> than BINARY (base64) or (gzip) because compression does not work well on
> binary streams.

I addressed this in my presentation at Urbana
(http://wiki.ivoa.net/internal/IVOA/InterOpMay2012Applications/votable-plenary.pdf,
page 10).
The figures I got suggested that BINARY compression was slightly,
but not significantly, worse than TABLEDATA compression.
The consensus among those present was, marginally, that BINARY2
was still desirable to allow more efficient non-compressed VOTables.

Mark

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/