Proposed erratum to clarify arraysize="1"

[Markus Demleitner]

Hi Mark,

On Sat, Feb 10, 2018 at 12:13:34AM +0000, Mark Taylor wrote:

[on http://wiki.ivoa.net/twiki/bin/view/IVOA/VOTable-1_3-Err-3]

> On Fri, 9 Feb 2018, Markus Demleitner wrote:
> 
> > I would, however, argue that in order to have a clear standard, we
> > should upgrade the shoulds to musts, i.e., for sect. 2.2:
> >
> >   Note: the arraysize attribute must be present if, and only if,
> >   each table cell for the FIELD is intended to be treated as an
> >   array. Hence, arraysize="1" must not be used except in the unusual
> >   case that the table cells contain single values that are
> >   intended to be understood as single-value arrays.
> >
> > And for sect. 4.1:
> >
> >   The arraysize attribute must be omitted unless the corresponding
> >   table cell contents is intended to be understood as an array.
> 
> This seems to me to be going beyond what's reasonable for an Erratum.
> Software and existing VOTable documents that were written in good
> faith in accordance with (at least a plausible reading of) the standard
> would go from being legal to illegal if these MUSTs were introduced.

True.  I claim that's within what an Erratum might need to do, but
that's for another (DocStd) discussion.

For this concrete problem, I'd just like to have clear rules in the
future in order to fix the current mess, where two of the most popular
VOTable consumers disagree, making very common use cases work on one
and break on others (see below).  However, ...

> > Applied to arraysize="1" this would mean that VOTable parsers would
> > probably need to let users override the behaviour that arraysize="1"
> > are arrays, and I'd argue that's not something we should impose on
> > implementors.
> 
> This erratum already (of necessity) imposes the suggestion to
> "exercise flexibility" on implementors:
> 
>    "However, clients may still wish to exercise flexibility
>     there since not all services are compliant with these
>     clarified semantics."
> 
> In practice, as this advice acknowledges, many clients will have
> to cope with both corrected and uncorrected behaviours.
> Otherwise, it's very likely things will break.  I certainly
> don't plan to 'fix' STIL/STILTS/TOPCAT so that all arraysize="1"
> values it encounters will turn into 1-element arrays.

...that's a clear statement.  And I have a lot of sympathy for that,
having been scolded for fixing (arguably) broken behaviour in
deployed systems just very recently.  Now, writing an erratum that
makes the (arguably) most widely used implementation invalid for the
forseeable future is silly.  So, I retract my previous proposition.

Still, I'd like this erratum to make it very clear that if you're
producing arraysize="1" scalars, you should stop doing that as fast
as possible.

How would everyone feel about the following wording?

  Section 2.2 - append the following text to end of the section:

    Note: VOTable parsers should interpret any FIELD with an
    arraysize attribute such that each table cell is treated as an
    array.  Hence, arraysize="1" SHOULD be interpreted as declaring
    single-value arrays.  VOTable writers MUST NOT add arraysize="1"
    to FIELD elements except in the unusual case that all table cells
    are to be understood as single-value arrays.

  And for sect. 4.1:

    On writing, the arraysize attribute must be omitted unless the
    corresponding table cell contents is intended to be understood as
    an array.

This would let parsers be lenient if their authors so desire, but
would provide sufficient leverage to dissuade people from further
producing "bad" VOTables -- because:

> that the behaviour of that software, rather than the content of the
> standard, ought to be changed.  Does anybody know which software
> does in fact interpret arraysize="1" as a 1-element array?

Astropy does, for instance, and the incompatiblity between TOPCAT and
astropy, two major consumers of VOTables for Joe Astronomer, is what
made me complain in the first place.

But then there's the question of principle:

> But to invalidate existing legal services and documents with an
> erratum looks questionable to me.  I wouldn't oppose a SHOULD here

I'd say that at least depends on how "obvious" the error is.  If we
forbade errata from changing validation criteria, I'd argue they
would be rather pointless.  Sometimes, they will make valid things
that were previously invalid, sometimes it's the other way round.  I
think the more interesting question is whether or not the error is
"obvious".  Marks argument about the provenance of this feature (FITS
binary tables) and this:

> which could be strengthened to a MUST in future VOTable revisions.
> (Though I feel that even a SHOULD might be an abuse of the Erratum
> process; it's not really clear that this is a "clarification" of
> the original intention rather than finally deciding which side of
> a long-standing fence to come down on).

are valid points to make the error non-"obvious".  Or make it not an
error at all.

Rats.  But, Mark, suppose I'd like to fix the horrible user
experience that something that works in STILTS breaks as soon as
people move it to (or interface it with) astropy as quickly as possible
-- assuming you don't like the wording above, what's the strongest
wording you could live with?

        -- Markus