Proposed erratum to clarify arraysize="1"

[Tom Donaldson]

Mark,

You raise several good points.  Since you would be basically OK with these changes in a new VOTable version, I think most of the concerns you raise center around the scope of what errata can/should do.  That's an interesting discussion to have, both in general and specifically for this arraysize issue.

For me and this specific erratum, the bottom line is one of practicality.  There is benefit in encouraging VOTable writers to comply with this interpretation of the spec.  The most effective means of that may actually be adding the warning to validators like votlint.  Beyond that, the erratum just seems like the most sensible place to make the clarification official.

Since this is an erratum, I was thinking in terms of putting the wording there that we would have put in the original if we had realized the lack of clarity.  In that frame of mind, the use of “must” makes sense.  That is what the spec should have said.  (I do see this as a clarification rather than a change.  Section 4.1 does already say, “The arraysize attribute exists when the corresponding table cell contains more than one…”. )

In this case, I don’t see much practical difference between “should” and “must”.  Either way, some services may update to remove arraysize, and some won’t.  We won’t be any less interoperable either way.  Clients, as always, can exercise flexibility to maximize their usefulness.

I may be missing some of the practical reasons why it’s bad to declare existing services noncompliant. Although there has historically been a stigma attached to noncompliance, that is certainly not an intent here.  Instead the hope is that noncompliance would be a mechanism for informing providers and encouraging change where practical.

So again, the bottom line for me is that I’ll be happy to see validators updated, and will agree to pretty much any consensus wording in the erratum itself.

Thanks,
Tom


From: <apps-bounces at ivoa.net> on behalf of Markus Demleitner <msdemlei at ari.uni-heidelberg.de>
Date: Monday, February 12, 2018 at 4:10 AM
To: Applications WG <apps at ivoa.net>
Subject: Re: Proposed erratum to clarify arraysize="1"

Hi Mark,

On Sat, Feb 10, 2018 at 12:13:34AM +0000, Mark Taylor wrote:

[on http://wiki.ivoa.net/twiki/bin/view/IVOA/VOTable-1_3-Err-3]

On Fri, 9 Feb 2018, Markus Demleitner wrote:
> I would, however, argue that in order to have a clear standard, we
> should upgrade the shoulds to musts, i.e., for sect. 2.2:
>
>   Note: the arraysize attribute must be present if, and only if,
>   each table cell for the FIELD is intended to be treated as an
>   array. Hence, arraysize="1" must not be used except in the unusual
>   case that the table cells contain single values that are
>   intended to be understood as single-value arrays.
>
> And for sect. 4.1:
>
>   The arraysize attribute must be omitted unless the corresponding
>   table cell contents is intended to be understood as an array.
This seems to me to be going beyond what's reasonable for an Erratum.
Software and existing VOTable documents that were written in good
faith in accordance with (at least a plausible reading of) the standard
would go from being legal to illegal if these MUSTs were introduced.

True.  I claim that's within what an Erratum might need to do, but
that's for another (DocStd) discussion.

For this concrete problem, I'd just like to have clear rules in the
future in order to fix the current mess, where two of the most popular
VOTable consumers disagree, making very common use cases work on one
and break on others (see below).  However, ...

> Applied to arraysize="1" this would mean that VOTable parsers would
> probably need to let users override the behaviour that arraysize="1"
> are arrays, and I'd argue that's not something we should impose on
> implementors.
This erratum already (of necessity) imposes the suggestion to
"exercise flexibility" on implementors:
    "However, clients may still wish to exercise flexibility
     there since not all services are compliant with these
     clarified semantics."
In practice, as this advice acknowledges, many clients will have
to cope with both corrected and uncorrected behaviours.
Otherwise, it's very likely things will break.  I certainly
don't plan to 'fix' STIL/STILTS/TOPCAT so that all arraysize="1"
values it encounters will turn into 1-element arrays.

...that's a clear statement.  And I have a lot of sympathy for that,
having been scolded for fixing (arguably) broken behaviour in
deployed systems just very recently.  Now, writing an erratum that
makes the (arguably) most widely used implementation invalid for the
forseeable future is silly.  So, I retract my previous proposition.

Still, I'd like this erratum to make it very clear that if you're
producing arraysize="1" scalars, you should stop doing that as fast
as possible.

How would everyone feel about the following wording?

  Section 2.2 - append the following text to end of the section:

    Note: VOTable parsers should interpret any FIELD with an
    arraysize attribute such that each table cell is treated as an
    array.  Hence, arraysize="1" SHOULD be interpreted as declaring
    single-value arrays.  VOTable writers MUST NOT add arraysize="1"
    to FIELD elements except in the unusual case that all table cells
    are to be understood as single-value arrays.

  And for sect. 4.1:

    On writing, the arraysize attribute must be omitted unless the
    corresponding table cell contents is intended to be understood as
    an array.

This would let parsers be lenient if their authors so desire, but
would provide sufficient leverage to dissuade people from further
producing "bad" VOTables -- because:

that the behaviour of that software, rather than the content of the
standard, ought to be changed.  Does anybody know which software
does in fact interpret arraysize="1" as a 1-element array?

Astropy does, for instance, and the incompatiblity between TOPCAT and
astropy, two major consumers of VOTables for Joe Astronomer, is what
made me complain in the first place.

But then there's the question of principle:

But to invalidate existing legal services and documents with an
erratum looks questionable to me.  I wouldn't oppose a SHOULD here

I'd say that at least depends on how "obvious" the error is.  If we
forbade errata from changing validation criteria, I'd argue they
would be rather pointless.  Sometimes, they will make valid things
that were previously invalid, sometimes it's the other way round.  I
think the more interesting question is whether or not the error is
"obvious".  Marks argument about the provenance of this feature (FITS
binary tables) and this:

which could be strengthened to a MUST in future VOTable revisions.
(Though I feel that even a SHOULD might be an abuse of the Erratum
process; it's not really clear that this is a "clarification" of
the original intention rather than finally deciding which side of
a long-standing fence to come down on).

are valid points to make the error non-"obvious".  Or make it not an
error at all.

Rats.  But, Mark, suppose I'd like to fix the horrible user
experience that something that works in STILTS breaks as soon as
people move it to (or interface it with) astropy as quickly as possible
-- assuming you don't like the wording above, what's the strongest
wording you could live with?

        -- Markus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/apps/attachments/20180212/b67fad0c/attachment-0001.html>