Questions about UTF-8 in VOTable

[Markus Demleitner]

Dear François,

On Thu, Jul 02, 2026 at 08:51:43PM +0200, Francois Ochsenbein via apps wrote:
> that removing the possibility of specifying that a non-numerical column
> is made of ascii-only characters does not raise more comments or
> concerns?… A few more comments embedded below:

Oh, don't worry: There has been plenty of discussion.  I've been
trying to *somehow* improve the unsatisfying VOTable char situation
since at least 2012, and these attempts got stuck numerous times.
Mark's valiant attempt is the result of a dedicated plenary (IIRC)
in College Park.

I will freely admit that we'd certainly not design this like this if
we started from scratch.  But I am fairly sure we have reached a
result that is the least bad of all the various options we have
looked at over time, and that includes the nD-array decoding that
*is* abominable.

In the end, how I'd frame the question is: What's the least
unexpected thing a VOTable parser can do when encountering characters
>= 0x80 in VOTable char?  I give you that in 2005, multiple answers
were justifiable, and the "reject" answer you gave turned out to be
the wisest.

These days, and I'd be *very* surprised if this were different 20
years from now, it is as clear that it'll be UTF-8 as it was in 2005
that 7 bit characters are ASCII.  Sure, back then there still was
EBCDIC and other horrors.  You were wise enough to keep this
possibility out of VOTable.

Today, despite all the uglyness involved, it is wise to finally give
these eighth bits a meaning; and utf-8 is the least surprising thing
we can do today, despite all the trouble that this brings with it.

One minor remark, though:

> ==> On 2026-06-29 à 13:02+0100,
>       Mark Taylor <m.b.taylor at bristol.ac.uk> wrote:
> >Nearly all software is written these days in an environment in which
> >strings are assumed Unicode, but that doesn't mean that programmers
> >spend their time worrying about the fact that Aries is represented
> >by two code points or that there is no unique way to encode something
> >that looks like an Omega or accented characters.  Comparison of two
> >UTF-8 sequences for equality *is* an easy operation, though it will
> >not necessarily yield true for two strings whose pixel rendering is
> >identical.
> >
>
> Sorry to disagree with the "transparency" of Unicode in the various
> languages: for instance length("άβγ♈︎♉︎😊𝕏") gives 12 in Javascript
> or Java, while Python3 or awk return 10, which is the correct number of
> Unicode code-points in this string(*). Similarly extracting a substring

Ummmmm... that's a function of the *VOTable* library: Will it do the
UTF-8 de- and encoding for you or won't it?  I expect most will, and
then any sensible len-type function will be the number of unicode
codepoints, independently of the language.

When a VOTable library doesn't do the decoding, the length will be
the number of bytes in front of the first 0x00 or arraysize,
whichever is smaller -- again, this is a function of the library and
not of the language.

That the language objects you get from a parse function depend on the
library is nothing new; it's been this way with xtypes for a long,
long time (example: xtype="timestamp": Do you get a string or some
structured time representation?)

So, I'm fairly confident that we won't see major desasters; a certain
problem can be very old VOTables that use some legacy 8-bit encoding
-- but thanks to the no-eighth-bit rule that's been in place so far
these have been broken all the time, and well-behaved VOTable
libraries will have displayed question marks -- which they will, in
all likelihood, still do.

Thanks,

            Markus