Questions about UTF-8 in VOTable

[Mark Taylor]

François,

thank you for this succinct summary of where we are.

In order to decide whether we are going to carry on with the 
Unicode-related changes in the current VOTable 1.6 Working Draft 
(which has already had quite a lot of opportunity for discussion,
see e.g. http://mail.ivoa.net/pipermail/apps/2025-November/001793.html
and https://github.com/ivoa-std/VOTable/pull/71 )
or whether we need to rethink it, this is a good opportunity
to ask the question that François highlights: do people care that
under the existing VOTable 1.6 WD there is no general way to signal
that a column contains ASCII rather than Unicode text?  

Evidently Francois's answer is yes; mine is no, because in practice 
it doesn't change how software is going to treat the content 
(you can read earlier entries in this thread for more detail on 
these positions).  Such an indicator would be a nice to have;
it is provided for fixed-width strings by the co-opting of the width
attribute, but options that provide a indication of this distinction
in all cases would be considerably more disruptive to updates of
the VOTable format than what we are proposing.

If there are other people with an interest in VOTable who do have
a strong opinion about this that is contrary to the approach in the
current VOTable 1.6 WD, then please speak up to say:

  1. Why it is important to flag ASCII-ness of VOTable columns,
     and in particular what benefit such a flag would grant to
     VOTable-consuming applications or code.

  2. What alternative approach you prefer, e.g:
     - leave VOTable as it is now with inadequate Unicode-handling
       capabilities
     - a new datatype (unreadable by pre-1.6 software) for Unicode content
     - (ab)use of existing FIELD attributes or introduction of new ones

Mark

On Fri, 10 Jul 2026, Francois Ochsenbein via apps wrote:

> Dear Markus,
> 
> Thank you for the historical explanations but still the question of *how
> to specify that a <FIELD> is made of ascii characters only* has no answer
> in the current VOTable draft… But apparently nobody cares ?
> Cheers, François
> 
> Le lun. 6 juil. 2026 à 14:48, Markus Demleitner via apps <apps at ivoa.net> a
> écrit :
> 
> > Dear François,
> >
> > On Thu, Jul 02, 2026 at 08:51:43PM +0200, Francois Ochsenbein via apps
> > wrote:
> > > that removing the possibility of specifying that a non-numerical column
> > > is made of ascii-only characters does not raise more comments or
> > > concerns?… A few more comments embedded below:
> >
> > Oh, don't worry: There has been plenty of discussion.  I've been
> > trying to *somehow* improve the unsatisfying VOTable char situation
> > since at least 2012, and these attempts got stuck numerous times.
> > Mark's valiant attempt is the result of a dedicated plenary (IIRC)
> > in College Park.
> >
> > I will freely admit that we'd certainly not design this like this if
> > we started from scratch.  But I am fairly sure we have reached a
> > result that is the least bad of all the various options we have
> > looked at over time, and that includes the nD-array decoding that
> > *is* abominable.
> >
> > In the end, how I'd frame the question is: What's the least
> > unexpected thing a VOTable parser can do when encountering characters
> > >= 0x80 in VOTable char?  I give you that in 2005, multiple answers
> > were justifiable, and the "reject" answer you gave turned out to be
> > the wisest.
> >
> > These days, and I'd be *very* surprised if this were different 20
> > years from now, it is as clear that it'll be UTF-8 as it was in 2005
> > that 7 bit characters are ASCII.  Sure, back then there still was
> > EBCDIC and other horrors.  You were wise enough to keep this
> > possibility out of VOTable.
> >
> > Today, despite all the uglyness involved, it is wise to finally give
> > these eighth bits a meaning; and utf-8 is the least surprising thing
> > we can do today, despite all the trouble that this brings with it.
> >
> > One minor remark, though:
> >
> > > ==> On 2026-06-29 à 13:02+0100,
> > >       Mark Taylor <m.b.taylor at bristol.ac.uk> wrote:
> > > >Nearly all software is written these days in an environment in which
> > > >strings are assumed Unicode, but that doesn't mean that programmers
> > > >spend their time worrying about the fact that Aries is represented
> > > >by two code points or that there is no unique way to encode something
> > > >that looks like an Omega or accented characters.  Comparison of two
> > > >UTF-8 sequences for equality *is* an easy operation, though it will
> > > >not necessarily yield true for two strings whose pixel rendering is
> > > >identical.
> > > >
> > >
> > > Sorry to disagree with the "transparency" of Unicode in the various
> > > languages: for instance length("άβγ♈︎♉︎😊𝕏") gives 12 in Javascript
> > > or Java, while Python3 or awk return 10, which is the correct number of
> > > Unicode code-points in this string(*). Similarly extracting a substring
> >
> > Ummmmm... that's a function of the *VOTable* library: Will it do the
> > UTF-8 de- and encoding for you or won't it?  I expect most will, and
> > then any sensible len-type function will be the number of unicode
> > codepoints, independently of the language.
> >
> > When a VOTable library doesn't do the decoding, the length will be
> > the number of bytes in front of the first 0x00 or arraysize,
> > whichever is smaller -- again, this is a function of the library and
> > not of the language.
> >
> > That the language objects you get from a parse function depend on the
> > library is nothing new; it's been this way with xtypes for a long,
> > long time (example: xtype="timestamp": Do you get a string or some
> > structured time representation?)
> >
> > So, I'm fairly confident that we won't see major desasters; a certain
> > problem can be very old VOTables that use some legacy 8-bit encoding
> > -- but thanks to the no-eighth-bit rule that's been in place so far
> > these have been broken all the time, and well-behaved VOTable
> > libraries will have displayed question marks -- which they will, in
> > all likelihood, still do.
> >
> > Thanks,
> >
> >             Markus
> >
> >
> 

--
Mark Taylor  Astronomical Programmer  Physics, Bristol University, UK
m.b.taylor at bristol.ac.uk          https://www.star.bristol.ac.uk/mbt/