Questions about UTF-8 in VOTable

[Francois Ochsenbein]

Dear Markus,

Thank you for the historical explanations but still the question of *how
to specify that a <FIELD> is made of ascii characters only* has no answer
in the current VOTable draft… But apparently nobody cares ?
Cheers, François

Le lun. 6 juil. 2026 à 14:48, Markus Demleitner via apps <apps at ivoa.net> a
écrit :

> Dear François,
>
> On Thu, Jul 02, 2026 at 08:51:43PM +0200, Francois Ochsenbein via apps
> wrote:
> > that removing the possibility of specifying that a non-numerical column
> > is made of ascii-only characters does not raise more comments or
> > concerns?… A few more comments embedded below:
>
> Oh, don't worry: There has been plenty of discussion.  I've been
> trying to *somehow* improve the unsatisfying VOTable char situation
> since at least 2012, and these attempts got stuck numerous times.
> Mark's valiant attempt is the result of a dedicated plenary (IIRC)
> in College Park.
>
> I will freely admit that we'd certainly not design this like this if
> we started from scratch.  But I am fairly sure we have reached a
> result that is the least bad of all the various options we have
> looked at over time, and that includes the nD-array decoding that
> *is* abominable.
>
> In the end, how I'd frame the question is: What's the least
> unexpected thing a VOTable parser can do when encountering characters
> >= 0x80 in VOTable char?  I give you that in 2005, multiple answers
> were justifiable, and the "reject" answer you gave turned out to be
> the wisest.
>
> These days, and I'd be *very* surprised if this were different 20
> years from now, it is as clear that it'll be UTF-8 as it was in 2005
> that 7 bit characters are ASCII.  Sure, back then there still was
> EBCDIC and other horrors.  You were wise enough to keep this
> possibility out of VOTable.
>
> Today, despite all the uglyness involved, it is wise to finally give
> these eighth bits a meaning; and utf-8 is the least surprising thing
> we can do today, despite all the trouble that this brings with it.
>
> One minor remark, though:
>
> > ==> On 2026-06-29 à 13:02+0100,
> >       Mark Taylor <m.b.taylor at bristol.ac.uk> wrote:
> > >Nearly all software is written these days in an environment in which
> > >strings are assumed Unicode, but that doesn't mean that programmers
> > >spend their time worrying about the fact that Aries is represented
> > >by two code points or that there is no unique way to encode something
> > >that looks like an Omega or accented characters.  Comparison of two
> > >UTF-8 sequences for equality *is* an easy operation, though it will
> > >not necessarily yield true for two strings whose pixel rendering is
> > >identical.
> > >
> >
> > Sorry to disagree with the "transparency" of Unicode in the various
> > languages: for instance length("άβγ♈︎♉︎😊𝕏") gives 12 in Javascript
> > or Java, while Python3 or awk return 10, which is the correct number of
> > Unicode code-points in this string(*). Similarly extracting a substring
>
> Ummmmm... that's a function of the *VOTable* library: Will it do the
> UTF-8 de- and encoding for you or won't it?  I expect most will, and
> then any sensible len-type function will be the number of unicode
> codepoints, independently of the language.
>
> When a VOTable library doesn't do the decoding, the length will be
> the number of bytes in front of the first 0x00 or arraysize,
> whichever is smaller -- again, this is a function of the library and
> not of the language.
>
> That the language objects you get from a parse function depend on the
> library is nothing new; it's been this way with xtypes for a long,
> long time (example: xtype="timestamp": Do you get a string or some
> structured time representation?)
>
> So, I'm fairly confident that we won't see major desasters; a certain
> problem can be very old VOTables that use some legacy 8-bit encoding
> -- but thanks to the no-eighth-bit rule that's been in place so far
> these have been broken all the time, and well-behaved VOTable
> libraries will have displayed question marks -- which they will, in
> all likelihood, still do.
>
> Thanks,
>
>             Markus
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/apps/attachments/20260711/16810f91/attachment.htm>