Marking up HEALPix index columns in VOTable

[Mark Taylor]

Thomas,

the primary case we're thinking about here is the results of TAP queries.
For instance, consider the following ADQL, which uses a custom
User-Defined Function implemented by Markus's TAP service:

    SELECT ivo_healpix_index(raj2000, dej2000, 8) AS hpx8, COUNT(*) as nsrc
    FROM wise.main
    GROUP BY hpx8
    ORDER BY hpx8

This generates a histogram binned by healpix index, or to put it
another way an all-sky density map.  Since it's TAP, the result
is a table, typically transferred in VOTable format.  The question is,
how does the TAP service mark up the column resulting from
the ivo_healpix_index UDF so that clients know what they're getting
and can treat them accordingly, e.g. when attempting to make a
healpix map plot?

More generally, I'm adding various bits of healpix-related
functionality to topcat/stilts (density map construction and
visualisation).  This kind of data set really does have the
form of a table, so it seems like you should be able to serialize
it with appropriate metadata in VOTable form.  I admit it also
makes sense to think of some such datasets as image-like in
some circumstances too.

Mark

On Mon, 8 Aug 2016, Thomas Boch wrote:

> Hi Mark,
> 
> what's the motivation to have HEALPix maps in VOTable ?
> Not to say I'm against the idea, but I'd like to understand better the use
> case first. If had to output HEALPix maps, I'll go with the FITS serialization
> as binary tables (and associated keywords ORDERING, COORDSYS, NSIDE) which can
> be produced/is understood by the HEALPix library and the tools using it.
> 
> Cheers,
> Thomas
> 
> Le 08/08/2016 à 18:55, Mark Taylor a écrit :
> > [I've crossposted this to apps and semantics, but I suggest
> > followups to Apps only]
> > 
> > Dear colleagues,
> > 
> > Gregory Mantelet, Markus Demleitner and I have been having some
> > discussions about exchange of HEALPix maps within the VO.
> > 
> > The type of data we're considering is a table for which one column
> > contains a HEALPix index (an integer which identifies a tile on the sky),
> > and other columns contain quantities relating to to that tile
> > (source density, extinction, mean G magnitude, whatever).
> > The question we're trying to answer is: how does a data producer
> > mark up the healpix-index column in a VOTable so that consumers
> > can recognise it as such?
> > 
> > The following UCD1+ is defined:
> > 
> >      Q  |  pos.healpix      |  Hierarchical Equal Area IsoLatitude
> > Pixelization
> > 
> > So far so good; however, marking up a column like this provides
> > insufficient information to make it useful, because to make sense
> > of a HEALPix index you need to know what Order N of tiling it represents
> > (HEALPix splits the sky into 12*4^N tiles, where N is an integer
> > that for practical purposes falls in the range 0..29).
> > Since UCD words cannot be parameterised (beyond composing them with
> > other UCD words) there doesn't seem to be any way to encode this
> > Order information in the UCD.
> > 
> > Markus's clever idea is to do something like this:
> > 
> >       <FIELD datatype="int" ucd="pos.healpix">
> >         <VALUES type="legal">
> >           <MAX>49151</MAX>
> >         </VALUES>
> >       </FIELD>
> > 
> > Since the legal maximum value for the column is 49151 (=12*(4^6)-1),
> > it's possible to infer that N=6 for this healpix column.  Note that
> > since type="legal" this does not say that the actual contents of
> > the column hit that maximum (i.e. some pixels might be absent, but the
> > declaration would not change).
> > 
> > This is kind of horrible, but at the same time the meaning is fairly
> > unambiguous, and it requires no new standardisation activity.
> > So, we are likely to go ahead with marking up HEALPix columns in
> > this way (Markus's and Gregory's services will emit VOTables like
> > this, and topcat will understand them) unless anybody persuades
> > us different.
> > 
> > We have considered various other options too, but on balance we
> > think they're even worse.  Briefly:
> > 
> >     - Abuse UCDs by trying to parameterise them:
> >          pos.healpix(6)?
> >          pos.healpix;index=6?
> > 
> >     - Introduce new UCD words to (0<=N<=29, so the number of new words
> >       would be quite, but not ridiculously, large):
> >          pos.healpix0, pos.healpix1, ... pos.healpix29?
> >          pos.healpix;arith.0, pos.healpix;arith.1, ... pos.healpix;arith.29
> > 
> >     - Xtype magic:
> >          xtype="healpix(6)"
> >          xtype="tilings:healpix(6)"
> > 
> >     - Some table-level indication of the order:
> >          <TABLE>
> >            <PARAM name="healpix-order" datatype="int" value="6">
> >            <FIELD datatype=int ucd="pos.healpix"/>
> >              ...
> >          </TABLE>
> >        this would work because in practice, it's hard to imagine you
> >        could have multiple columns in a single table with different
> >        healpix orders.  This is somewhat analogous to how the
> >        ad-hoc FITS healpix serialization works.
> >        Note it still needs some convention to identify the healpix-order
> >        PARAM.
> > 
> >     - Some business with GROUP/FIELDref/PARAM elements at table level.
> >       Note this also needs some convention to identify the healpix-order.
> > 
> >     - Reinvent UCD syntax????
> > 
> > 
> > Actually it is even worse than this: there are two alternative
> > conventional schemes for assigning HEALPix indices to a tiling
> > at a given order, known as NESTED and RING, and you need to know
> > which one is intended.  Furthermore, the actual sky position
> > corresponding to a given pixel is dependent on how the tiling is
> > attached to the celestial sphere (e.g. equatorial, ecliptic, galactic).
> > So even if you know N you don't have enough explicit information
> > to make sense of a healpix index.  However, we are currently
> > following MOC in assuming the NESTED scheme, and it's probably
> > reasonable to take the sky system as implicit, so we're ignoring
> > those details for now.
> > 
> > Any bright ideas, wise comments etc welcome.
> > 
> > Mark
> > 
> > --
> > Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> > m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
> 

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/