From M.B.Taylor at bristol.ac.uk  Mon Aug  8 18:55:31 2016
From: M.B.Taylor at bristol.ac.uk (Mark Taylor)
Date: Mon, 8 Aug 2016 17:55:31 +0100 (BST)
Subject: Marking up HEALPix index columns in VOTable
Message-ID: <alpine.LRH.2.20.1608081750110.29860@andromeda.star.bris.ac.uk>

[I've crossposted this to apps and semantics, but I suggest
followups to Apps only]

Dear colleagues,

Gregory Mantelet, Markus Demleitner and I have been having some
discussions about exchange of HEALPix maps within the VO.

The type of data we're considering is a table for which one column
contains a HEALPix index (an integer which identifies a tile on the sky),
and other columns contain quantities relating to to that tile
(source density, extinction, mean G magnitude, whatever).
The question we're trying to answer is: how does a data producer
mark up the healpix-index column in a VOTable so that consumers
can recognise it as such?

The following UCD1+ is defined:

    Q  |  pos.healpix      |  Hierarchical Equal Area IsoLatitude Pixelization

So far so good; however, marking up a column like this provides
insufficient information to make it useful, because to make sense
of a HEALPix index you need to know what Order N of tiling it represents
(HEALPix splits the sky into 12*4^N tiles, where N is an integer
that for practical purposes falls in the range 0..29).
Since UCD words cannot be parameterised (beyond composing them with
other UCD words) there doesn't seem to be any way to encode this
Order information in the UCD.

Markus's clever idea is to do something like this:

     <FIELD datatype="int" ucd="pos.healpix">
       <VALUES type="legal">
         <MAX>49151</MAX>
       </VALUES>
     </FIELD>

Since the legal maximum value for the column is 49151 (=12*(4^6)-1),
it's possible to infer that N=6 for this healpix column.  Note that
since type="legal" this does not say that the actual contents of
the column hit that maximum (i.e. some pixels might be absent, but the
declaration would not change).

This is kind of horrible, but at the same time the meaning is fairly
unambiguous, and it requires no new standardisation activity.
So, we are likely to go ahead with marking up HEALPix columns in
this way (Markus's and Gregory's services will emit VOTables like
this, and topcat will understand them) unless anybody persuades
us different.

We have considered various other options too, but on balance we
think they're even worse.  Briefly:

   - Abuse UCDs by trying to parameterise them:
        pos.healpix(6)?
        pos.healpix;index=6?

   - Introduce new UCD words to (0<=N<=29, so the number of new words
     would be quite, but not ridiculously, large):
        pos.healpix0, pos.healpix1, ... pos.healpix29?
        pos.healpix;arith.0, pos.healpix;arith.1, ... pos.healpix;arith.29

   - Xtype magic:
        xtype="healpix(6)"
        xtype="tilings:healpix(6)"

   - Some table-level indication of the order:
        <TABLE>
          <PARAM name="healpix-order" datatype="int" value="6">
          <FIELD datatype=int ucd="pos.healpix"/>
            ...
        </TABLE>
      this would work because in practice, it's hard to imagine you
      could have multiple columns in a single table with different
      healpix orders.  This is somewhat analogous to how the
      ad-hoc FITS healpix serialization works.
      Note it still needs some convention to identify the healpix-order
      PARAM.

   - Some business with GROUP/FIELDref/PARAM elements at table level.
     Note this also needs some convention to identify the healpix-order.

   - Reinvent UCD syntax????


Actually it is even worse than this: there are two alternative
conventional schemes for assigning HEALPix indices to a tiling
at a given order, known as NESTED and RING, and you need to know
which one is intended.  Furthermore, the actual sky position
corresponding to a given pixel is dependent on how the tiling is
attached to the celestial sphere (e.g. equatorial, ecliptic, galactic).
So even if you know N you don't have enough explicit information
to make sense of a healpix index.  However, we are currently
following MOC in assuming the NESTED scheme, and it's probably
reasonable to take the sky system as implicit, so we're ignoring
those details for now.

Any bright ideas, wise comments etc welcome.

Mark

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/

From stephane.erard at obspm.fr  Mon Aug  8 20:11:49 2016
From: stephane.erard at obspm.fr (=?iso-8859-1?Q?St=E9phane_Erard?=)
Date: Mon, 8 Aug 2016 20:11:49 +0200
Subject: Marking up HEALPix index columns in VOTable
In-Reply-To: <alpine.LRH.2.20.1608081750110.29860@andromeda.star.bris.ac.uk>
References: <alpine.LRH.2.20.1608081750110.29860@andromeda.star.bris.ac.uk>
Message-ID: <277E2FEB-E7E2-409F-9A04-9830522B8FA0@obspm.fr>

Dear Mark and all

I've been wondering about the same thing last spring for possible use on planetary surfaces (~spherical ones at least)
To me that was clearly a system with 2 coordinates, Nside and tile number (plus possibly ordering nested/ring as a 3rd one - it doesn't hurt in EPNCore since we always have 3 spatial coordinates available). 
Your baseline solution may be better if there is a simple way to parse it in the tools.

Regards
St?phane

Le 8 ao?t 2016 ? 18:55, Mark Taylor <M.B.Taylor at bristol.ac.uk> a ?crit :

> [I've crossposted this to apps and semantics, but I suggest
> followups to Apps only]
> 
> Dear colleagues,
> 
> Gregory Mantelet, Markus Demleitner and I have been having some
> discussions about exchange of HEALPix maps within the VO.
> 
> The type of data we're considering is a table for which one column
> contains a HEALPix index (an integer which identifies a tile on the sky),
> and other columns contain quantities relating to to that tile
> (source density, extinction, mean G magnitude, whatever).
> The question we're trying to answer is: how does a data producer
> mark up the healpix-index column in a VOTable so that consumers
> can recognise it as such?
> 
> The following UCD1+ is defined:
> 
>    Q  |  pos.healpix      |  Hierarchical Equal Area IsoLatitude Pixelization
> 
> So far so good; however, marking up a column like this provides
> insufficient information to make it useful, because to make sense
> of a HEALPix index you need to know what Order N of tiling it represents
> (HEALPix splits the sky into 12*4^N tiles, where N is an integer
> that for practical purposes falls in the range 0..29).
> Since UCD words cannot be parameterised (beyond composing them with
> other UCD words) there doesn't seem to be any way to encode this
> Order information in the UCD.
> 
> Markus's clever idea is to do something like this:
> 
>     <FIELD datatype="int" ucd="pos.healpix">
>       <VALUES type="legal">
>         <MAX>49151</MAX>
>       </VALUES>
>     </FIELD>
> 
> Since the legal maximum value for the column is 49151 (=12*(4^6)-1),
> it's possible to infer that N=6 for this healpix column.  Note that
> since type="legal" this does not say that the actual contents of
> the column hit that maximum (i.e. some pixels might be absent, but the
> declaration would not change).
> 
> This is kind of horrible, but at the same time the meaning is fairly
> unambiguous, and it requires no new standardisation activity.
> So, we are likely to go ahead with marking up HEALPix columns in
> this way (Markus's and Gregory's services will emit VOTables like
> this, and topcat will understand them) unless anybody persuades
> us different.
> 
> We have considered various other options too, but on balance we
> think they're even worse.  Briefly:
> 
>   - Abuse UCDs by trying to parameterise them:
>        pos.healpix(6)?
>        pos.healpix;index=6?
> 
>   - Introduce new UCD words to (0<=N<=29, so the number of new words
>     would be quite, but not ridiculously, large):
>        pos.healpix0, pos.healpix1, ... pos.healpix29?
>        pos.healpix;arith.0, pos.healpix;arith.1, ... pos.healpix;arith.29
> 
>   - Xtype magic:
>        xtype="healpix(6)"
>        xtype="tilings:healpix(6)"
> 
>   - Some table-level indication of the order:
>        <TABLE>
>          <PARAM name="healpix-order" datatype="int" value="6">
>          <FIELD datatype=int ucd="pos.healpix"/>
>            ...
>        </TABLE>
>      this would work because in practice, it's hard to imagine you
>      could have multiple columns in a single table with different
>      healpix orders.  This is somewhat analogous to how the
>      ad-hoc FITS healpix serialization works.
>      Note it still needs some convention to identify the healpix-order
>      PARAM.
> 
>   - Some business with GROUP/FIELDref/PARAM elements at table level.
>     Note this also needs some convention to identify the healpix-order.
> 
>   - Reinvent UCD syntax????
> 
> 
> Actually it is even worse than this: there are two alternative
> conventional schemes for assigning HEALPix indices to a tiling
> at a given order, known as NESTED and RING, and you need to know
> which one is intended.  Furthermore, the actual sky position
> corresponding to a given pixel is dependent on how the tiling is
> attached to the celestial sphere (e.g. equatorial, ecliptic, galactic).
> So even if you know N you don't have enough explicit information
> to make sense of a healpix index.  However, we are currently
> following MOC in assuming the NESTED scheme, and it's probably
> reasonable to take the sky system as implicit, so we're ignoring
> those details for now.
> 
> Any bright ideas, wise comments etc welcome.
> 
> Mark
> 
> --
> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/


From hessman at astro.physik.uni-goettingen.de  Tue Aug  9 07:36:39 2016
From: hessman at astro.physik.uni-goettingen.de (Frederic V. Hessman)
Date: Tue, 9 Aug 2016 07:36:39 +0200
Subject: Marking up HEALPix index columns in VOTable
In-Reply-To: <alpine.LRH.2.20.1608081750110.29860@andromeda.star.bris.ac.uk>
References: <alpine.LRH.2.20.1608081750110.29860@andromeda.star.bris.ac.uk>
Message-ID: <A211F888-4106-4028-92EC-4D1031B4495A@astro.physik.uni-goettingen.de>

Mark,

> On 8 Aug 2016, at 18:55, Mark Taylor <m.b.taylor at bristol.ac.uk> wrote:
> 
> Gregory Mantelet, Markus Demleitner and I have been having some
> discussions about exchange of HEALPix maps within the VO.
> ...
> So far so good; however, marking up a column like this provides
> insufficient information to make it useful, because to make sense
> of a HEALPix index you need to know what Order N of tiling it represents
> (HEALPix splits the sky into 12*4^N tiles, where N is an integer
> that for practical purposes falls in the range 0..29).
> Since UCD words cannot be parameterised (beyond composing them with
> other UCD words) there doesn't seem to be any way to encode this
> Order information in the UCD.
> 
> Markus's clever idea is to do something like this:
> 
>     <FIELD datatype="int" ucd="pos.healpix">
>       <VALUES type="legal">
>         <MAX>49151</MAX>
>       </VALUES>
>     </FIELD>
> 
> Since the legal maximum value for the column is 49151 (=12*(4^6)-1),
> it's possible to infer that N=6 for this healpix column.  Note that
> since type="legal" this does not say that the actual contents of
> the column hit that maximum (i.e. some pixels might be absent, but the
> declaration would not change).
> 
> This is kind of horrible, but at the same time the meaning is fairly
> unambiguous, and it requires no new standardisation activity.
> So, we are likely to go ahead with marking up HEALPix columns in
> this way (Markus's and Gregory's services will emit VOTables like
> this, and topcat will understand them) unless anybody persuades
> us different.

Clever but beyond horrible.  Why can?t you simply

>   - Some business with GROUP/FIELDref/PARAM elements at table level.
>     Note this also needs some convention to identify the healpix-order.

Yes, but the number of metadata elements needed is very small, say

	?pos.healpix.order"	=	pos.healpix;meta.number	=	N

	?pos.healpix.tiling"	=	pos.healpix;pos.wcs.ctype	=	?NESTED? or ?RING"

The first works because the map needs some other number other than the number of tiles and the only other HEALPIX number is N.

I used pos.ctype in the 2nd in a slightly different manner from what we?re used to, but this UCD should be freed from its FITS fetters, since the meaning should be clear - the tiling scheme expresses a world coordinate system ?type?.

Rick


From msdemlei at ari.uni-heidelberg.de  Thu Aug 18 10:45:40 2016
From: msdemlei at ari.uni-heidelberg.de (Markus Demleitner)
Date: Thu, 18 Aug 2016 10:45:40 +0200
Subject: VOResource 1.1: Vocabularies, part 1
Message-ID: <20160818084540.GA13472@victor>

Dear Registry, dear Semantics,

In moving to VOResource 1.1, we have some topics touching Semantics,
so I'm cc:-ing semantics as a heads-up.  However, I'd suggest
(Registry-related) discussion to take place on the Registry list
rather than on both.

VOResource 1.0 managed some vocabulary-like term lists within the
schema.  This turned out to be a major pain on several occasions
(e.g., when we'd liked to have introduced a "uses"-relationship for
linking tools and tutorials, or when we wanted to consolidate content
levels to make them handlable); in particular with our old policy of
changing the target namespace when the schema content changes, each
vocabulary addition would have broken the entire infrastructure.

Even with our new namespace policy, a schema update is relatively
expensive, and many of the word lists can actually be used like
vocabularies, where nothing really breaks if new terms are
introduced.

Hence, for VOResource 1.1 [1] I propose to move the following elements'
content models from schema-constrained to vocabulary-constrained

 * content/contentLevel
 * content/type
 * relationship/relationshipType
 * date/@role

These are to be managed according to the rules currently developing
in the semantics WG.

Does anyone disagree with this general approach?

The individual vocabularies I'd like to discuss separately, except
for content/type -- this is just pulled straight from VOResource 1.0.
While I'd say that this piece of metadata hasn't been terribly
usefule in Registry practice it seems plausible that it might yet
become useful as the VO evolves.  And anyway, if we removed it, we'd
(at least in theory) have to bump the major version number, and that
we don't want.

But perhaps someone has good ideas on how to enhance it?

While it's still in development, the vocabulary is available 
Norman-style from
http://docs.g-vo.org/vocab-test/relationship_type/2016-08-17/relationship_type.html
-- Norman-style means that you can use the accept header to choose
between Turtle, RDF/XML, and HTML (which is the default and therefore
what your run-of-the-mill browser sees)[2].


Cheers,

         Markus


[1] In case you've just tuned in: A WG-internal draft of VOResource
1.1 is available from http://docs.g-vo.org/VOResource.pdf, or in
source from
http://volute.g-vo.org/svn/trunk/projects/registry/VOResource

[2] Yes, the HTML version isn't pretty.  In case you feel like a bit
of good ol' HTML+CSS, you'd be most welcome to improve appreances
in convert.py and friends; see
http://volute.g-vo.org/svn/trunk/projects/registry/VOResource/terms


From dower at stsci.edu  Thu Aug 18 15:53:14 2016
From: dower at stsci.edu (Theresa Dower)
Date: Thu, 18 Aug 2016 13:53:14 +0000
Subject: VOResource 1.1: Vocabularies, part 1
In-Reply-To: <20160818084540.GA13472@victor>
References: <20160818084540.GA13472@victor>
Message-ID: <FC45BCE8269EDC4E90EE818CBE3A73E233B24F32@EXCHMAIL1.stsci.edu>

From the registry end, I'm for externalizing the vocabularies for these elements, as we spoke about in Stellenbosch. I'd strongly suggest keeping a suggested/shorthand list of the original terms in the VOResource document. This is trivial to do and helps folks who aren't necessarily inclined to delve so deeply avoid necessarily looking up these extra documents and getting more overwhelmed, which already sometimes happens with the existing VOResource architecture.

--Theresa

> -----Original Message-----
> From: registry-bounces at ivoa.net [mailto:registry-bounces at ivoa.net] On Behalf
> Of Markus Demleitner
> Sent: Thursday, August 18, 2016 4:46 AM
> To: registry at ivoa.net
> Cc: semantics at ivoa.net
> Subject: VOResource 1.1: Vocabularies, part 1
> 
> Dear Registry, dear Semantics,
> 
> In moving to VOResource 1.1, we have some topics touching Semantics, so I'm
> cc:-ing semantics as a heads-up.  However, I'd suggest
> (Registry-related) discussion to take place on the Registry list rather than on
> both.
> 
> VOResource 1.0 managed some vocabulary-like term lists within the schema.
> This turned out to be a major pain on several occasions (e.g., when we'd liked to
> have introduced a "uses"-relationship for linking tools and tutorials, or when we
> wanted to consolidate content levels to make them handlable); in particular with
> our old policy of changing the target namespace when the schema content
> changes, each vocabulary addition would have broken the entire infrastructure.
> 
> Even with our new namespace policy, a schema update is relatively expensive,
> and many of the word lists can actually be used like vocabularies, where nothing
> really breaks if new terms are introduced.
> 
> Hence, for VOResource 1.1 [1] I propose to move the following elements'
> content models from schema-constrained to vocabulary-constrained
> 
>  * content/contentLevel
>  * content/type
>  * relationship/relationshipType
>  * date/@role
> 
> These are to be managed according to the rules currently developing in the
> semantics WG.
> 
> Does anyone disagree with this general approach?
> 
> The individual vocabularies I'd like to discuss separately, except for content/type
> -- this is just pulled straight from VOResource 1.0.
> While I'd say that this piece of metadata hasn't been terribly usefule in Registry
> practice it seems plausible that it might yet become useful as the VO evolves.
> And anyway, if we removed it, we'd (at least in theory) have to bump the major
> version number, and that we don't want.
> 
> But perhaps someone has good ideas on how to enhance it?
> 
> While it's still in development, the vocabulary is available Norman-style from
> http://docs.g-vo.org/vocab-test/relationship_type/2016-08-
> 17/relationship_type.html
> -- Norman-style means that you can use the accept header to choose between
> Turtle, RDF/XML, and HTML (which is the default and therefore what your run-
> of-the-mill browser sees)[2].
> 
> 
> Cheers,
> 
>          Markus
> 
> 
> [1] In case you've just tuned in: A WG-internal draft of VOResource
> 1.1 is available from http://docs.g-vo.org/VOResource.pdf, or in
> source from
> http://volute.g-vo.org/svn/trunk/projects/registry/VOResource
> 
> [2] Yes, the HTML version isn't pretty.  In case you feel like a bit
> of good ol' HTML+CSS, you'd be most welcome to improve appreances
> in convert.py and friends; see
> http://volute.g-vo.org/svn/trunk/projects/registry/VOResource/terms