Data set metadata schemas

[Anita Richards]

I have learnt from the debate on Registry Schema but I can't add much
except to say that I think the differences will only be resolved in
use.  We should start applying the schema we have got to real data
sets and science queries (even if these have to be carefully selected
at first) - and that seems to mean starting from
http://www.ivoa.net/internal/IVOA/IvoaResReg/ResourceServiceMetadataV7.pdf
(RSMV7). So to that extent I agree with Bob, and maybe that is not
controversial as I think I am only talking about what Tony says it is
OK for, i.e. "Sky coverage services" if that is taken to include
spatial spectral and temporal coverage and other metadata tied to data
sets. That is, what information does the Registry need to hold to
select potentially useful data sets and use their metadata to select
the appropriate services to then access the data for processing (but I
am not commenting on the parts of the registry which describe
appropriate services).

I would like to understand better how feasible it is to link sections,
for example an entry in COMMUNITY may be the same as one for
Contributor in CURATION.

We also need to think a bit more about how to aquire the dataset
metadata. At least at first, we want to make sure this is done in an
examplary fashion because we will be judged by the results, so it is
no good using difficulty in getting information as an excuse.  In my
experience with the 4 data sets so far, the relevant information is
not held in one place, it requires human searching of web-sites and
human discrimination, for example to decide what is the region of
regard for a catalogue - PSF (but what about systematic errors)? Pixel
size (but this is arbitrary in radio images)? Largest error given (may
be spurious/huge)? Eventually we will have algorithms to help decide
but these will be evolved through experience, not trying to imagine
all possible circumstances.  For data sets which are actively curated,
we can ask someone to fill in a questionaire, however, again, we will
only discover what is open to misinterpretation after a few rounds
with archivists.  More seriously, we do not yet have the kudos to get
people to fill them in unless they are already VO enthusiasts and even
then they often just point you at web sites with (usually) far too
much detail.

However, I suggest that we start by designing a plain text form which
gives examples/selections where appropriate.  This could be
interpreted and written to xml using a perl script, which would also
catch the commonest ambiguities (metre/meter) and unit conversions.
We can progress to a web form as long as it is really
platform-independent and avoids problems with over-long selection
lists, instability if completion is interrupted etc.  We are going to
have to solve this problem anyway of course for user input! The
protocols for submitting data sets to CDS are one precedent, I would
welcome comments from people involved with that.

The AstroGrid Registry work-group have created a set of Resource
Registry schemas for AstroGrid.  These are based on RSMV7 with a few
additions suggested by trying to use them to describe four real
datasets.  I apologise for the baby xml, I am trying to learn - all
mistakes are my responsibility alone. I also apologise for possibly
reinventing (but less adequately) the schemas linked to
http://www.ivoa.net/internal/IVOA/IVOARegWp03/MDinXML-Summary.html -
however I think I am covering a small part of this in more detail.  I
also note that these are based on RSMV6 which explaiins some of the
differences in organisation.

 You can find my schemas for AstroGrid at
http://wiki.astrogrid.org/bin/view/Astrogrid/RegistryIt02Schema - see
a little way down the page:

------------------------------
------------------------------

"Iteration 2 resource registry schema

...
resourceRegistry.xsd and the component schemas for describing an
astronomical/solar/STP resource: identity.xsd, curation.xsd,
content.xsd, service.xsd."
...

"Examples of the identity, curation and content xml files (in a single
file) have been prepared for the 1XMM (x-ray sattelite), SURF (Solar),
USNO-B (reference stars), WFCSUR (Isaac Newton Telescope survey)
archives."

and
http://wiki.astrogrid.org/bin/view/Astrogrid/RegistryUnits
which explains where/why I have added to RSMV7.  In summary, the
differences are:

CURATION

1) I have added some elements to describe the size of data sets - in
   Mb, and for tabular data nRows/nCols, or nPixels for 'image' data
   (extensible to any number of dimensions).  This is to aid
   optimising the order of query execution and in case servers have
   limits on the size of data which can be returned/need to invoke a
   cutout server for images etc.

CONTENT

2) Added element for UCDs - this will be for dumb matching at first,
   can become more sophisticated or moved to a different level as UCDs
   become more sophisticated.

3) Added spatial region Healpix - this is the CMB way of indexing the
   sky, added at the request of the Planck people.  At the coarsest
   there are 12 regions.

4) In a future iteration we should extend region of regard to the
   spectral and temporal regimes.  NB I don't think this is the same
   as resolution in most cases; for source lists the error may be
   greater than the resolution (e.g. systematic errors due to
   reference source position uncertainty) or less (point source at
   good signal-to-noise); for images the spatial size of a single
   image is the same as the resolution for e.g. 1D radio spectra, but
   not for a radio synthesis or a CCD image.

5) Added UNKNOWN to cframe types, spectral waveband coverage, might
   want this elsewhere as well.  At present this is mainly because I
   do not know how to deal with solar data but it might be a useful
   general distinction between 'exists but unknown' v. 'NULL'.

6) In a future iteration, add after object count coverage etc., the
   spatial fraction of the BOX etc. covered by images, and similarly
   for spectral and temporal coverage.

7) Added Resolution (spatial spectral temporal)

8) Added Data Quality (spatial spectral temporal)

Other future additions

   * Allow coverage to include multiple non-contiguous regions in
     spatial spectral and temporal domains, e.g. to allow for discrete
     radio wavebands covering 1.3-1.7 GHz, 4.5-6.7, 21-24 GHz etc.
     These are not bandpasses because any individual observation
     probably only covers a smaller region e.g. 16 MHz within this.
     Similarly for optical observations of variable stars which can
     only be observed when they are in the night sky, etc. This is a
     more spohisticated variant on 6) above.

   * Allow element values to be inter-dependent, e.g. the radio
     resolution depends on wavelength and in the above example varies
     by a factor of almost 20.

   * We discussed adding (probably to CURATION) a set of elements to
     cover linked data sets, for example if the properties of observed
     sources and the coverage of the facility used are in separate
     tables. However this may be covered by the separate Data
     Collection section? Or is this more like an entire data centre
     e.g. MAST, LEDAS?

SERVICE

9) Added maximum image size allowed by service to the restrictions.
   There could be more, e.g. maximum time interval to search etc.

best wishes

Anita

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Dr. Anita M. S. Richards, AVO Astronomer
MERLIN/VLBI National Facility, University of Manchester,
Jodrell Bank Observatory, Macclesfield, Cheshire SK11 9DL, U.K.
tel +44 (0)1477 572683 (direct); 571321 (switchboard); 571618 (fax).