Observation data model comments

[Anita Richards]

On Sat, 8 May 2004, Alberto Micol wrote:

> > e.g. spatial coverage in objects/deg^2
>
> I would say that objects/deg^2 does not belong to "spatial coverage", it's
> instead a way to characterise the observation.
I don't quite understand what you mean - what class should objects/deg^2
be under? (in either case it is under the Characterisation superclass!)

This is what I meant: In the discussions about the Registry and Resource
metadata, we discussed what spatial coverage and filling factor meant for
observed data (images etc.) and for extracted data - source caalogues etc.

For images, the spatial coverage outer bounds are something like a box in
RA and Dec.  This may be subdivided into a few areas described here by
Support. These two classes would probably be the same for extracted source
catgalogues.

Within that, the coverage may again be sparse, for example a multi-beam
survey which only covers certain strips, but rather than describe that in
complicated detail I think that the filling factor for images describes
how much of the area within each support is, on average, observed.  Or it
could be that if you had dithered over the sky with several partly
overlapping WFPC2 pointings, the resulting image fits within a square but
is too complicated to describe exactly, so you ahve covered 80% of the
support.

However for extracted source catalogues, if the sources are close to
point-like, then the most useful number for the user is the surface
density of sources (e.g. if selecting among catalogues which are otherwise
similar, you might want to search the one with the best fractional
coverage first).

Your comment has made me realise that there is another case, which is
catalogues of sources which are extended e.g. of HII regions or CO, in
which case maybe the relevant filling factor is the fraction of the image
with emission above the lower bound in flux density.  But most catalogues
don't make this obvious - at best you get the dimensions of a 2D Gaussian
component fitted to the source or something.

> ---
> So, the observation is the same, it's the version that changes.
> (Hence, the requirement is that a VO identifier must be able to handle
> versioning.)
I will check if that works in the situations I am thinking of.

> You seem to separate Processing from Provenance. I would have tought that
> Processing is part of Provenance. Suppose that you want to describe a product
> which was generated by combining other products coming even from different
> instruments; in such case I would say that Processing and Provenance are very
> much intertwined.
>

Maybe that is before I put up my latest plot.  I have now put Processing
back as a part of Provenance but I am still a bit concerned as different
processing methods will give data with different characterisation e.g.
different synthesised beam size - for interferometry data you can
re-weight the visibilities (as many times as you like) to give images with
lower spatial resolution but greater sensitivity to extended flux (greater
Max spatial scale) and lower noise rms - or v.v., higher spatial
resolution but also higher noise and you lose extended flux of low surface
brightness.  So I am still a bit concerned whether in practice that can
all be taken care of with versioning.

> ---
>
> Maybe a twiki problem ...
> > The gray boxes show classes which exchange information with other models,
> I can't see gray boxes, they are all white on my netscape/linux.
> I will try with macosx/safari ...
Sorry, next time I will make them darker.  They are the ones which have
lower-case prefixes (registry, quantity, community)
They are OK with linux/mozilla (although the grey is similar to the blue
of Characterisation on the screen, but not in hard copy).

> ---
>
> In the last table:
>
> Bounds: for Flux more than the noise rms I would use the limiting flux
> for the lower limit, and I would add the saturation level for the upper limit.
That is the difference between models for different situations, in radio
interferometry the limiting flux is partly dependent on how you process
the data and is normally taken as 3sigma noise (or more if noise is
non-Gaussian) measured from the off-source image or region of the spectral
or visibility domain.  Similarly there is not usually a saturation upper
limit (unless you are observing the Sun or very bad rfi or something else
which makes a receiver go non-linear).  There is a limit on the
signal-to-noise ratio which you can achieve.  However in Bounds I put
quantities which are fixed limits for any one data product

> Support: for Flux I would use the min and max fluxes in the data.

That is a good idea as long as it is only applied where relevant. In the
case of radio data, that is not always obvious or useful, especialy the
min.  which is some possibly negative noise level for total intensity
images, whilst for Stokes Q it can be a meaningful negative value.  If you
want to know what is the minimum surface brightness object which can be
found in total intensity in the data then some multiple of the noise rms
is the best way.
Max is rather specialised for visibility data, for example in the combined
visibilities plotted v. baseline length the max tells you what the total
flux density within the field of view is on the max spatial scale to which
the interferometer is sensitive.  Radio models have to understand this but
I don't think that it will do anything other than confuse most VO tools or
users. However  for total intensity images the max (in Jy/beam or
Jy/object) could be useful, and for polarisation or spectral absorption
images the min is too - thanks for suggesting that.

However that reminds me of another issue: most radio images are measured
in Jy/beam (or mJy/beam etc.) but none of the VO tools I ahve used can
interpret that.  As I hope I ahve made clear, the beam size is roughly
determined by the provenance but within a factor of 2 it is a property
of variable processing - noted as Restoring Beam under Resolution under
Characterisation.  The pixel size of an image is arbitrarily chosen (there
are sensible limits but it is not a direct function of resolution).
If you use e.g. AVO-Aladin or SExtractor to 'measure' the flux at a
particular pixel the flux density is actually in Jy/beam, but the tools
think that it is in Jy/pixel or in some cases Jy/arcsec^2.

I think that the most important development for VOs to handle radio data
properly is to figure out how to extract the beam size from (often
obscure) places in FITS headers for radio images and use the information
intelligently.  This could be by applying a linear multiplication factor
to convert to Jy/pixel or Jy/arcsec^2 on the fly when required, e.g. for
comparison with non-radio data (it is much easier than converting
magnitudes to physical units!).  However results from source exctraction
should ideally be in both units as radio astronomers will expect Jy/beam.
>
> Sensitivity: for Temporal I would say "exposure map"
This is hard to generalise for radio interferometry.  To make a good image
this is the synthesis time, that is, the time it takes for the earth to
tur to provide enough baseline coverage.  For a sparse or linear array
this is ideally 12 hr; for the VLA in a compact configuration it can be as
little as 15 mins.  However if you are observing a variable source (or a
source with a variable core) you can extract a light curve from the
visibilities with a time resolution limited only by sensitivity, a single
scan (typically 5-10 mins) in some cases, could be an hour or more.
Occasionally a single integration (which corresponds to your exposure?)
may be useful, but that is rare.

>
> Sample precision: for Spatial -> pixel scale,
As I've explained, if interferometry image data have been conventionally
reduced, the pixel is a rough indicator of sample precision, but it isn't
an immutable property of the detector.  For extracted source catalogues
you can get to much higher point source position precision depending on
the signal-to-noise; for the fidelity of imaging faint extended sources it
is the natural synthesised beam.

> for Flux -> readout noise for CCD  data.
I guess it is image(etc) noise again for interferometry.

As an aside, there is a whole additional set of jargon for single dish
observations, where you may need to know how to convert Tant (antenna
temperature) into non-instrument-dependent units (using Tsys, efficiency
and sensitivity) etc. etc.

The main lesson to me from this, is that I think we will never end up with
one table which covers every domain and even if we did the words chosen
would mean different things to different people, causing confusion.  This
excercise (which we should indeed repeat for other domains, such as your
HST expertise, x-ray etc.) should just tell us if the column and row
headings of the matrix are usable, and then establish a translation
mechanism at the most convenient point.  For example, the VO needs to know
SensitivityFunction.Spatial and an interferometry data centre can provide
a usable form of that function (initially a pair e.g. 50% @ 5', or even
just 5'), whilst a small single-dish data provider might require the VO to
calculate the simple expression lambda/diameter from the FITS header.

---
There is another confusing issue in the Characterisation table.  The Flux
entry for Bounds includes limiting flux, which I presume is the 3sigma (or
whatever) noise level.  The Flux entry for Sensitivity is linearity.  That
is perfectly internally logical.... but most astronomers are used to
thinking of the limiting flux as the sensitivity of their instrument.

I suugest renaming either or both classes:
Bounds > SensitivityBounds
Sensitivity > SensitivityFunction
- what do you think?

>
> I love the fact that now the Radio domain is represented!
> (That's a challenge, though, since I need to understand/think radio now ...)

Thanks for your comments, it is very useful to discuss this.  But I am not
expecting the model to be modified to cover every interferometry quirk,
the main thing is to start from the tools and science cases which we will
work with in the next year and identify what information is needed for
those.

all the best
a


- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Dr. Anita M. S. Richards, AVO Astronomer
MERLIN/VLBI National Facility, University of Manchester,
Jodrell Bank Observatory, Macclesfield, Cheshire SK11 9DL, U.K.
tel +44 (0)1477 572683 (direct); 571321 (switchboard); 571618 (fax).