XMM-Newton clarification wrt: Re: OAI as VO Harvesting Interface

[posuna at iso.vilspa.esa.es]

Dear Clive, dear Registry members,


although I have to admit I didn't have time to read in detail all the
mails that have been circulating these days, I found this one from Clive
where the XMM-Newton archive at VILSPA was mentioned, and I would like
to comment a couple of points without entering too much in detail.


> The public XMM-Newton archive (at Vilspa) makes metadata accessible
> down to at least the 4th level of the hierarchy, but to retrieve data
> (I think) you have to get it chunked to the observation level.  That's
> partly because the archive is run on top of a relational DBMS (Oracle)
> which encourages flattening.  Our internal XMM database is deeper, as
> it's run on an object-oriented DBMS, which encourages more baroque
> structures.


The public (and proprietary data) XMM-Newton Science Data archive at
VILSPA that can be found at http://xmm.vilspa.esa.es/xsa has got a
powerful java User Interface that allows for searches on many XMM
related parameters. 

Products can be downloaded from this interface in "Observation chunks",
as Clive says, but also in "Exposure" chunks, and as well in what is
called "Custom" chunks. These latest chunks define what the
XMM-Newton Archive Scientist has decided to be the smallest package of
data that makes sense as an isolated download. 

However, there is another way to access the XMM-Newton Science
Archive data which is through the Archive Interoperability system (whose
SIAP compatible version we released recently and announced at the
interop mailing list) that can be found at:

http://xsa.vilspa.esa.es:8080/aio/doc/

You can see in the AIO Users Manual that you can download any type of
products, including individual files, from this service. 

This AIO system is making use of our XMM-Newton Archive Three-tier
architecture which accesses our RDBMS (Sybase, by the way) through a
proper Bussiness Logic that isolates the database access from any form
of external presentation, allowing for changes in the DB without
affecting other parts of the system. 

I do not believe that Relational DBMS favour flattening and that Object
Oriented DBMs don't; any flattening on querying a database system comes
normally from a wrong Data Model or request system, but not from the
database system itself. Our AIO package allows request up to the finest
grain possible (a product file) in our project.



Cheers,

Pedro Osuna.


On Fri, 12 Sep 2003 09:06:31 +0100 (BST)
Clive Page <cgp at star.le.ac.uk> wrote:

> On Wed, 10 Sep 2003, Ray Plante wrote:
> 
> > Could you clarify, perhaps with an example, the nature of the
> > hierarchical structure that needs to be captured?  (There are
> > different possible types of hierarchies, and I want to make sure
> > we're thinking about the same one.)
> 
> I can think of 2 examples (and thought I'd posted one, if these aren't
> clear let me know):
> 
> Archive of reduced/published such as CDS
> 
> CDS has 3 main services: Simbad, Aladin, Vizier
>   Vizier has ~10,000 catalogues
>     A typical catalogue has ~10 columns
>       Each column has attributes: name, datatype, units, description
> 
> An observational archive such as XMM-Newton or HST:
> 
> XMM-Newton archive contains a few thousand observations
>   Observation (pointing period) has data on 3 instruments
>     X-ray camara data divided into exposures (period with same config)
>       Each exposure has data from 3 cameras (MOS1, MOS2, PN)
>         Camera data divided into data types (event-list, image,
>         cal...)
> 
> In practice there's some flattening of these essentially hierarchical
> structures, Vizier has a flat list of ~10k tables, but they are
> grouped by waveband and in other ways, which is useful for the user,
> so there _could_ be at least one more level than is actually the case
> at present.
> 
> The public XMM-Newton archive (at Vilspa) makes metadata accessible
> down to at least the 4th level of the hierarchy, but to retrieve data
> (I think) you have to get it chunked to the observation level.  That's
> partly because the archive is run on top of a relational DBMS (Oracle)
> which encourages flattening.  Our internal XMM database is deeper, as
> it's run on an object-oriented DBMS, which encourages more baroque
> structures.
> 
> > I think the way to look at this is from a query perspective.  A
> > query to a registry--be it a complex search or just a simple
> > harvesting request--is going to return a bunch of matched things. 
> > The question is, what should those things be?
> 
> I think there are likely to be several types of query, more or less
> corresponding to different levels of these structures:
> 
> (a) Can you find me a nearby mirror of Vizier?
> (b) Which catalogues does Vizier have giving radio polarisation?
> (c) Which XMM exposures when source X was in the field have an
> exposure
>     time over 10 kiloseconds?
> 
> And I'm sure you can think of more.  A totally flat registry _could_
> perhaps just about handle all of these, but I'm doubtful.  Or perhaps
> some of these queries would be beyond what a registry would be able to
> answer and some or all of the query would have to be forwarded to the
> archive itself.
> 
> Regards
> 
> -- 
> Clive Page
> Dept of Physics & Astronomy,
> University of Leicester,    Tel +44 116 252 3551
> Leicester, LE1 7RH,  U.K.   Fax +44 116 252 3311
> 


-- 

Pedro Osuna Alcalaya


SOFTWARE Development Group
XMM-Newton Science Archive				
e-mail: Pedro.Osuna at esa.int
Tel + 34 91 8131314  				

European Space Agency
VILLAFRANCA Satellites Tracking Station
P.O. Box 50727 
E-28080 Villafranca del Castillo
MADRID - SPAIN