true XML

[Ed Shaya]

Guy Rixon wrote:

>Ed Shaya wrote:
>
>  
>
>>OK.  Now the crux of this all is that the Tabular form of this
>>is done by a auto-flattener that simply turns this into a
>>series of tables exactly as an XML to DB converter does.
>>Now you have a compact form to transfer and one that can go
>>right into your local/private DB.  If it is an object
>>relational DB you should be able to access the data with the
>>original true XML.
>>    
>>
>
>Interesting.  I have a few questions:
>
>Where do we get the "auto-flattener"?  Is it some generic tool, or does IVOA
>have to develop it?
>
>  
>
At the metadata WG I was asked that and I said look at XMLDB.  But, I 
now realize that the
problem is really quite easy solved by simple transformation (of the 
XSLT type or an equivalent).
Our knowledge is describable by the following simple serialization of a 
general ontology:

<!ELEMENT Objects (Object+)> <!-- Root of Objects.dtd -->
<!ELEMENT Object (SimpleQuantity*,QuantitywithMetadata*,
                                                
ArrayQuantities,QuantityWithObjects*)
<!ATTLIST Object
             type    CDATA>  <!-- specify type of Object (star, galaxy, 
table, datacenter) -->
             name CDATA>

<!-- Just a property of the object with a value, error, and units -->
<!ELEMENT SimpleQuantity (Value,Error, Units)>
 <!ATTLIST  SimpleQuantity name CDATA>

<! --Like the simple one but add some metadata in a hierarchy -->
<!ELEMENT QuantitywithMetadata (Value, Error, Units, Metadata)
  <!ATTLIST QuantitywithMetadata name CDATA>
<!ELEMENT Metadata (Quantity+)>

<!ELEMENT ArrayQuantity (image*, spectra*, timeSeries*) >
<!-- We'll just hold these in FITS for now, but of course we could 
construct separate Quantity tables for them -->
<!ELEMENT image (FITSFile)>
<!ELEMENT FITSFile EMPTY>
  <!ATTLIST FITSFile href URL>
<!ELEMENT spectra (FITSFile)>
<!ELEMENT timeSeries (FITSFile)>
<!ELEMENT image EMPTY>
 <!ATTLIST image href URL>

<!-- Now we have a Quantity which holds memberOf, as in galaxies are 
members of clusters -->
<!ELEMENT QuantityWithObjects (Object*)>
<!ATTLIST type CDATA>

Done, more or less.   I am thinking this is actually too general, and 
the VO.xsd will be a restriction of this. 
Now we want to flatten this. 

At the top level, take all Objects of the same type and put them into a 
table,
do the same for all other types. 

Be sure to create a master table that has two columns (ObjectType, 
tablePointer).

Put Object/@name into first column.
Each SimpleQuantity goes into three columns or so, value, error, units.
     If all the  units are the same, then can collapse that column into 
virtual column. 
Do the same with QuantityWithMetadata but now an additional column is 
created and the Metadata is placed there whole as either an xs:anytype 
or as a string. 
The ArrayQuantity goes into a <Group type="images"> and the columns are  
"type of file":  FITS in this case,  and "URL":  give the href value.

Finally QuantityWithObjects becomes a link
<A href="locationOfNewTable">typeValue</A>. 
If there are several different type of Objects in this node, then it may 
need to be broken
out into separate tables.

>How about the "auto-un-flattener" for recovering the hierarchical form from
>the tabular form?  Is it always lossless?  Does it need artifacts other than
>the results tables ex the auto-flattener to work?
>
>  
>
The crumbler is created by exactly reversing the proceedure. 

>Is the "compact form" some existing standard or do we have to define it?
>  
>
I think VOTable can handle this.  I am not sure about the links down the 
hierarchy of tables.

>Is it in one logical chunk of meany (e.g. one per table)?  
>
No.

>Is it streamable?
>  
>
Either VOTable or this Objects.xml are streamable, I think.

>How is it entered into the various RDBMS we use? (One would expect to need
>adaptors for each DBMS.)
>  
>
The Table view of the data should fly into a DBMS.  And hopefully, you 
can extract it with
the above Objects.dtd or its equivalent xsd.

>Can we make an auto-flattener that produces existing formats like binary FITS?
>
>  
>
Could, if FITS could handle a hierarchy of tables.  Or, if you allow one 
FITSTable to hyperlink to another.

>Cheers,
>Guy
>
>Guy Rixon 				        gtr at ast.cam.ac.uk
>Institute of Astronomy   	                Tel: +44-1223-337542
>Madingley Road, Cambridge, UK, CB3 0HA		Fax: +44-1223-337523
>
>  
>