How do we get metadata into the registry network?

[Ray Plante]

Hi Clive,

On Fri, 25 Jun 2004, Clive Page wrote:
> One notable omission from the Registry interfaces document dated June 16th
> is any way of loading metadata from the primary sources, which are the
> files or databases in each data centre.  

> Where to store metadata?

The reason you haven't found much on this topic is perhaps because it is 
not an issue we need to standardize on.  As you alluded to, many centers 
will need to automate the exposure of their metadata by hooking the 
harvesting interface into their local metadata management systems.  How 
this will be done will vary widely.  Certainly, it will help if we can 
provide general tools that assist with this.  

This is how it's been approached in the NVO:  

  o  For the data provider that has only a few resources (e.g. <10) to 
     register, we have built registration forms at our various 
     registries that providers can fill out.  I encourage you to try out 
     the page at NCSA:

       http://nvo.ncsa.uiuc.edu/nvoregistration.html

     With this form, a provider uses a public publishing registry to 
     expose their resources; that is, the records are store at the remote 
     registry.  

  o  For the provider that has a few more resources (e.g. 10-100), they 
     may prefer to manage their records locally.  For this set, we have 
     begun building deployable registries that have the harvesting 
     interface built in.  Examples include NCSA's VORegistry-in-a-Box 
     (http://nvo.ncsa.uiuc.edu/VO/software/VORegistry-In-A-Box.html) and 
     Caltech's Carnivore (http://mercury.cacr.caltech.edu:8080/carnivore/)

  o  For the provider with many resources that have sophisticated 
     metadata management systems (like Vizier), they will want to plug the 
     harvesting interface directly into database.  The examples from the 
     NVO prototype are HEASARC and CDS.  These chose to build the 
     harvesting interface themselves (leveraging, I believe, existing OAI 
     tools).  

> So - who is going to develop these tools?

As illustrated from the example above, development is shared between 
providers and VO developers.  If we want see broad participation from 
providers, the cost must be low; thus, we must recognize:

  1.  Incremental levels of compliance--how much providers register, how 
      much info they provide, and what standard services they deploy--
      makes it easier for providers build up participation gradually as 
      they can afford.

  2.  The more tools we can give providers to get over the difficult bits, 
      the more compliance they can afford.

I think both these are in evidence in the above example.  

cheers,
Ray