discussion status & my recommendations

[KevinBenson]

Very good stuff Ray.  I would agree with almost everything there including
tabling the idea of stamping, as you say it is complex and probably not
worth it at this moment.
----
The only one I am still unclear about is verificationLevel and not sure if I
agree with the notion still that any registry can change this value (at
least not on the vr:Resource).

>From what I gather from previous e-mails (and this e-mail) there may be some
apps that are going to query on this attribute, and that is what bothers me
if our registries will be different.  I understand our full registries may
not be exactly the same because of different harvesting times, but by and
large they should be very close.  And if scientist are collaborating on a
app pointed at different registries and depending on how the registry uses
the verificationLevel no entries may show up for the app at all (if the
registry set resources at a low level).

*Other problems that you need to make clear on how this would work out.
If we state that a registry can keep the same value harvested from another
full registry then (how or who) do you know gave the original
verificationLevel?  (Or does this matter?)
Depending on how this is implemented, does this mean that if the
publishing/original registry user updates a Registry entry to make it great
going probably from a verificationLevel of 0 to 4, that he/she might have to
start contacting other full Registries to re-evaluate that entry otherwise
it might sit at 0 for a long time when it deserves to be a 3 or 4 (that
would sound like a lot of work by each registry to have to go back and
re-evaluate entries)
A final comment is this makes some extra work to change the logic around to
have the ability to change an external Resource (not owned/managed). And
then figure out if you should take or not take the verificationLevel from
another full registry. But yes probably could be done with not to much
headache.

If we do decide to carry on with this, is there any objection that I just
default all entries to the "best and highest" level possible to make sure
all apps will be happy. Only Joking .....for now :)

Don't get me wrong, I do like the idea of DataQuality and VerificationLevel
just not sure if we are going about it the correct way, unfortunately I
don't have any other suggestion at the moment.

Cheers,
Kevin

-----Original Message-----
From: owner-registry at eso.org [mailto:owner-registry at eso.org]On Behalf Of
Ray Plante
Sent: 09 May 2005 13:09
To: registry at ivoa.net
Subject: discussion status & my recommendations


Hi RWGers,

With the interop meeting quickly approaching I thought it would be good
summarize the status of currently unresolved issues with the Registry
Interface (RI) standard and make some recommendations so that we can move
forward.

Given the state of the discussion, I think we should be aiming at a v1.0
Working Draft for RI rather than a Proposed Recommendation (it's
currently an internal WD).  Once we have confirmed that we all have
working implementations, we should elevate it to PR status.  Changes we
make as part of that transition (which could include some of the
suggestions below that I am recommending we table) should be based on our
working experience.

1.  harvestedFrom

It was suggested that we add this as an attribute to the base Resource
type and would be used to help track problem records.  It was meant to
address an ambiguity when tracing a record that may have made multiple
hops.  Some registries keep this information already internally.  By
making this part of VOResource it would expose it through our standard
interfaces.

I should note that in the discussions, it was not made clear exactly what
the value should be, an IVOA registry identifier or an OAI URL base
(current implementations use the latter).

Recommendation:  Given the lack of clarity of its usefulness and what the
value should be, I think this suggestion should be tabled for now.

2.  registry record curation/stamping

This thread started with the concern that there would be parts of a
VOResource record that a particular registry could update (as opposed to
the original publisher of the record) and that, subsequently, VOResource
records would not be identical regardless of where it was retrieved.  The
registry-updatable information would include "harvestedFrom",
"verificationLevel", and possibly "status", all encoded as attributes to
the base Resource type.

The above concerns led to the suggestion to pull this information out of
the VOResource record and into a special sibling node called something
like "stamping".  The "stamping" part and the VOResource part
would be combined to be our registry record.  This would allow this
stamping information to be used to select resources in a registry
search.  The motivation for the "stamping" node is to clearly
separate what is provided by the publisher and what is provided by the
registry.  It also provides a place for other registry-updatable
information (e.g., "liveliness").

Recommendation:  I recommend that we table the creation of the "stamping"
node.  We added "verificationLevel" as an attribute to the base Resource
type but not "harvestFrom" (see #1), and we indicate that "status" only be
updateable by the original publisher.

Although the "stamping" node is a very nice idea, it represents a major
change to our metadata standard.  The cost of updating our
implementations would be high--too high for something that has not been
prototyped, yet.  Also, it opens the door to add lots of other information
that hasn't be sufficiently motivated as being needed nor prototyped and
demonstrated.

If a "verificationLevel" is the only thing we allow to be
registry-updatable, we can add it as an optional attribute and have
minimal impact on the current implementations.  We can see how this simple
adjustment works out to see if a separate "stamping" node is needed.
Meanwhile, if we want to start prototyping a "stamping" node, we can do it
using a separately defined (OAI) format.

3.  verificationLevel

This is a new item that is proposed to be added as an attribute (see
also #2).  Fully defined in the v1.1 of the RM (WD), it is intended to
indicate a level at which the metadata (and service interface, when
appropriate) has been checked to be "good" according to the registry
holding the record.  Some of the criteria for this evaluation is set by
the registry.  A registry may choose to preserve the value it finds in the
harvested record or override it according to its own evaluations.
Users/applications may use this attribute to select resources as part of a
search.

Recommendation:  I recommend we add this to VOResource as an attribute
to the base Resource type.  This is critical for putting into place good
registry curation practices which is a major area of focus now that we
have some operating registries.

4.  ADQL => RQL

The RI current specifies restricted forms of ADQL (only a Where node is
given) and XPath to enable constraint-based searching.  The need for these
restrictions and presence of XMatch in ADQL, which is not needed by
registries, motivated a suggestion that we branch off from the ADQL and
maintain our own Registry Query Language based on ADQL.

Recommendation:  We should not yet break off from ADQL for v1.0.  We
should update RI to use ADQL v0.91.

The issue of how our standards should depend on each other is a profound
one that we can't fully understand until more of our standards become
stable (i.e. reach rec status).  I recommend that we build to ADQL v0.91
and reexamine this issue based on our experience before we go to PR
status.

--------

Your comments, as always, are welcome.  In this last week before Kyoto, we
should concentrate on things that help us wrap up the RI specification.

cheers,
Ray