[Ops] Inconsistency between full searchable registries

[Theresa Dower]

Thomas,

As an update, yesterday we finished full-harvesting the VizieR registry. We recovered 1369 records that had been updated previously but missed in our incremental harvest. It looks like they were all due to the previously mentioned issue with publishing versus noted update time. This brings the total number of active VizieR records in our registry up to 14326.  There seem to be  a few missed records from 3 of the 7 months this covers, so Menelaus’ monthly full-harvest method looks like a good solution from our standpoint.

Thanks again for bringing this to our attention,
--Theresa


From: ops-bounces at ivoa.net [mailto:ops-bounces at ivoa.net] On Behalf Of Theresa Dower
Sent: Monday, March 14, 2016 1:26 PM
To: Thomas Boch; registry at ivoa.net
Cc: ops at ivoa.net
Subject: Re: [Ops] Inconsistency between full searchable registries

Thomas,

Hello! I know we at STScI have had occasional issues for years with VizieR records slipping through the daily harvesting cracks.  The last time I worked on this with Sebastien Derriere, we determined that the way VizieR records are published, sometimes the ‘created’ date is not set to the date of publishing records into the VizieR registry, thus when we harvest incrementally daily, records can be missed. This may still be an issue. I am curious whether this is still happening, or whether we have an ingest bug on our end that is for some reason not being logged.

I know that Euro-VO re-harvests entire registries from scratch (not incrementally) on a regular but infrequent basis, which would explain them having these records we have missed. I will re-harvest VizieR registry by hand and see how many new records we get.  We at STScI are also publishing an update to our registry software and database in the coming weeks, and will get a new fresh harvest from every registry at that time. Once we have the new registry software running operationally I will work on making that re-harvest a semi-regular automated event, as Euro-VO does.

Thank you for bringing this issue up!
--Theresa

From: ops-bounces at ivoa.net<mailto:ops-bounces at ivoa.net> [mailto:ops-bounces at ivoa.net] On Behalf Of Thomas Boch
Sent: Monday, March 14, 2016 9:44 AM
To: registry at ivoa.net<mailto:registry at ivoa.net>
Cc: ops at ivoa.net<mailto:ops at ivoa.net>
Subject: [Ops] Inconsistency between full searchable registries

Hi Registry-enthusiasts,

I would like to report on an inconsistency I found between resources available in the EuroVO registry and in the VAO/STScI registry.

I am performing daily a full harvesting (through OAI PMH) of the registry in order to retrieve and filter out services of interest to Aladin Desktop. I used to query the STScI registry for this task until I found out some active resources were missing (for instance ivo://cfa.tdc/hectospec/hectospec_public.ssap.q/ssa). I then switched to the EuroVO registry and just found out that some other resources, for instance ivo://nasa.heasarc/skyview/skyview, were also missing (but available in the STScI registry).

The full list of missing resources for each registry is attached to this message. From a quick look:

- STScI registry is mostly missing 1300 VizieR resources

- EuroVO registry is mostly missing HEASARC services. Menelaus confirmed me that they had an issue with querying the HEASARC registry.


What should I do ? I am not really keen on querying the two registries and merging the results, as I feel this should not be done at my side. I would expect consistency between full registries, at least for resources older than 1 week. Am I missing something ?


Cheers,

Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/registry/attachments/20160316/e79bcb5f/attachment-0001.html>