RegTAP service discovery queries

[Markus Demleitner]

Hi Mark,

On Mon, Jun 17, 2019 at 01:50:46PM +0000, Mark Taylor wrote:
> On Thu, 13 Jun 2019, Markus Demleitner wrote:
> 
> > Hi Mark,
> > 
> > On Thu, Jun 13, 2019 at 10:32:00AM +0000, Mark Taylor wrote:
> > > some of the recommended queries (Section 10) in PR-RegTAP-1.1
> > > have changed since REC-RegTAP-1.0, in particular the the standard
> > > service search constraints
> > > 
> > >    "intf_role='std'" instead of "intf_type='vs:paramhttp'"
> > > 
> 
> > > Might it break anything?  Should I only use these altered forms
> > 
> > intf_role=std: Not in the Registry as of a few weeks back.  Meaning:
> > all vs:ParamHTTP interfaces TOPCAT might be interested in had a
> > role="std", too.  
> 
> I was about to do this, but in tests I see that (e.g.) the resource
> ivo://svo.cab/arches, which is apparently an SSA service
> (I can make SSA queries to it) does have intf_type='vs:paramhttp'
> but does not have intf_role='std'.  From http://dc.g-vo.org/tap
> at 2019-06-17:
> 
> [...]
>
> So it looks like I'm losing search results with this change.
> I haven't attempted a more principled test to see how widespread this
> kind of thing is.  Can you comment?

Well, fresh out of vacation, this is a bit of a shock.

First, on why I've been mistaken to believe all is well: A classic
case of NULL traps.  You see, 

  select ivoid
  from rr.interface
  where intf_type='vs:paramhttp'
  and not intf_role='std'

returns empty indeed, because where intf_role isn't 'std', it's NULL
and (not NULL='std') is true.  Oh dang.

Thanks for checking this.  The query I should have run is something
like:

  select ivoid, intf_type, intf_role, standard_id
  from rr.interface
  natural join rr.capability
  where intf_type='vs:paramhttp'
  and standard_id not like 'ivo://ivoa.net/std/vosi%'
  and intf_role is null

-- this will give me all interfaces that claim to conform to a
standard (other than VOSI; I'm not concerned about discovering them
here -- see caproles for details).

This now returns a shocking 513 records.  Ouch.

Luckily, most of them are noise in some way or other (various
pre-IVOA VOSI things that really aren't std, Helio stuff, "stap",
delegation endpoints...).  There are quite a few SODA and Datalink
interfaces in there because DaCHS didn't set role="std" for them.  It
does now, but since I don't think any client is doing anything with
there, there's no particular hurry to roll out that fix.

Realistically, I'd say the thing matters for the protocols we're
actually discovering, so I'm settling for

  select ivoid, intf_type, intf_role, standard_id
  from rr.interface
  natural join rr.capability
  where intf_type='vs:paramhttp'
  and intf_role is null
  and standard_id in (
	  'ivo://ivoa.net/std/tap',
	  'ivo://ivoa.net/std/sia',
	  'ivo://ivoa.net/std/sia#query-2.0',
	  'ivo://ivoa.net/std/ssa',
	  'ivo://ivoa.net/std/conesearch')

to find the services clients migrating would be losing.  If you spot
other standard_ids that have usable and used standards involving
discovery queries, please let me know.

Anyway, that's still 130 services, but almost all of them are either from
wfau, jvo, or svo.  It's totally doable to contact the operators and
ask them to fix their records, and I'll do that.  I'll report back
here when no working services are on this list.

Sorry for forgetting about SQL's NULL semantics (and thanks again for
paying attention),

         Markus