IVOA-science in Softid

[Mark Taylor]

Markus,

I think this is a good idea.  As you say it's not the answer to
all the VO's unwanted crawler-related problems, but it may allow
services to take some steps to work round problems of overloading
and usage statistic distortion in this area.

I've implemented it in prototype from the client side for topcat 
and stilts:

  https://www.star.bristol.ac.uk/mbt/releases/topcat/pre/topcat-full_softid.jar

Depending on how this discussion goes I may merge this change into
future public releases.

Mark

On Tue, 23 Dec 2025, Markus Demleitner via apps wrote:

> Dear Colleagues,
> 
> As a small piece in the puzzle of how to deal with stupid crawlers
> overloading our services, I have occasionally suggested a small
> secret handshake between legitimate clients and our servers in hopes
> that might be enough to avoid the APIkeycopalypse.
> 
> I am, indeed, rather convinced that "follow a standard" would already
> work pretty well to keep out most "AI crawlers", and that
> quasi-standard could be our note on operational identification
> of software components <http://ivoa.net/documents/Notes/softid/>
> (which has quite a few other use cases, too, so if you are writing VO
> clients or servers, please have a look).
> 
> For clients, this note has the provision of an IVOA comment.  So far,
> the idea has been that validators would have something like
> 
>   User-Agent: STILTS/3.1-4 (IVOA-test http://validators.org/results) Java/1.8.0_181
> 
> in their request headers (crawlers would give IVOA-copy), and
> diligent beancounters would remove such requests (done for validation
> in this case) from their usage analyses (as questionable as these may
> be even then).
> 
> So far, the standard said that "normal" usage should not give any
> purpose.  Now, giving "science" as a purpose would exactly be such a
> follow-the-standards handshake.
> 
> I have now written up a proposal for how this would look like as
> a PR against softid: <https://github.com/ivoa/softid/pull/1>.
> 
> What do people think?  As a client, would you adopt it?  As a server,
> can you imagine any use for IVOA-science in user agents?
> 
> If sufficiently many people feel this is a good idea: Perhaps it's
> then time to try and get softid endorsed?
> 
> Thanks[*],
> 
>          Markus
> 
> [*] Enjoy your holidays if you have them!
> 
> 

--
Mark Taylor  Astronomical Programmer  Physics, Bristol University, UK
m.b.taylor at bristol.ac.uk          https://www.star.bristol.ac.uk/mbt/