[Ops] UserAgentUsage

Thu Dec 6 19:14:05 CET 2018

Mark, Ops:

With my MAST devops hat on, short information in the user-agent string is by far the most useful, and I think putting in 'IVOA-validate' and the standards being tested  is great, and enough to:

1. filter on in software and

2. if an issue arises, trigger a human to check an IP if available, or use a provided contact address, or ask around to a short list of folks running validators.

Any information in additional custom headers is going to get dropped in at least one of our logging pipelines and never seen; the additional work on the client end will not be immediately usable for us at all. If anyone else wants to use it, go for it, but that minimum level of information in the user-agent header is what we can immediately start looking for.

--Theresa

STScI, Archives

________________________________
From: ops-bounces at ivoa.net <ops-bounces at ivoa.net> on behalf of Mark Taylor <M.B.Taylor at bristol.ac.uk>
Sent: Thursday, December 6, 2018 7:54:22 AM
To: Rick Ebert
Cc: IVOA Operations IG
Subject: Re: [Ops] UserAgentUsage

Rick and Tom,

first, good catch about the "validate"/"validator" inconsistency,
I've fixed this now (to "validate", since it seems easier to get
verbs than nouns to match the idea of an operational purpose).

Concerning the scope and syntax of the <optional-extra-text>, I'm
inclined to agree with Rick that it's not necessary or desirable
to define more syntax here.  For one thing, as he says, encouraging
more content in that string is somewhat against the RFC2616 injunction
that the text should be "short and to the point", and additional detail
if required could be put into custom headers.  But also, my guess is
that if recipients of this information want to use the User-Agent
content beyond the fact that something like "(IVOA-validate" is present,
the additional text will be consumed by humans rather than machines.
If that's the case, it doesn't really matter whether you write

   resultURL=http://validators.org/results standard=sia2.0,votable1.4 contact=ops at institution.edu

or just

   http://validators.org/results sia2.0,votable1.4 ops at institution.edu

since the convention as currently written does not constrain the
syntax or semantics of that string, and human readers can pretty
easily work out what's going on either way.

Since, other things being equal, keeping these notes/conventions
short is always a good idea, I think just adding a bit more
clarification about the sort of detail that is or isn't welcome
in this field might do the job just as well.  On the other hand,
encouraging a simple name=value syntax for these items is fairly
straightforward too, so maybe it is worth it (along with advice
not to try to put too much detail in there).

I'd certainly be happy to hear from other service operators
who have opinions about this, in particular if people think that
they might want to have this kind of operational detail available
in a format that's easy to parse.  If that turns out to be
a significant need, maybe we should think about some conventions
on X-IVOA-whatever headers as well as Rick suggests.  Though I
have to admit from an implementation point of view I'd find inserting
new headers much harder to do in e.g. taplint than editing User-Agent,
which happens to be pretty easy in Java at least.

Mark

On Wed, 5 Dec 2018, Rick Ebert wrote:

>
> The first bit of this discussion i heard at the IVOA meeting College Park.
>
> I think putting the (ivoa-validator ...) tag in the UA string is a fine
> idea - in fact, i am already looking for it in NED logs.
> I am not sure looking at the UserAgentUsage wikipage if its
> ivoa-validate or ivoa-validator ??
>
>
> I like the idea of having a URL for more information (contact, and
> links to public results ... anything related to the validator.)
> But please stop there.  The simple example strings in the Usage note,
> are just right.
>
> The client can put more information into
>  HTTP headers X-IVOA-whatever
> and operators can log those if they like.
> but opening UA strings to an entire dictionary of ancillary information
> makes the string horrendously long.
>
> The URL should provide more information, not just results: operational information
> (how often it runs,  target list, who to contact and how for trouble, all
> on a webpage.) would be really useful sometimes ...
>
> I don't think the there needs to be keywords, provided the strings ID, and URL
> are unbroken by whitespace).
>
> Googlebot/2.1 (+http://www.google.com/bot.html)
> is a perfectly legitimate UA string.  To have a string that can be
> detected no matter what combination of tools is involved is handy.
> IVOAbot/3.1 (+https://www.botinfo.edu/) STILTS/3.1-4
>
> With NED, if you declare your client as a 'Bot' we are unlikely to
> count it as "science use" - but it would be nice to have even in
> those cases a way to say we supported 40,000 ivoa service validation
> requests this year.
>
> To have a string that can be detected no matter what combination of
> tools is involved is handy - which is why i like putting the
> (ivoa-validate +https://url...) in the UA.
>
>
> In addition to validate,monitor,harvest, a  "proxy"  (or "user")
> op-purpose could indicate explicitly that the query is being made
> in direct response to a science user query at a higher layer
> ( in other words "definitely count this" as use of the VO).
> But this starts to go beyond the idea of the UA string and creates
> more clutter.
>
> Again for this, I would suggest taking it ALL out of the UA string,
> except the
>   (ivoa-validate +https//inforurl.org/)
> and put the deeper information in a separate HTTP: header
>
> Rick...
>
>
> On 12/4/18 9:02 AM, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
> > Hi Mark,
> >
> > This looks fine to me generally, but I'm a little concerned that if it
> > turns out to be useful then we may have made it difficult to expand the
> > kinds of additional information that are included in the comment.  Right
> > now the only additional information we are considering is the URL at
> > which the validation results may
> > be seen.  That's probably all we'll need, but it seems to me that we'd
> > leave it a little more open if we were to include a standard prefix so
> > that if we
> > eventually want to provide something else we can just use a different
> > prefix.  E.g., you give the example
> >
> > User-Agent: STILTS/3.1-4 (ivoa-validator http://validators.org/results)
> > Java/1.8.0_181
> >
> > Here the ivoa- lets us know we've got some IVOA ops info here.  The
> > validator gives us the specific usage, and the
> > http://validators.org/results is where validator results are given.  I'd
> > like to see that last a little more explicit as
> >
> > User-Agent: STILTS/3.1-4 (ivoa-validator
> > resultURL=http://validators.org/results) Java/1.8.0_181
> >
> > where we've agreed that resultURL (or some other string) defines what
> > the token i.  Next year if this turns out to be popular we could add some
> > other (presumably optional) keyword to give some other information.
> > Even now, this allows the provider to include information they think is
> > important
> > that isn't the results URL.
> >
> > Just to give an example, maybe the querier wants to indicate the
> > standard[s] being validated, so they'd put
> >
> > User-Agent: STILTS/3.1-4 (ivoa-validator
> > resultURL=http://validators.org/results standard=sia2.0,votable1.4)
> > Java/1.8.0_181
> >
> > or instead of a URL for the results, an Email to contact with questions
> >
> > User-Agent: STILTS/3.1-4 (ivoa-validator contact=ops at institution.edu)
> > Java/1.8.0_181
> >
> > What do you think?
> >
> >      Tom
> >
> > Mark Taylor wrote:
> >> Dear Ops,
> >>
> >> at the recent Interop we had some discussions about how we can/should
> >> use the User-Agent HTTP header to allow service operators to identify
> >> requests originating from things like validators as distinct from
> >> user science requests.  Following this, I have drafted some text
> >> on this wiki page:
> >>
> >>     http://wiki.ivoa.net/twiki/bin/view/IVOA/UserAgentUsage
> >>
> >> I think it more or less reflects what we concluded at College Park,
> >> but there are a few details and open questions I've filled in.
> >> If you're interested in this topic, please take a look and
> >> post comments here or to me, or just edit the wiki page.
> >>
> >> If people are in agreement with this approach, I'll try to
> >> implement it in my clients, and we can talk about whether it
> >> makes sense to issue it as an IVOA Note.
> >>
> >> Mark
> >>
> >> --
> >> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> >> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
> >> _______________________________________________
> >> ops mailing list
> >> ops at ivoa.net
> >> http://mail.ivoa.net/mailman/listinfo/ops
> >
> > _______________________________________________
> > ops mailing list
> > ops at ivoa.net
> > http://mail.ivoa.net/mailman/listinfo/ops
> >
> >
>
>
> --
> Rick Ebert
> Caltech/IPAC
> Engineer, NASA/IPAC Extragalactic Database
>

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
_______________________________________________
ops mailing list
ops at ivoa.net
http://mail.ivoa.net/mailman/listinfo/ops
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/ops/attachments/20181206/930829a8/attachment-0001.html>