[Ops] Draft concept for standard validator output

[Mark Taylor]

On Thu, 12 May 2016, Markus Demleitner wrote:

> On Thu, May 12, 2016 at 03:02:01AM -0400, Tom McGlynn (NASA/GSFC Code 660.1) wrote:
> > Attached find a sketch of a possible standard output format for validators.
> > It uses a relatively simple but recursive structure to accommodate both
> 
> I refer to my personal Gospel, the Zen of python:
> 
>   $ python -c "import this" | grep nested
>   Flat is better than nested.

I tend to agree with Markus in this case that flatness is preferable,
though I might be persuaded otherwise (it's more a call for validator
consumers than validator producers).

> I think all of these can be essentially be represented as a sequence
> of 
> 
>   <test_id>, <test_description>, <test_result>, [<see_also>]
> 
> (where <see_also> could be a link to the spec or a page discussing
> how to fix things) plus some "global" (e.g., VOTable PARAM)
> parameters:
> 
> <validator id>, <validator description>, <global result>
> 
> with <global result> one of success, warnings, errors (or whatever).

The <test-id>,<test-description>,<test-result>}[] model doesn't fit
too well with the way that I write validators.
I don't really think in terms of testing a feature,
determining pass/fail, and moving on.  Instead I hit the service
in whatever ways I can think of, look for failures or anomalies,
and report them.

So typically my testing of a given behaviour or standard requirement
is not done in one place, it is distributed over large parts of
the validator execution.  For instance when votlint (standalone or
within taplint) validates a VOTable, every time it encounters a
TD element it checks that the content matches the corresponding
FIELD declaration.  Clearly, it's not a good idea to emit a
test=pass message each time.
It would be possible for me to gather all the messages in that
category for a single later report but (a) it would be a pain in
implementation terms and (b) it could impact on scalability 
(streaming very large VOTables).

What would work for me?

   message-type (info, error, warning, ...)
   message-ID (implementation specific, some repeatable tag)
   message-text (human-readable, free-form)
   see-also (structured? pointer to standard name, version?, section?)

This does not preclude the feature-test->report model, since one
could write

   message-type = PASS
   message-ID = ....
   message-text = Tested feature XXX, no problems
   see-also = VOFoo, v3.9, sec 10.1.1

but a straight map of taplint or votlint into this model would not
emit many message-type=PASS messages.

Mark

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/