IVOA Document Standards RFC V1.2

[Norman Gray]

Bob and all, hello.

I put off taking part in this discussion back in July, but given the  
recent TCG discussion about the requiring or not of implementations,  
I'd like to add something tentatively here.  There are two strands:  
what should funders want, and how should they be persuaded they're  
getting what they need.

On 2009 Jul 3, at 23:43, Robert Hanisch wrote:

>>>>> I would probably agree fully with you if, say, IVOA were a dues- 
>>>>> based
>>>>> membership organization, with its own financial resources to build
>>>>> validators, or test those developed by others.  But I see a  
>>>>> distinction,
>>>>> albeit a subtle one, between the projects' willingness to  
>>>>> contribute to
>>>>> development of a standard and their ability to provide  
>>>>> supporting software
>>>>> and validators on a timescale that does not jeopardize the  
>>>>> promotion
>>>>> process.

This is slightly parenthetical, but given that standardisations are  
all about the _reputation_ of the standards body, then I think it's at  
least _reasonable_ for a standards body to say 'if we're going to  
invest our reputation in a standard, then we can expect participants  
to invest resources in the standard they claim to support'.  End  
parenthesis.


Bob, responding to Mark, said:

>>> As above.  Agreement on these standards is a major milestone.   
>>> Read any of
>>> the NVO Quarterly Reports to NSF (all are on the NVO web page, in  
>>> the
>>> document collection) and you can see how important these are in  
>>> terms of
>>> deliverables.

I don't much like Bob's argument here, but I do respect it as an  
important one, and I want to criticise it on its own terms.

The various funders want to support the development of standards which  
(a) are good for the discipline, and (b) are respected enough that  
they are taken up by the community.  The funders want some evidence  
that this is happening, and ideally that it happened as a result of  
their support.  'Number of standards agreed' is a convenient metric  
for progress for obvious reasons -- it's quantitative and reasonably  
predictable.  However I think it is not a good metric for measuring  
progress towards either of the goals the funder _actually_ has.

The IVOA is one of a range of standards bodies in the world.  I've had  
some exposure to the standards of ISO, W3C, IETF, ECMA and OCLC.   
Those bodies have different funding models, cultures, and timescales,  
and their standards have very different authorities, from the  
(generally) high-status, pragmatic and broadly implemented IETF ones,  
to the joke ECMA ones.  A common criticism of ISO standards is that  
they're produced by highly political committees, and are barely  
implementable -- bad.

The IVOA's funders (I hope) want IVOA standards to be at the upper end  
of that range of authority, but agreeing standards too early -- going  
off half-cocked, in the rush to report number of standards agreed --  
could militate _against_ our standards having the authority we and  
they want.

So what _can_ we show to the funders?

Mark mentioned:

>> maybe there's
>> another way forward: declare an intermediate document stage of
>> "accepted in principle, no major changes expected, but revisions
>> possible on the basis of implementation experience".  This stage
>> could be sold as the one that counts to the funding agencies.

This is very similar to the W3C's 'Candidate recommendation' stage  
(one of the few bits of W3C process that we didn't lift).  This is the  
stage before 'proposed recommendation' [1], and is for standards where  
'W3C believes the technical report is stable and appropriate for  
implementation. The technical report may still change based on  
implementation experience.'   Thus it's a formal (reportable)  
milestone, with the implication that a lot of hard work has been done  
and the only changes between now and PR will be based on  
implementation experience.  You don't really need implementations to  
get to CR, but you do need at least one implementation of every  
feature in order to proceed beyond PR.

I'm not pushing this CR stage as a magical solution, but I believe  
that something like this is necessary for the IVOA, and desirable for  
the IVOA's funders, if someone can work out how to tell them so.

   * We don't want IVOA standards to get the 'ignore v1.0' reputation  
which we'll get if we rush standards to REC too quickly -- that just  
wastes time and makes us look cheap.

   * I think any standard is a poor standard if it proceeds to REC  
without implementations, validators, or some other public concrete  
evidence that it's _ready_. Having the WG Chair say 'I can feel it in  
my bones' does not count. [that is, I'm firmly in the must-implement  
rather than should-implement camp]

   * I think a standard is a poor standard, if it normatively depends  
on still-draft standards. [this shouldn't need saying]

   * If no-one can be bothered to implement a pre-REC standard, then  
that is evidence that that standard should not exist. [sure, there are  
plenty of people for whom this would be too early to implement, but if  
there aren't any early-adopters, what's the point?]

> It was understood from the beginning of IVOA that standards would  
> evolve and
> that first versions would likely need to change.  I found something  
> I wrote
> back in March 2004, concerning the registry standards, that I think  
> remains
> relevant today:
>
> We need to reach agreements quickly, build to them, and iterate to  
> improve
> them.  We need to be willing to build and revise, or even in some  
> cases
> throw away and start again.  We have set up a standards process that
> recognizes change, and is intended to be responsive to change. I do  
> not
> believe we can afford to set 2-3 year schedules for reaching  
> consensus. If
> it takes this long, the community upon which we will utimately  
> depend for
> support will see us as purveyors of snake-oil, and will blow us off.

I agree with the sentiment, and agree with the need for iterations and  
(particularly) discarding when necessary.  Standards do need updating  
and improving.  But that doesn't imply making v1.0 standards throwaway.

The problem is that we _do_ appear to have 2-3 year schedules for  
reaching consensus.  If that's too long (and I'd agree with 'snake  
oil' for ambitious proposals in permanent draft), then we need to  
shorten the process earlier, perhaps by insisting that v1 proposals  
are less ambitious, but not by skipping an important late stage.

A 'standard' which turns out to be unimplementable is also snake oil,  
and I'd argue that it's _worse_ than one in permanent draft, because  
at least the permanent doesn't pretend to be a fully thought-through  
document.

I appreciate that this may be a hard sell to funders (and I say this  
while up to the elbows in my half-written grant proposals).  You know  
what you (plural) signed up for!

All the best (and good luck),

Norman


[1] http://www.w3.org/2005/10/Process-20051014/tr.html#cfi

-- 
Norman Gray  :  http://nxg.me.uk
Dept Physics and Astronomy, University of Leicester, UK