IVOA Document Standards RFC V1.2

[Robert Hanisch]

On 9/20/09 6:02 PM, "Norman Gray" <norman at astro.gla.ac.uk> wrote:

> 
> Bob and all, hello.
> 
> I put off taking part in this discussion back in July, but given the
> recent TCG discussion about the requiring or not of implementations,
> I'd like to add something tentatively here.  There are two strands:
> what should funders want, and how should they be persuaded they're
> getting what they need.
> 
> On 2009 Jul 3, at 23:43, Robert Hanisch wrote:
> 
>>>>>> I would probably agree fully with you if, say, IVOA were a dues-
>>>>>> based
>>>>>> membership organization, with its own financial resources to build
>>>>>> validators, or test those developed by others.  But I see a
>>>>>> distinction,
>>>>>> albeit a subtle one, between the projects' willingness to
>>>>>> contribute to
>>>>>> development of a standard and their ability to provide
>>>>>> supporting software
>>>>>> and validators on a timescale that does not jeopardize the
>>>>>> promotion
>>>>>> process.
> 
> This is slightly parenthetical, but given that standardisations are
> all about the _reputation_ of the standards body, then I think it's at
> least _reasonable_ for a standards body to say 'if we're going to
> invest our reputation in a standard, then we can expect participants
> to invest resources in the standard they claim to support'.  End
> parenthesis.
I agree in principle.  I also see practical problems, such as the funding
shortfalls we are facing in the UK and US (the latter being temporary, we
believe, and the former requiring a new approach).

> Bob, responding to Mark, said:
> 
>>>> As above.  Agreement on these standards is a major milestone.
>>>> Read any of
>>>> the NVO Quarterly Reports to NSF (all are on the NVO web page, in
>>>> the
>>>> document collection) and you can see how important these are in
>>>> terms of
>>>> deliverables.
> 
> I don't much like Bob's argument here, but I do respect it as an
> important one, and I want to criticise it on its own terms.
> 
> The various funders want to support the development of standards which
> (a) are good for the discipline, and (b) are respected enough that
> they are taken up by the community.  The funders want some evidence
> that this is happening, and ideally that it happened as a result of
> their support.  'Number of standards agreed' is a convenient metric
> for progress for obvious reasons -- it's quantitative and reasonably
> predictable.  However I think it is not a good metric for measuring
> progress towards either of the goals the funder _actually_ has.
I did not mean to suggest that just the number of standards agreed upon is
itself much of a metric.  It reminds me of "Standards are great, let's have
lots of them!"  But, progress towards those standards that are agreed to be
necessary -- that is a reasonable metric.

> The IVOA is one of a range of standards bodies in the world.  I've had
> some exposure to the standards of ISO, W3C, IETF, ECMA and OCLC.
> Those bodies have different funding models, cultures, and timescales,
> and their standards have very different authorities, from the
> (generally) high-status, pragmatic and broadly implemented IETF ones,
> to the joke ECMA ones.  A common criticism of ISO standards is that
> they're produced by highly political committees, and are barely
> implementable -- bad.
> 
> The IVOA's funders (I hope) want IVOA standards to be at the upper end
> of that range of authority, but agreeing standards too early -- going
> off half-cocked, in the rush to report number of standards agreed --
> could militate _against_ our standards having the authority we and
> they want.
> 
> So what _can_ we show to the funders?
Actually, at this point I don't think our funders are particularly
interested in the standards per se.  More on that later.

> Mark mentioned:
> 
>>> maybe there's
>>> another way forward: declare an intermediate document stage of
>>> "accepted in principle, no major changes expected, but revisions
>>> possible on the basis of implementation experience".  This stage
>>> could be sold as the one that counts to the funding agencies.
> 
> This is very similar to the W3C's 'Candidate recommendation' stage
> (one of the few bits of W3C process that we didn't lift).  This is the
> stage before 'proposed recommendation' [1], and is for standards where
> 'W3C believes the technical report is stable and appropriate for
> implementation. The technical report may still change based on
> implementation experience.'   Thus it's a formal (reportable)
> milestone, with the implication that a lot of hard work has been done
> and the only changes between now and PR will be based on
> implementation experience.  You don't really need implementations to
> get to CR, but you do need at least one implementation of every
> feature in order to proceed beyond PR.
I worry about inserting another step in the process when already we have
considerable difficulty getting IVOA participants to follow the process we
have. 

> I'm not pushing this CR stage as a magical solution, but I believe
> that something like this is necessary for the IVOA, and desirable for
> the IVOA's funders, if someone can work out how to tell them so.
> 
>    * We don't want IVOA standards to get the 'ignore v1.0' reputation
> which we'll get if we rush standards to REC too quickly -- that just
> wastes time and makes us look cheap.
> 
>    * I think any standard is a poor standard if it proceeds to REC
> without implementations, validators, or some other public concrete
> evidence that it's _ready_. Having the WG Chair say 'I can feel it in
> my bones' does not count. [that is, I'm firmly in the must-implement
> rather than should-implement camp]
I agree -- always have -- that RECs are better with the prior implementation
experience.  The "should" is a very strong one, in that we need to be
convinced that it is worth promoting a document to REC without
implementations.

>    * I think a standard is a poor standard, if it normatively depends
> on still-draft standards. [this shouldn't need saying]
> 
>    * If no-one can be bothered to implement a pre-REC standard, then
> that is evidence that that standard should not exist. [sure, there are
> plenty of people for whom this would be too early to implement, but if
> there aren't any early-adopters, what's the point?]
> 
>> It was understood from the beginning of IVOA that standards would
>> evolve and
>> that first versions would likely need to change.  I found something
>> I wrote
>> back in March 2004, concerning the registry standards, that I think
>> remains
>> relevant today:
>> 
>> We need to reach agreements quickly, build to them, and iterate to
>> improve
>> them.  We need to be willing to build and revise, or even in some
>> cases
>> throw away and start again.  We have set up a standards process that
>> recognizes change, and is intended to be responsive to change. I do
>> not
>> believe we can afford to set 2-3 year schedules for reaching
>> consensus. If
>> it takes this long, the community upon which we will utimately
>> depend for
>> support will see us as purveyors of snake-oil, and will blow us off.
> 
> I agree with the sentiment, and agree with the need for iterations and
> (particularly) discarding when necessary.  Standards do need updating
> and improving.  But that doesn't imply making v1.0 standards throwaway.
I don't know where this idea is coming from, Norman.

> The problem is that we _do_ appear to have 2-3 year schedules for
> reaching consensus.  If that's too long (and I'd agree with 'snake
> oil' for ambitious proposals in permanent draft), then we need to
> shorten the process earlier, perhaps by insisting that v1 proposals
> are less ambitious, but not by skipping an important late stage.
> 
> A 'standard' which turns out to be unimplementable is also snake oil,
> and I'd argue that it's _worse_ than one in permanent draft, because
> at least the permanent doesn't pretend to be a fully thought-through
> document.
> 
> I appreciate that this may be a hard sell to funders (and I say this
> while up to the elbows in my half-written grant proposals).  You know
> what you (plural) signed up for!
A bit more on what funders expect... with a US perspective.  At this point I
don't think NSF or NASA care the least about how many standards we agree
upon in the IVOA.  In fact, they would rather see development efforts going
into things that are more visible to astronomers and that directly have an
impact on research.

Do we have enough standards to support VO research tools?  For the most
part, I'd say yes.  Further iterations will enhance capabilities and exploit
new technologies.  So the standards effort should not go dormant, but the
balance between standards development and tools, interfaces, documentation,
robustness, etc., needs to change in favor of the latter.

cheers,
Bob

> All the best (and good luck),
> 
> Norman
> 
> 
> [1] http://www.w3.org/2005/10/Process-20051014/tr.html#cfi