IVOA Document Standards RFC V1.2

[Mark Taylor]

On Thu, 2 Jul 2009, Robert Hanisch wrote:

> On behalf of the Standing Committee on Standards and Processes, I have
> posted responses to the comments that came in during the recent RFC.  (And I
> apologize that these responses were not posted in a more timely manner!)
> Please see the RFC page for details.
> 
> http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/DocStdRFC2
> 
> In order to make sure that there is broad review and understanding of this
> document, as it is the basis for all document promotion processes in the
> IVOA, I am posting this overview to the stdproc and interop distribution
> lists.  Detailed discussion is best carried out on the stdproc list, but
> that list has not been used much for a few years.  Further comments can also
> be posted on the RFC page above.
> 
> Several comments on the RFC were posted regarding the need for interoperable
> implementations and validators.  The Committee agrees that some
> strengthening of the language in the document on this matter is warranted.
> In particular, item 4 concerning the promotion from Working Draft to
> Proposed Recommendation currently reads:
> 
> "4.     Each feature of the Working Draft has been implemented. Preferably,
> the Working Group should be able to demonstrate two interoperable
> implementations of each feature, and validation tools should be available.
> If the chair of the Working Group believes that broader review is critical,
> the chair may advance the document to Proposed Recommendation even without
> adequate implementation experience.  In this case, the document status
> section should indicate why the chair promoted the document directly to
> Proposed Recommendation.  A report describing the implementations or any
> associated validation tools may be published as a Note, or may be documented
> as part of the Request for Comments (see below)."
> 
> Several people argued that having at least two interoperable implementations
> and at least one validator service should be mandatory.  As noted in the RFC
> response, however, the Committee feels that making software implementations
> mandatory is beyond the scope of the IVOA.  One might argue that if one or
> more VO projects invests the efforts into developing the standard itself, it
> follows that they would put effort into implementations and validators.  But
> this could put a lot of pressure on schedule and resources of individual
> projects, and could lead to major delays in the promotion process.
> 
> I think we can remove the "Preferably" language above, which seems very
> soft, and make the text a SHOULD in the canonical sense.  Similarly the
> "may" in the last sentence can become SHOULD.
> 
> If we are to embrace the idea that implementations and validators are to
> become mandatory, it would be important to hear from the project managers as
> to whether or not they are willing to support this requirement.  Speaking as
> project manager of NVO I would have some reticence, especially now when we
> have no resources to follow through.
> 
> Please follow up with comments to stdproc at ivoa.net, and if you wish to
> participate in this discussion make sure you are on that list.

Bob,

thanks for the response.  I would like to take issue with it on two
main grounds: firstly that I don't find the 'out of scope' argument
convincing given what is clearly in scope for the IVOA, and secondly
that I believe the effort saved by not requiring these things is
in fact illusory.

>From the response on the RFC page:

  "The problem we see with making such implementations mandatory is
   really one of scope and authority of the IVOA. The IVOA has no
   funding to support such activities, and thus the IVOA has no basis
   for demanding such things. If we were in position to fund such work,
   that would be one thing, but were are not; we rely on the goodwill
   and resources of the VO projects.
      -- BobHanisch on behalf of the Committee"

I don't follow the argument that since the IVOA does not fund writing 
of implementations or validators, it cannot mandate them.  
Exactly the same could be said about writing the text of the
standards documents themselves.  Writing and reviewing the standards 
can require a great deal of time and effort from contributors, 
as I'm sure that you and many readers of this list know first hand, 
but the IVOA feels able to state requirements about these without 
providing any funding for the authorship or review process.

  "If we make interoperable implementations and a validator mandatory,
   how do we make sure that they get developed? Who validates the
   validator? We have seen already that even our most experienced software
   engineers have written validators with errors in them.
      -- BobHanisch on behalf of the Committee"

We make sure that they get developed in the same way that we
(in principle, at any rate) currently ensure the quality of the written
standards: by public review from those IVOA members who feel 
sufficiently motivated to participate, and by witholding Recommendation
status until agreement among those participants has been reached.  
Regarding validating the validators: obviously I'm not suggesting 
that associated software with a bug count which is provably zero is
to be a requirement.  As with the standards text, the authors make
their best efforts in good faith, and if others scrutinising their
work believe there are problems with it these are addressed by 
discussion.  Yes this system is open to abuse (people claiming they
have a validator when it doesn't do much validation), but we're 
mostly grown ups here.

The measure of success of the IVOA is surely not how many standards
it can churn out, it's more about the amount those standards are used, 
which in turn has to depend on the existence and quality of the 
implementations of those standards.  Surely, a standard only becomes 
worth anything when at least one usable implementation exists.  Here is 
the most important part of my argument:

   Postulate: If your goal is to reach an implementation, it is much 
      easier (you can do it both faster and better; also, for the
      benefit of project managers, cheaper) to write the 
      implementation and validation alongside the standard than it is 
      to write the standard on paper, get it set in stone, and then 
      start to write the software.

The reason is that when you write it down on paper, there is a very 
good chance that you will write things which are contradictory or 
ambiguous or conflict with other established standards or are hard or 
impossible to implement.  These are things you will stand a much much
better chance of spotting if you are writing software alongside the
text.  The arguments for implementations and validators are similar 
but slightly different; I can elaborate on request.

Based on my experience I believe that the postulate above is true: 
I would like to know whether the DocStd committee believe it is 
true, false, or just not relevant here.

Finally, I should concede that the software requirement is clearly
inapplicable in some cases - for instance I'm not suggesing that
a validation suite be required for the DocStd document itself.
So some concession about applicability should be included in the
discussion of requirements.

I'm happy to hear other people's views on this ...

Mark

-- 
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/