IVOA Document Standards RFC V1.2

[Robert Hanisch]

Thanks Mark, for the (as usual) thoughtful comments.

First, let me quote myself in the e-mail message I sent earlier today:

"One might argue that if one or more VO projects invests the efforts into
developing the standard itself, it follows that they would put effort into
implementations and validators.  But this could put a lot of pressure on
schedule and resources of individual projects, and could lead to major
delays in the promotion process."

I agree completely with you that doing implementations in parallel with
standards development is the best way to go, with the obvious exception that
some of the IVOA standards, as you point out, are not really about software
implementations.

But making the implementations and the validator mandatory for promotion is
where I, and Christophe and Francoise, feel it is necessary to pause.  For
example, if right now we were to enforce this requirement on TAP, with the
resource constraints we have in several of the major projects, we would be
forced to stop the promotion process.  This would be an awful situation for
IVOA, I think.  We want prototypes to inform the standards development
process, for sure, we SHOULD have them, but from a management perspective I
think we have to avoid boxing ourselves into a corner.

It is also a bit ambiguous just how one proves that implementations are
interoperable.  So far we have taken people's word, but IVOA has no means to
independently test these assertions.

I would probably agree fully with you if, say, IVOA were a dues-based
membership organization, with its own financial resources to build
validators, or test those developed by others.  But I see a distinction,
albeit a subtle one, between the projects' willingness to contribute to
development of a standard and their ability to provide supporting software
and validators on a timescale that does not jeopardize the promotion
process.

Finally, in any case, we have a revision system that is there to deal with
problems and shortcomings found in earlier versions.  If V1.0 of a standard
ends up having problems -- unsupportable features, inconsistencies, etc.
(and we have had this experience) -- then we fix the problems in V1.x or
V2.0.

So, as I said to begin with, I do believe we need to strengthen the
language, but I am very reluctant to make things mandatory.

Bob

On 7/2/09 1:33 PM, "Mark Taylor" <m.b.taylor at bristol.ac.uk> wrote:

> On Thu, 2 Jul 2009, Robert Hanisch wrote:
> 
>> On behalf of the Standing Committee on Standards and Processes, I have
>> posted responses to the comments that came in during the recent RFC.  (And I
>> apologize that these responses were not posted in a more timely manner!)
>> Please see the RFC page for details.
>> 
>> http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/DocStdRFC2
>> 
>> In order to make sure that there is broad review and understanding of this
>> document, as it is the basis for all document promotion processes in the
>> IVOA, I am posting this overview to the stdproc and interop distribution
>> lists.  Detailed discussion is best carried out on the stdproc list, but
>> that list has not been used much for a few years.  Further comments can also
>> be posted on the RFC page above.
>> 
>> Several comments on the RFC were posted regarding the need for interoperable
>> implementations and validators.  The Committee agrees that some
>> strengthening of the language in the document on this matter is warranted.
>> In particular, item 4 concerning the promotion from Working Draft to
>> Proposed Recommendation currently reads:
>> 
>> "4.     Each feature of the Working Draft has been implemented. Preferably,
>> the Working Group should be able to demonstrate two interoperable
>> implementations of each feature, and validation tools should be available.
>> If the chair of the Working Group believes that broader review is critical,
>> the chair may advance the document to Proposed Recommendation even without
>> adequate implementation experience.  In this case, the document status
>> section should indicate why the chair promoted the document directly to
>> Proposed Recommendation.  A report describing the implementations or any
>> associated validation tools may be published as a Note, or may be documented
>> as part of the Request for Comments (see below)."
>> 
>> Several people argued that having at least two interoperable implementations
>> and at least one validator service should be mandatory.  As noted in the RFC
>> response, however, the Committee feels that making software implementations
>> mandatory is beyond the scope of the IVOA.  One might argue that if one or
>> more VO projects invests the efforts into developing the standard itself, it
>> follows that they would put effort into implementations and validators.  But
>> this could put a lot of pressure on schedule and resources of individual
>> projects, and could lead to major delays in the promotion process.
>> 
>> I think we can remove the "Preferably" language above, which seems very
>> soft, and make the text a SHOULD in the canonical sense.  Similarly the
>> "may" in the last sentence can become SHOULD.
>> 
>> If we are to embrace the idea that implementations and validators are to
>> become mandatory, it would be important to hear from the project managers as
>> to whether or not they are willing to support this requirement.  Speaking as
>> project manager of NVO I would have some reticence, especially now when we
>> have no resources to follow through.
>> 
>> Please follow up with comments to stdproc at ivoa.net, and if you wish to
>> participate in this discussion make sure you are on that list.
> 
> Bob,
> 
> thanks for the response.  I would like to take issue with it on two
> main grounds: firstly that I don't find the 'out of scope' argument
> convincing given what is clearly in scope for the IVOA, and secondly
> that I believe the effort saved by not requiring these things is
> in fact illusory.
> 
> From the response on the RFC page:
> 
>   "The problem we see with making such implementations mandatory is
>    really one of scope and authority of the IVOA. The IVOA has no
>    funding to support such activities, and thus the IVOA has no basis
>    for demanding such things. If we were in position to fund such work,
>    that would be one thing, but were are not; we rely on the goodwill
>    and resources of the VO projects.
>       -- BobHanisch on behalf of the Committee"
> 
> I don't follow the argument that since the IVOA does not fund writing
> of implementations or validators, it cannot mandate them.
> Exactly the same could be said about writing the text of the
> standards documents themselves.  Writing and reviewing the standards
> can require a great deal of time and effort from contributors,
> as I'm sure that you and many readers of this list know first hand,
> but the IVOA feels able to state requirements about these without
> providing any funding for the authorship or review process.
> 
>   "If we make interoperable implementations and a validator mandatory,
>    how do we make sure that they get developed? Who validates the
>    validator? We have seen already that even our most experienced software
>    engineers have written validators with errors in them.
>       -- BobHanisch on behalf of the Committee"
> 
> We make sure that they get developed in the same way that we
> (in principle, at any rate) currently ensure the quality of the written
> standards: by public review from those IVOA members who feel
> sufficiently motivated to participate, and by witholding Recommendation
> status until agreement among those participants has been reached.
> Regarding validating the validators: obviously I'm not suggesting
> that associated software with a bug count which is provably zero is
> to be a requirement.  As with the standards text, the authors make
> their best efforts in good faith, and if others scrutinising their
> work believe there are problems with it these are addressed by
> discussion.  Yes this system is open to abuse (people claiming they
> have a validator when it doesn't do much validation), but we're
> mostly grown ups here.
> 
> The measure of success of the IVOA is surely not how many standards
> it can churn out, it's more about the amount those standards are used,
> which in turn has to depend on the existence and quality of the
> implementations of those standards.  Surely, a standard only becomes
> worth anything when at least one usable implementation exists.  Here is
> the most important part of my argument:
> 
>    Postulate: If your goal is to reach an implementation, it is much
>       easier (you can do it both faster and better; also, for the
>       benefit of project managers, cheaper) to write the
>       implementation and validation alongside the standard than it is
>       to write the standard on paper, get it set in stone, and then
>       start to write the software.
> 
> The reason is that when you write it down on paper, there is a very
> good chance that you will write things which are contradictory or
> ambiguous or conflict with other established standards or are hard or
> impossible to implement.  These are things you will stand a much much
> better chance of spotting if you are writing software alongside the
> text.  The arguments for implementations and validators are similar
> but slightly different; I can elaborate on request.
> 
> Based on my experience I believe that the postulate above is true:
> I would like to know whether the DocStd committee believe it is
> true, false, or just not relevant here.
> 
> Finally, I should concede that the software requirement is clearly
> inapplicable in some cases - for instance I'm not suggesing that
> a validation suite be required for the DocStd document itself.
> So some concession about applicability should be included in the
> discussion of requirements.
> 
> I'm happy to hear other people's views on this ...
> 
> Mark