roadmap 2010-2011

[Doug Tody]

Hi All -

I have some more general comments on SIAV2 inspired by Francois's
mail below, all of which I agree with.

As Francois notes progress on SIA2 has been delayed in the past year
mostly because our small DAL science team was diverted onto the higher
priority ObsTAP project (long planned GDS query).  PQL was similarly
delayed, however that directly depends upon ObsDM so this was probably
just as well.  With ObsTAP/ObsDM soon to stabilize it is time to
resume work on these two high priority and interlinked interfaces.

Replacing a comprehensive SIA (or SSA etc.) specification by multiple
documents is not necessarily such a great idea.  These are science
software interfaces, used by data providers and client application
writers who are often only cursorily familiar with VO technology.
Expecting these folks to read 10 documents and divine how it all
comes together in one specific interface is unrealistic and is not
the way to go.  Rather, we should bring the most important bits
together in one document describing the interface in question,
and refer to these other documents for the details (VOTable, Char,
DAL arch, etc. are examples).  Versioning would also be a serious
problem with the splitting approach, as these documents are likely
to evolve on different schedules.  I do agree however that some of
the details can be left to these more general common documents.

A second problem with the splitting approach is that we are dealing
here with object interfaces, not generic query languages or data
formats.  We need to describe how the VO bits come together to
deal with a specific type of data - image, spectrum, catalog, time
series, etc.  These are all complex data objects with custom data
model attributes and access methods.  How they differ is at least
as important as what they have in common - there is a reason why
50 years of digital astronomy evolved this way.

A third problem with the splitting approach Francois also mentioned,
which is that it would certainly take much longer than what we are
doing now.  Considerable work has already been done on SIA2, and
we already have a sizeable draft spec which outlines most of the
interface.  Some prototyping has already been done.  While work was
dealyed in the past year, good progress has been made on understanding
key problems such as how to deal with cube data.  We are not far
now from completing a first working draft to use for more extensive
prototyping as well as community review, e.g., by the radio community
to review the cube capabilities.

I suggest we have two primary requirements for SIA2:

     o   Upgrade the very successful SIA1 to the DAL2 and DM standards,
         providing consistency with SSA, ObsTAP, and the other modern
         interfaces.  This should preserve the simplicity of SIA1 for
         basic image discovery and access, including some virtual data
         capabilities as we already have in SIA1.

     o   Add support for cube data.  This is *essential* to support
         data from modern instruments - we have no adequate way
         currently to support large radio data cubes in the VO.
         The SIA2 design effort has already indicated that this can
         be done without compromising the basic 2D capabilities, and
         I expect prototyping will prove this.  The key thing is that
         the astronomy Image model is inherently multidimensional;
         we have decades of FITS experience to prove this.

I suggest we have a further round of work on the SIA2 draft spec this
fall, and review the situation in the Dec IVOA interop sessions.
Assuming that the WG decides that a major refactoring of SIA2 is
not warranted, we can shoot for completion of the first full draft
spec in the several months following the interop, and get on with
prototyping and review.

         - Doug



On Tue, 14 Sep 2010, Francois Bonnarel wrote:

> Hi Pat,
> Le 10/09/2010 21:35, Patrick Dowler a écrit :
>>  In the recent past we have had success in bringing more complex standards
>>  to completion by breaking them down into manageable pieces and re-using
>>  existing standards wherever possible. I would like to continue with this
>>  approach and propose the following as a way forward. Obviously there are
>>  many things to collectively consider and decide
>> 
> You wrote in your other mail that you thought this mail will be a little 
> provocative. Bingo! Thank you anyway because  I think the DAL WG will 
> eventually put some attention on SIA2.
> Apparently your assumption is c that SIA2 was not pushed to recommendation 
> earlier because the standard as it is now is too complex.
> My opinion is different. Historically it seems that  IVOA DAL Working group 
> was nether able to push more than one standard at a time... In the stone age 
> we had Cone Search, (2001 ?)
> in the bronze age SIA1 (2002).
> The beginning of history was devoted to SSA (2003/2007) and more recent ages 
> to TAP (2007/2010).
> After that came suddenly ObsTap by a strong commitment from the EXec 
> (2009/2010).
> Anyway ideas for SIA2 are there since a long time. There have been 
> presentations and/or prototypes demos
> at nearly all interops  since fall 2007. But with everybody paying attention 
> to TAP probably this was let in
> the shadow.  There are documents which got  very little comments up to now.
>
> SIAV2 Analysis at Baltimore Interop time: 
> http://www.ivoa.net/internal/IVOA/SiaInterface/SIA-V2-Analysis.pdf
>
> SIAV2 working Draft : 
> http://www.ivoa.net/internal/IVOA/SiaInterface/WD-SIAP-2.0-20091104.pdf
>
> The general landscape of DAL was described here:
> DAL2 Service Architecture and Standard Profile Version 1.0 IVOA Note 21 May 
> 2010.
>
> The main drawback of the latter document is that it doesn't integrate 
> "ObsTap". But ObsTAp
> is nothing else than the DataDiscovery method of the "Generic dataset" as it 
> has been identified
> by intense discussions in the Obstap separate discussion about one year ago.
>
> Two points which are important to me:
>      - before splitting the effort we should all look to what is there, 
> comment and prototype or implement
> and say what is wrong or complex with that. SIA2 is pushing the old SIA1 one 
> concepts and technologies to SSA level
> at least. And it's easy to adapt SIA1services into SIA2.  The main criticism 
> so far has been that  integrating  2D images
> and cubes will be too complex and delay the recommendation. This can be 
> discussed and maybe we can layer in time. I personnaly think than splitting 
> everything  as you propose, Pat, will  be much longer.
>     - SIA2 is really for images and parameters and methods are there to 
> provide actual images (or cubes) not for
> complex datasets or products which may contain or not images like ObsTap is 
> able to do. I well tell more on this
> in commenting Doug Tody's answer number 2 to your email
>
> Cheers
> François
> ...
>
>>  1. remove all query parameters from SIAv2 and actively work on PQL
>>
>>  2. extract the "data linking" from SIAv2 and develop it as an independent
>>  standard that can be used in multiple services
>>
>>  3. extract non-query parameters, error handling, VOTable usage, etc from
>>  SIAv2 and develop it as a base standard for all DAL services; this
>>  standard would be called Data Access Layer Interface, or DALI :-)
>>
>>  4. pass work on the ImageDM to the DM-WG
>>
>>  With this sort of approach, the SIAV2 specification will be the glue that
>>  pulls the required elements together: DALI-sync, ImageDM, PQL, and
>>  DataLink.
>>
>>  Likewise, SSAv2 would require at least DALI-sync, SpecDM, PQL, and
>>  DataLink. When we get to making an update to TAP, it would also slim down
>>  and refer to these and other standards. Other standards (SimDAP, for
>>  example) could be defined to require DALI-sync, SimulationDM, and ADQL.
>>
>>  * overly ambitious roadmap
>>
>>  TAP (or DAL) extension schema (RFC Q4-2010): DAL+Registry need to begin
>>  work on this immediately and finalise at Nara 2010.
>>
>>  PQL (design/proto Q4-2010, RFC Q1-2011): work must start now, looking for
>>  editor, author list will emerge during discussions
>>
>>  DALI (design/proto Q4-2010, RFC Q1-2011): work must start now, looking for
>>  editor, author list will emerge during discussions
>>
>>  DataLink (design/proto Q4-2010, RFC Q1-2011): work must start now with
>>  mailing list discussion of plausibility, use cases, and scope
>>
>>  SIAv2 (prototypes and WD by Q2-2011): depends on ImageDM and above
>>  standards
>>
>>  SSAv1.x (TBD): depends on whether fixes to SpecDM require it and on
>>  urgency SSAv2 (TBD): depends on SpecDM, PhotDM, above standards, and
>>  requirements
>>
>> 
>
>