[TAP] sync vs async

[Linde, A.E.]

That's just more of the same, Doug. Leave all the project comparisons alone - they're irrelevant and the way you present them is emotive whether you acknowledge it or not, even more so below.

Stick to the use cases and present them. If you have data service people who are unable to write asynch access to their data and unwilling to use such libraries to provide it then list them; get them onside to present their case. That then is a use case for having equal synch and asynch protocols. Stick to the use cases and technical arguments - leave the project comparisons out because you always make it sound like denigration of others' work.

T.


On 09/03/2009 04:40, "Douglas Tody" <dtody at nrao.edu> wrote:

Hi Tony -

On Fri, 6 Mar 2009, Linde, A.E. wrote:

> On 05/03/2009 16:19, "Doug Tody" <dtody at nrao.edu> wrote:

> > The main difference between NVO and the other VO projects seems to be
> > that we also consider basic data access by scientists and science
> > application writers to be a high priority, and want to provide
> > simplified access to support such applications.
>
> Hi Doug, I wasn't commenting on the case for or against any aspect of the
> standard, just pointing out that your language was tending to the emotive
> rather than discursive which I thought might be unhelpful.

My comments were not emotional, nor did I intend to suggest that
any of our projects are not trying to fully serve the user community.
Rather I was trying to make a serious observation about how our project
strategies differ, which we clearly see in negotiations on standards
such as TAP.  This does not mean that any one approach is better, just
that any international standard has to represent all the major projects
and use cases involved.

At the risk of over-generalizing things I suggest we have two different
project strategies or styles in play here.  Lets call these the "big
system" approach, and the "open system" approach.  It is not that
simple of course and there are elements of both approaches in all our
VO projects, but I believe there is some validity in classifying things
this way.

In the "big system" approach we set out to design and build the entire
system, or most of it anyway, including architecture, interfaces,
services, and major client applications such as portals, as one overall
effort.  The same Project may do most of this work, e.g., delivering
service frameworks to data providers so that they can put up their data,
and providing an overall system which is directly usable by the end user,
via Web portals, workflows, client toolkits, etc.  This is probably the
quickest and most efficient way to get a sophisticated system up fast,
but runs the risk of not getting sufficient user feedback and buy-in
until well into the system development effort, plus the approach is
perhaps too organized and centrally controlled to work well for a broad
international collaboration which must federate a wide variety of systems
and projects serving differing user communities.

In the "open system" approach we do not try to define and build the
full system in one go as one would for a conventional software project,
rather we start with a well thought out architectural concept and early
on try to define and deploy fairly simple facilities consistent with
this architecture, with a high priority to get community involvement and
investment in the project early on.  We may not be able to do all that
much initially, but we can iterate, and we make sure that the simple
facilities provided early on are consistent with a more general longer
term architecture.  Once the community gets used to using the fairly
simple astronomy-oriented facilities provided, we can introduce more
complex technology relatively painlessly provided a simple entry point
into the system continues to be provided.  In fact once the users begin
using the system for serious applications they will probably demand the
more sophisticated facilities.  Early involvement of the user community
will likely help define what is required as the system evolves.

We have some experience within astronomy with both of these approaches
to large system software development.  The big integrated system
approach can work very well for well defined projects, especially when
the participation of the target user community is ensured (e.g., a
telescope system or the reduction system for an instrument or collection
of instruments where the users have no place else to go).  Most large
astronomy projects are of this type.

Science software, that is software which astronomers use for their
personal research, is quite another matter.  In this arena it is not
unusual for a large funded project to build a lot of software which ends
up getting little use by the community.  The users instead may end up
using some fairly crude program written by some astronomer-programmer
down the hall, which may be pretty awful in software engineering terms,
but which provides exactly what the user needs to process or analyze
their data.  The most successful systems will provide both such simple
well-targeted facilities as well as more powerful facilties which are
not available elsewhere.

This is getting to be a long email so I will end by simply stating
that there can be good reasons for not merely going for what we "know"
will be required to do the full, general problem.  When it comes to
science software the users can just go elsewhere if the stuff they see
looks like overkill for their project - they may make that choice in 30
seconds after we invest 50 manyears in a project.  A multi-level approach
with a simple entry point backed up by more fully-featured facilities,
if the user gets motivated to dig deeper, is almost always a wiser path.
We need both sophisticated end-user facility software (cross match portals
etc.) as well as simpler user-contributed but familiar and well-tested
software, including integration of legacy applications.

Infrastructure such as the DAL interfaces (including TAP), which are used
by applications to access science data, must serve both types of use case.
Hence we need a generalized interface addressing the harder applications
as well as a basic subset which provides simple facilities for the most
common use cases.  This makes it possible for a user to get started in
10 minutes with a simple synchronous GET, but they can do more if and
when they find that this is insufficient.  We use things like generic
table conainers (e.g., votable) to leverage existing astronomy tools,
rather than native XML since high level users have little use for things
like automatic code generators or data bindings (although there may be
unusual cases which warrant this).  We need to be able to implement new
facility-grade Project software to provide powerful new tools, but to
be successful we also need to engage the users to adapt or add their
own software.

> I completely agree that the standards need to address the use cases of
> all the projects involved in the IVOA. But we also need to be open to
> alternatives about how those use cases are addressed.

Indeed.  We respect your choices and are willing to support them in
these standards - we need the advanced capabilities in NVO eventually
in any case.  You should seek to understand our approach as well.
These approaches need not be in conflict, they reflect the same
underlying architecture.  It is better to have a multi-level approach
in any case.

        - Doug



--
Tony Linde
Project Manager
Department of Physics & Astronomy
University of Leicester
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ivoa.net/pipermail/dal/attachments/20090309/64c13e3c/attachment-0003.html>