TAP RFC [VOSI]

[Douglas Tody]

Hi Alberto -

Did you see my proposed compromise approach just posted today?
It may provide a nice solution to this problem of the "real" and
"meta" operations having a different character, as well as a nice
separation of the basic service pattern common to all services from
the service-specific operations which we layer on top of this.

While getCapabilities has not yet been fully specified in SSA it
is needed as a client operation (as well as for the more registry
oriented VOSI funtionality), and is a defined part of the interface.
We will need it for advanced client use cases such as we have already
seen with TSAP and SLAP, and it would nice to have it available to
client apps via a consistent interface pattern.

The more important point is that REQUEST and VERSION are already in use
for what you refer to as "real" (data oriented) operations.  Hence this
pattern is already in use and implemented in both client and server
code, and has been for a couple of years now.  It works well for the
object-oriented/RPC interface model which is what is most natural for
the DAL interfaces since they define complex operations upon complex
objects such as images and spectra.  The resource-oriented REST model
also works very well for a certain class of use-cases, mostly where
we want to describe persistent state (job list, file system, etc.).

The biggest issue with the REST approach (which I think I was
the first to suggest within VO as an alternative to SOAP) is the
logical model.  At the risk of oversimplification, REST models a
possibly stateful resource of some sort much like a filesystem.
With something like /proc in Unix we see that this can be very
successful in providing a simple way to represent state, and obviously
it works very well for actual files (or for Web pages or "resources"
in HTTP).  But for objects where we are more interested in actions
than in describing state the OO model defining methods upon objects
(operations with parameters) is preferable.  REQUEST with params
is very straightforward; it is merely a method with parameters.
If we go down the uber-REST route trying to use it everywhere then I
suspect we will fall into a very old failure mode where an otherwise
useful technology fails because it is being applied too widely.
Our HTTP-based Web services should support both REST and OO depending
upon what is most natural for the problem being addressed (and perhaps
even alternatives such as SOAP in some other cases as well, e.g. where
we have very large interfaces).

> REAL REQUESTs could be posed in sync or async mode.
> META operations are just meta operations that immediately return something 
> (META+async not foreseen).

Yes.

> In the above-mentioned context, could you please elaborate more on this 
> sentence:
>>> if we remove REQUEST then it just breaks the REQUEST/VERSION
>>> based interface model and there is no way of getting around it for
>>> client applications which use that model.

We want to achieve interface consistency, at least within a certain
domain of functionality within an interface.  Hence we would like to
be able to tell a user writing client applications that 1) here are
the service operations defined for this type of object, each with its
own capabilities and parameters, and 2) to compose the URL required
to invoke such an operation using the HTTP protocol you just take the
baseURL, operation name, and operation parameters and follow the given
composition rule.  We want to avoid special cases.  VOSI and UWS are
not an issue as a science application will rarely see these anyway,
as they are system level functionality likely to be handled by the
client toolkit used by the application.

> Are you mentioning a real substantial problem or is it "just" that is not 
> nice to break the symmetry?

I think interface consistency across a family of interfaces is more
than just "nice".  Users will expect this, and it promotes code reuse
in service frameworks and client-side libraries.

> (Little parenthesis that adds to my confusion regarding all this:
> I never understood why it is up to the client to specify sync or async;
> how can the client know in advance if a query CAN be answered quickly by a 
> given server? IT CAN NOT!
> I would think it is up to the server to decide whether to provide back a sync 
> or async response.

There is some truth in this but it is not that simple.  The service
interface is quite different for sync and async, with sync being vastly
simpler on the client side (hence many user apps will try to use this
unless forced to do otherwise at gunpoint).  If we care at all about
users who write code we should try to provide sync where we can, even
if an operation might fail and return a error status.

The exception to this is that a sync request could (if we decide to
provide it) auto-initiate a job and return a different type of status
response.  SIAV1 actually mentions something like this, where a getData
could return a status code indicating that the request was accepted
and the image was being staged so the client should try again later
to fetch the image.  This sort of approach can provide a middle ground
in some cases.

 	- Doug






On Sun, 27 Sep 2009, Alberto Micol wrote:

>
> Dear Doug,
>
> Before casting my vote, I'd like to understand better the issue you 
> mentioned.
> So far SSA has "reserved" the various operations, including getCapabilities,
> which means that such operation (though depicted in the SSA) is not yet in 
> use.
> Hence, formally, there is no risk of compromising existing operational 
> services by choosing "resources" instead of "requests".
>
> Currently used REQUEST's values (in SSA) are: getData and queryData.
> REQUEST is therefore used to specify REAL operations on the data.
>
> getCapabilities, getAvailability, getTableMetadata are (or, better, will be) 
> META (so to say) operations.
>
>
> REAL REQUESTs could be posed in sync or async mode.
> META operations are just meta operations that immediately return something 
> (META+async not foreseen).
>
>
> In the above-mentioned context, could you please elaborate more on this 
> sentence:
>>> if we remove REQUEST then it just breaks the REQUEST/VERSION
>>> based interface model and there is no way of getting around it for
>>> client applications which use that model.
>
> I do not appreciate the risk you are alerting us about, hence I'd like to 
> hear more about it.
> Are you mentioning a real substantial problem or is it "just" that is not 
> nice to break the symmetry?
>
> (Little parenthesis that adds to my confusion regarding all this:
> I never understood why it is up to the client to specify sync or async;
> how can the client know in advance if a query CAN be answered quickly by a 
> given server? IT CAN NOT!
> I would think it is up to the server to decide whether to provide back a sync 
> or async response.
> Sync and async should hence be output parameters, not input. )
>
> Many thanks for any clarification,
> Alberto
>
>
> On 24 Sep 2009, at 19:14, Doug Tody wrote:
>
>> On Thu, 24 Sep 2009, Guy Rixon wrote:
>> 
>>> I vote for using resources.
>>> I have a TAP implementation that has VOSI both as resources and via the 
>>> REQUEST parameter.
>> 
>> Hence it is quite easy to support both approaches in an actual
>> implementation.  And since with VOSI an opaque URL is used to define
>> the VOSI endpoint in the registry, if an implementation wants to add a
>> REST URL to implement getCapabilities or other VOSI operation in their
>> service, applications which are based upon the VOSI interface will
>> automatically take that route.  So this is available in either case.
>> But if we remove REQUEST then it just breaks the REQUEST/VERSION
>> based interface model and there is no way of getting around it for
>> client applications which use that model.
>>
>> 	- Doug
>> 
>>> 
>>> On 24 Sep 2009, at 15:07, Patrick Dowler wrote:
>>> 
>>>> On Monday 14 September 2009 02:00:09 Keith Noddle wrote:
>>>>> In consultation with Christophe, I therefore propose that we extend the
>>>>> TAP RFC to 25th September to allow the above details to come together.
>>>> Last two days, last issue from rfc page:
>>>> In the current spec, there are two resources under the base URL and VOSI 
>>>> are
>>>> accessed via one of them:
>>>> async
>>>> sync
>>>> sync?REQUEST=getAvailability
>>>> sync?REQUEST=getCapabilities
>>>> sync?REQUEST=getTableMetadata
>>>> There is a proposal to access VOSI as resources off the base URL (more 
>>>> pure
>>>> REST style vs standard DAL REQUEST param). The motivation is to move VOSI 
>>>> out
>>>> from under sync so that both async and sync could be made optional (both
>>>> required in PR). The new resource structure would be:
>>>> async
>>>> sync
>>>> availability
>>>> capabilities
>>>> tableMetadata
>>>> We have had many discussions on this topic, made compromises all around, 
>>>> and
>>>> never reached a unanimous agreement. There are good reasons for both
>>>> approaches. The current draft is consistent with DAL style as embodied in
>>>> SSA1. The proposed makes it cleanly possible to have optional async/sync
>>>> execution of jobs (currently we require both, but we could relax this in 
>>>> a
>>>> future version of TAP very easily if needed).
>>>> To be very clear: both of these work fine, so there is nothing broken and 
>>>> to be
>>>> fixed. This is a stylistic issue (RPC vs REST) and about making plausible
>>>> future changes as easy and painless as possible. Given that, I don't see 
>>>> any
>>>> point in dredging up all the arguments again.
>>>> The crude WG concensus is that more people favour using resources over 
>>>> the
>>>> REQUEST parameters, but I would like to call a vote to verify that and 
>>>> will
>>>> change the draft to match the WG decision. If you care, please respond to 
>>>> this
>>>> message and chose one of:
>>>> use REQUEST parameter
>>>> use resources
>>>> I will tally and start rewriting on Monday (28th).
>>>> -- 
>>>> Patrick Dowler
>>>> Tel/Tél: (250) 363-0044
>>>> Canadian Astronomy Data Centre
>>>> National Research Council Canada
>>>> 5071 West Saanich Road
>>>> Victoria, BC V9E 2M7
>>>> Centre canadien de donnees astronomiques
>>>> Conseil national de recherches Canada
>>>> 5071, chemin West Saanich
>>>> Victoria (C.-B.) V9E 2M7
>>> 
>>> 
>