ObsCoreDM and DAL for proprietary data

Wed May 19 18:29:34 PDT 2010

Hi Petr -

Just some quick comments (and thanks for the interest, we are sorry you
couldn't make it here).

We did remember proprietary data - some observatory data policies allow
the metadata to be public before the data is publically available.
In this case the data would be visible but the access_url would be NULL.
There might be some optional metadata to indicate when the data would
be publically available.  Secure PI access is a more advanced issue
not addressed by ObsTAP, but there are obvious possibilities, e.g. use
an authenticated link (HTTPS with cert), and then the URL might not be
null anymore.

> I am not sure how the ObsCoreDm should be linked to whole TAP stuff
> (including UWS async ....) - what I do not understand so far is the role
> of current SIA and SSA servers - should everyone support TAP access in
> ADDITION to the SSA/SIA to be able to return the basic

TAP is a separate service and does not affect SIA/SSA directly.
However it would be sensible to put the SIA/SSA metadata in a table in
which case it could be queried with TAP as well as used to drive the
typed services.  Also, ObsTAP is essentially the DAL GDS query, which
can be used to do global data discovery followed by a call to a typed
service for the more advanced VO data access.

Yes, a 1D FITS spectrum could be represented by this scheme.  The most
natural representation would probably a dataproduct_type of spectrum.1D
with access_format fits.image.

	- Doug

On Thu, 20 May 2010, Petr Skoda wrote:

>
> Hi all,
>
> although I had not been following the CharacDM nor ObsDM for about a year
> or more (sorry)  I am glad that some of the ideas we started with during
> the special "hotel session" during the Theory/GRID workshop in Garching
> and later in Baltimore seem to get to the realisation phase. Thanks to all
> your effort on it!
>
> As I have been following the Interop almost in real time on wiki
> (regretting I am not able to be with you - due to financial reasons), I
> have just briefly read the proposal of the WD 1.0-20100517 and all the
> presentation available on wiki.
>
> Concerning this I have comments to both the presentation of Mireille (on
> wiki it is named ObstapConvergence.pdf) and Francois (DataLink.pdf).
> It concerns the access_url and access_format in case of proprietary data.
>
> I think the very important science use case not mentioned in the Dave
> Schade's list is the action taken in case of proprietary data - very
> common in stellar astronomy with unknown and often unlimited (lifetime ;-)
> proprietary period.
>
> The question is - should the astronomer be allowed to discover only
> public data ? The most important information - e.g. for proposal or
> collaboration purposes is:
>
> "had someone observed my target ? In given time (e.g. has someone spectra
> taken just after nova outburst ?) in given spectral range (The behaviour
> of line XY is crucial for physics) . Or what was a time, band ... coverage
> of observation with given instrument .....
>
> In this case I just want to know the EXISTENCE and COVERAGE (or whole
> characterization) of the observation - NOT THE DATA ITSELF !
> Very important would be the curation part - best the direct contact to
> people who posses the data sets.
>
> With respect to this it is necessary to make clear what to do with
> mandatory access_url in case of data_rights=Proprietary (which is
> optional)
>
> Should the service respond by some "dummy URL" causing some error like
> "not found" or better with some "null" contents (that will not show
> anything, zero length...)
>
> Or should it return some verbose marker of the coverage (e.g. in case of
> proprietary imeage would show the empty rectangle or whole footprint - in
> Aladin - with possible text - "proprietary, Sorry" ;-) In case of spectra
> in SSA client just horizontal line at the 0.0 extending the wavelength
> ranges covered ??????
>
> The another issue to consider is - what to do in case the privileged
> people (e.g. PI) would like to use VO Tools to work with public data AS
> WELL AS with their PROPRIETARY (which would allow many VO-reluctant
> sceptics to publish their data and learn the VO tools).
>
> I think that the simple and feasible way could be based on similar idea as
> the GPS maps unlocking. (e.g. for Garmin) - You get the full product but
> without special key you see only contours of the maps, shorelines, lakes
> and the very coarse datail - e.g. large higways - not tourist tracks,
> small roads etc ...
>
> So the server could recognize the proprietary data and encrypt the real
> contents, still sending all metadata. The client would transparently
> identify the existence of encoding from metadata (and in the list of
> datasets discovered would place a "small key" icon next to them).
> Clicking on the key the description of details would occur - e.g. the
> expiration of proprietary period, contact, information what key is
> required etc ...
>
> One option is then:
> Inside this "more info label" could be the link or window where to put the
> key to unlock the data contents (only now would go the request to stream
> the real data ).
>
> Another option:
> The best way how to do this seems to me very similar to
> PGP/GPG.
>
> First importing some "restricted keys (known to the PI's or whole team
> -e.g. department) to the local "keyring".
> The client (e.g. Aladin or Splat) could check the possession of the keys
> trasparently while loading the ObsCore Curation  metadata (before creating
> the key icon) and use the key directly to unlock the data stream (which
> would be requested on-the-fly  for matching keyID). The server would send
> the proprietary data already encrypted.
>
> I feel that this mechanism could be easily added to clients as a part of
> understanding the ObsCore structure (best in form of some library) and the
> servers would just add some additional function for GPG encryption which
> would be used for the delivery of data marked in internal database as
> private/proprietary.
>
> I think this problem is loosely connected with Data link issues and the
> Alberto's comments to NULL or NOT NULL question ;-)
>
> I want to emphasize that this is not competing with Single-Sign-On stuff
> and the whole GRIDs security issues and  community servers in Astrogrid
> CEAs etc ...
>
> I am not sure how the ObsCoreDm should be linked to whole TAP stuff
> (including UWS async ....) - what I do not understand so far is the role
> of current SIA and SSA servers - should everyone support TAP access in
> ADDITION to the SSA/SIA to be able to return the basic
> provenance/characterization fulfilling some of the David's simple use
> cases ?
>
> But I see the importance of some controlled access to
> proprietary/restricted access data as a crucial step for disseminating the
> VO awareness and forcing the "hands on" experience of the conservative
> part of community (which suffers of the data-jealousy) as well as
> different closed consortia.
>
> ----------------------------------
> Another comments concerns the mime_type or obs:Access.Format
> As was already mentioned in Mireille's presentation, there seems to be a
> rich list of values (including FITS.GZ, TEXT, TAR ....)
> And Doug had mentioned in his examples (obsdp.pdf, page 5 and 9)  the
> fits.image.gz and even fits.uvfits ....
>
> So I am glad that the strict restriction imposing the FITS in VO means the
> BINTABLE FITS (which causes a lot of confusion in accessing some spectra -
> e.g. ESO FITS spectra  in VOSPEC) will be broken.
>
> Sorry to bother with this again ;-) but I have to ephasize again:
>
> 1) the 1D Image FITS spectra in the linear or log rebinned form (CRVAL1,
> CDELT1) as produced by most IRAF/MIDAS users/pipelines are the substantial
> part of optical spectra produced by ground based telescopes.
> 2) the highly demanded service among stellar astronomers (including
> students) of VO tools is the possibility to use the VO to discover
> spectra finally donwloaded and processed in IRAF splot  - the final stage
> is then the wspectxt conversion to ASCII table for advanced processing....
>
> I just would like to remember the necessity of distinguishing the 1D FITS
> spectra from Bintable FITS in Access.Format when the ObscoreDM is
> finalized.
>
> Best regards
>
> Petr
>
> *************************************************************************
> *  Petr Skoda                         Phone : +420-323-649201, ext. 361 *
> *  Stellar Department                         +420-323-620361           *
> *  Astronomical Institute AS CR       Fax   : +420-323-620250           *
> *  251 65 Ondrejov                    e-mail: skoda at sunstel.asu.cas.cz  *
> *  Czech Republic                                                       *
> *************************************************************************
>