<div dir="ltr"><br>TL;DR - CADC implemented a prototype SODA async service that essentially collects a bunch of sync cutout requests into a single job (batch); it is operationally more efficient and not much more complex to implement.<br><br>Long version...<br><br>Picking up with the IRIS image example, I will now show how I have implemented SODA async: what it does and why one might want something like this.<br><br>the links:<br><br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/datalink?ID=caom:IRIS/f212h000/IRAS-25um">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/datalink?ID=caom:IRIS/f212h000/IRAS-25um</a><br><br>includes this SODA async descriptor with the same input params as the SODA sync descriptor.<br><RESOURCE type="meta" ID="soda-b4044ef4-4884-4ee5-9c99-1b0f66b0ad45" utype="adhoc:service"><br> <PARAM name="resourceIdentifier" datatype="char" arraysize="28" <br> value="ivo://<a href="http://cadc.nrc.ca/soda#async">cadc.nrc.ca/soda#async</a>" /><br> <PARAM name="standardID" datatype="char" arraysize="33" <br> value="ivo://<a href="http://ivoa.net/std/SODA#async-1.0">ivoa.net/std/SODA#async-1.0</a>" /><br> <PARAM name="accessURL" datatype="char" arraysize="*" <br> value="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async</a>" /><br> <GROUP name="inputParams"><br> <PARAM name="ID" datatype="char" ucd="" arraysize="*" value="ad:IRIS/I212B2H0" /><br> <PARAM name="POS" datatype="char" ucd="obs.field" arraysize="*" value="" /><br> <PARAM name="CIRC" datatype="double" ucd="obs.field" unit="deg" xtype="circle" arraysize="3" value=""><br> <VALUES><br> <MAX value="140.63049941314583 0.2007826788236291 8.778341996040131" /><br> </VALUES><br> </PARAM><br> <PARAM name="POLY" datatype="double" ucd="obs.field" unit="deg" xtype="polygon" arraysize="*" value=""><br> <VALUES><br> <MAX value="146.83673628162285 -6.408995958971017 134.3808989000012 -6.370135804011464 134.4242625446688 6.007430601323759 146.8700273500712 5.96918267453771" /><br> </VALUES><br> </PARAM><br> </GROUP><br> </RESOURCE><br> <br>Our SODA async service allows for multiple of any params. While this includes the ID parameter I don't see any particular value in that and if a client is using SODA via a service descriptor, then (as above) they'll have only one opaque ID in hand anyway and would have to go to considerable effort to put together multiple IDs. More on this below.<br><br>So, one can create an async (UWS) job with multiple cutouts (eg search for images that cover a cluster of galaxies and then cutout around all the galaxies in the image). I'll show this example with curl because it is more elaborate:<br><br>input file I can use with curl to post multiple positional cutouts (note the mix of POS, CIRC, and POLY):<br><br>===multicut.txt===<br>POS=circle 140 0 0.1&<br>POS=circle 66 10 20&<br>CIRC=140 0 0.1&<br>CIRC=140 1 0.1&<br>CIRC=70 0 0.2&<br>POLY=140 0 141 1 141 1 141 0&<br>POLY=20 20 21 20 21 21 20 21&<br>ID=ad:IRIS/I212B2H0<br>===<br><br>curl -v --data @multicut.txt <a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async</a><br><br>Look for the Location header and get the job, eg:<br><br>curl <a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async/p1zhcks2z1gfybq0">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async/p1zhcks2z1gfybq0</a><br><?xml version="1.0" encoding="UTF-8"?><br><uws:job xmlns:uws="<a href="http://www.ivoa.net/xml/UWS/v1.0">http://www.ivoa.net/xml/UWS/v1.0</a>" xmlns:xlink="<a href="http://www.w3.org/1999/xlink">http://www.w3.org/1999/xlink</a>"><br> <uws:jobId>p1zhcks2z1gfybq0</uws:jobId><br> <uws:runId /><br> <uws:ownerId xmlns:xsi="<a href="http://www.w3.org/2001/XMLSchema-instance">http://www.w3.org/2001/XMLSchema-instance</a>" xsi:nil="true" /><br> <uws:phase>PENDING</uws:phase><br> <uws:quote>2016-02-29T21:57:43.641</uws:quote><br> <uws:startTime xmlns:xsi="<a href="http://www.w3.org/2001/XMLSchema-instance">http://www.w3.org/2001/XMLSchema-instance</a>" xsi:nil="true" /><br> <uws:endTime xmlns:xsi="<a href="http://www.w3.org/2001/XMLSchema-instance">http://www.w3.org/2001/XMLSchema-instance</a>" xsi:nil="true" /><br> <uws:executionDuration>120</uws:executionDuration><br> <uws:destruction>2016-02-29T22:55:43.641</uws:destruction><br> <uws:parameters><br> <uws:parameter id="POS">circle 66 10 20</uws:parameter><br> <uws:parameter id="POS">circle 140 0 0.1</uws:parameter><br> <uws:parameter id="ID">ad:IRIS/I212B2H0</uws:parameter><br> <uws:parameter id="POLY">20 20 21 20 21 21 20 21</uws:parameter><br> <uws:parameter id="POLY">140 0 141 1 141 1 141 0</uws:parameter><br> <uws:parameter id="CIRC">70 0 0.2</uws:parameter><br> <uws:parameter id="CIRC">140 1 0.1</uws:parameter><br> <uws:parameter id="CIRC">140 0 0.1</uws:parameter><br> </uws:parameters><br> <uws:results /><br></uws:job><br><br>Ok, we have a PENDING job with a bunch of positional cutout params (and one ID value). Now to run the job:<br><br>curl -d 'PHASE=RUN' <a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async/p1zhcks2z1gfybq0/phase">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async/p1zhcks2z1gfybq0/phase</a><br><br>... wait a bit ... and get the job again:<br><br>curl <a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async/p1zhcks2z1gfybq0">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async/p1zhcks2z1gfybq0</a> <?xml version="1.0" encoding="UTF-8"?><br><uws:job xmlns:uws="<a href="http://www.ivoa.net/xml/UWS/v1.0">http://www.ivoa.net/xml/UWS/v1.0</a>" xmlns:xlink="<a href="http://www.w3.org/1999/xlink">http://www.w3.org/1999/xlink</a>"><br> <uws:jobId>p1zhcks2z1gfybq0</uws:jobId><br> <uws:runId /><br> <uws:ownerId xmlns:xsi="<a href="http://www.w3.org/2001/XMLSchema-instance">http://www.w3.org/2001/XMLSchema-instance</a>" xsi:nil="true" /><br> <uws:phase>COMPLETED</uws:phase><br> <uws:quote>2016-02-29T21:57:43.641</uws:quote><br> <uws:startTime>2016-02-29T21:58:05.928</uws:startTime><br> <uws:endTime>2016-02-29T21:58:05.995</uws:endTime><br> <uws:executionDuration>120</uws:executionDuration><br> <uws:destruction>2016-02-29T22:55:43.641</uws:destruction><br> <uws:parameters><br> <uws:parameter id="POS">circle 66 10 20</uws:parameter><br> <uws:parameter id="POS">circle 140 0 0.1</uws:parameter><br> <uws:parameter id="ID">ad:IRIS/I212B2H0</uws:parameter><br> <uws:parameter id="POLY">20 20 21 20 21 21 20 21</uws:parameter><br> <uws:parameter id="POLY">140 0 141 1 141 1 141 0</uws:parameter><br> <uws:parameter id="CIRC">70 0 0.2</uws:parameter><br> <uws:parameter id="CIRC">140 1 0.1</uws:parameter><br> <uws:parameter id="CIRC">140 0 0.1</uws:parameter><br> </uws:parameters><br> <uws:results><br> <uws:result id="ok-7" xlink:href="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/IRIS/I212B2H0?runid=p1zhcks2z1gfybq0&amp;cutout=%5B0%5D%5B235%3A275%2C258%3A298%2C*%5D">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/IRIS/I212B2H0?runid=p1zhcks2z1gfybq0&amp;cutout=%5B0%5D%5B235%3A275%2C258%3A298%2C*%5D</a>" /><br> <uws:result id="warn-6" xlink:href="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/soda-echo/NDAwfHRleHQvcGxhaW58Tm9Db250ZW50OiBhZDpJUklTL0kyMTJCMkgwIHZzIFBPTFk9MjAgMjAgMjEgMjAgMjEgMjEgMjAgMjE=">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/soda-echo/NDAwfHRleHQvcGxhaW58Tm9Db250ZW50OiBhZDpJUklTL0kyMTJCMkgwIHZzIFBPTFk9MjAgMjAgMjEgMjAgMjEgMjEgMjAgMjE=</a>" /><br> <uws:result id="ok-5" xlink:href="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/IRIS/I212B2H0?runid=p1zhcks2z1gfybq0&amp;cutout=%5B0%5D%5B271%3A279%2C254%3A262%2C*%5D">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/IRIS/I212B2H0?runid=p1zhcks2z1gfybq0&amp;cutout=%5B0%5D%5B271%3A279%2C254%3A262%2C*%5D</a>" /><br> <uws:result id="ok-4" xlink:href="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/IRIS/I212B2H0?runid=p1zhcks2z1gfybq0&amp;cutout=%5B0%5D%5B271%3A279%2C294%3A302%2C*%5D">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/IRIS/I212B2H0?runid=p1zhcks2z1gfybq0&amp;cutout=%5B0%5D%5B271%3A279%2C294%3A302%2C*%5D</a>" /><br> <uws:result id="warn-3" xlink:href="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/soda-echo/NDAwfHRleHQvcGxhaW58Tm9Db250ZW50OiBhZDpJUklTL0kyMTJCMkgwIHZzIENJUkM9NzAgMCAwLjI=">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/soda-echo/NDAwfHRleHQvcGxhaW58Tm9Db250ZW50OiBhZDpJUklTL0kyMTJCMkgwIHZzIENJUkM9NzAgMCAwLjI=</a>" /><br> <uws:result id="ok-2" xlink:href="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/IRIS/I212B2H0?runid=p1zhcks2z1gfybq0&amp;cutout=%5B0%5D%5B271%3A279%2C254%3A262%2C*%5D">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/IRIS/I212B2H0?runid=p1zhcks2z1gfybq0&amp;cutout=%5B0%5D%5B271%3A279%2C254%3A262%2C*%5D</a>" /><br> <uws:result id="warn-1" xlink:href="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/soda-echo/NDAwfHRleHQvcGxhaW58Tm9Db250ZW50OiBhZDpJUklTL0kyMTJCMkgwIHZzIFBPUz1jaXJjbGUgNjYgMTAgMjA=">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/soda-echo/NDAwfHRleHQvcGxhaW58Tm9Db250ZW50OiBhZDpJUklTL0kyMTJCMkgwIHZzIFBPUz1jaXJjbGUgNjYgMTAgMjA=</a>" /><br> </uws:results><br></uws:job><br><br>In there you see 7 results (1 per positional cutout in this case. I've named them with "ok" when the result is a FITS cutout, with "warn" if it isn't a FITS cutout but not a failure (eg the specified position doesn't include any pixels), and (if any had failed) with "error". The number is there because result names are unique. If you GET any of those URLs you will get the same output as having made that single request via SODA sync (eg same FITS cutout or same error message (code and message), eg: warn-1 from above gives:<br><br>HTTP/1.1 400 Bad Request<br>NoContent: ad:IRIS/I212B2H0 vs POS=circle 66 10 20<br><br>You will see that I'm using a special resource (soda-echo) to generate the error messages; this is based on my eariler comment that UWS doesn't support partial success/partial fail very well and this is how one can do it with what is available in UWS. I think this is better than trying to have 5 results and an error summary or document (for the other 2) in a job that reached the COMPLETED phase. And it keeps sync and async less different.<br>And yeah: the soda-echo URLs contain everything in the URL (base64 encoded for extra opaqueness) so I don't need server-side state :-)<br><br>Note: if the input is detected to be invalid the whole job fails (e.g. invalid ID).<br><br>** Why would I want SODA async **<br><br>As implemented here, SODA async is a simple way to collect a bunch of sync requests into a batch. In our implementation, this is more efficient because we only perform one query (per ID) to get the metadata necessary for all the computations. If the user was authenticated, this would also mean a single call to various other resources instead of many. I could imagine a storage system such as an off-the-shelf object store that required implementation of a cutout service to retrieve the data file to a processing area before performing the cutouts; in that kind of scenario async would permit one retrieval, many cutouts performed, results stored in some temporary storage area, and result URLs in the job pointing at those resulting files. We do not currently stage any results because we can run astronomy code inside the storage system, but I could see that changing in the future... <br><br>** What else? **<br><br>While multiple cutouts per ID can be much more efficient, I cannot see any reason to think that creating a job with multiple IDs would be any better than one job per ID. OK, I guess if you were bringing up a VM to process one job you might want it to do moer work, but there are lots of other ways to get rid of that overhead. So, it seems like I would agree with ID being single-valued in SODA, which by extension means that it would be nice/necessary for DataLink service descriptors to describe the multiplicity of input parameters.<br><br>As implemented, if you were run a SODA async job with 2 CIRC and 2 BAND (eg) will cause 4 cutouts to be computed (each combination). That may or may not be what users want, but they cannot really pick how to combine the multiple params for different axes: that requires some sort of structure. Markus suggested that this kind of thing should be done via a table upload with one row per desired combination of params. That could be done and since the params and values are defined consistently with VOTable usage it wouldn't be too hard to specify and implement, but the barrier to using this for techy astronomers would be a lot higher without some client tools they would actually use... <br><br>Finally, I will let others think about the use of POS vs. the more explicit CIRC and POLY with metadata in the custom service descriptors from the datalink service... and also whether we want to emphasise data-specific service descriptors and convey useful metadata vs. generic service descriptiors and getting the metadata some other way (a new capability such as {metadata} from the DAL architecture that was hinted in SIA-2.0). <br><br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On 29 February 2016 at 16:06, Patrick Dowler <span dir="ltr"><<a href="mailto:pdowler.cadc@gmail.com" target="_blank">pdowler.cadc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br>TL;DR - CADC implemented a prototype SODA services and use DataLink service descriptors to convey data-specific metadata and parameter info. Here we provide a cube example of the links, data-specific service descriptors, and SIDA sync cutouts. Since it is a cube, positional and energy (BAND) cutouts are enabled.<br><br>cube dataset: ID=caom:CGPS/MA1_DRAO-ST/HI-line<br><br>* ObsCore-1.1 metadata *<br><br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/sia/v2query?ID=caom:CGPS/MA1_DRAO-ST/HI-line" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/sia/v2query?ID=caom:CGPS/MA1_DRAO-ST/HI-line</a><br><br>note: due to a bug the em_exl column is null in the ObsCore output; we've fixed the bug but <br>not (yet) the content<br><br>* cube links *<br><br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/datalink?ID=caom:CGPS/MA1_DRAO-ST/HI-line" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/datalink?ID=caom:CGPS/MA1_DRAO-ST/HI-line</a><span class=""><br><br><br>The link-specific SODA descriptors (the ID attributes have UUIDs in them) e.g.:<br><br></span><RESOURCE type="meta" ID="soda-efd4e3f2-1edb-4172-8d7c-a502769a3fc1" utype="adhoc:service"><span class=""><br> <PARAM name="resourceIdentifier" datatype="char" arraysize="27" <br> value="ivo://<a href="http://cadc.nrc.ca/soda#sync" target="_blank">cadc.nrc.ca/soda#sync</a>" /><br> <PARAM name="standardID" datatype="char" arraysize="32" <br> value="ivo://<a href="http://ivoa.net/std/SODA#sync-1.0" target="_blank">ivoa.net/std/SODA#sync-1.0</a>" /><br> <PARAM name="accessURL" datatype="char" arraysize="*" <br> value="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync</a>" /><br> <GROUP name="inputParams"><br></span> <PARAM name="ID" datatype="char" ucd="" arraysize="*" value="ad:CGPS/cgps_ma1_hi_line_image" /><span class=""><br> <PARAM name="POS" datatype="char" ucd="obs.field" arraysize="*" value="" /><br> <PARAM name="CIRC" datatype="double" ucd="obs.field" unit="deg" xtype="circle" arraysize="3" value=""><br> <VALUES><br></span> <MAX value="25.055691919221182 61.31670742370859 3.620294640836429" /><span class=""><br> </VALUES><br> </PARAM><br> <PARAM name="POLY" datatype="double" ucd="obs.field" unit="deg" xtype="polygon" arraysize="*" value=""><br> <VALUES><br></span> <MAX value="28.91706938461008 58.26098083062218 19.210513573678497 59.14944023257296 20.380237118242576 64.23953415946808 31.706454362389877 63.197249950105615" /><br> </VALUES><br> </PARAM><span class=""><br> <PARAM name="BAND" datatype="double" ucd="em.wl;stat.interval" unit="m" xtype="interval" arraysize="2" value=""><br></span> <VALUES><br> <MAX value="0.21094492014316196 0.21110274396161874" /><br> </VALUES><br> </PARAM><br> </GROUP><br> </RESOURCE><br><br>TIME and POL are missing because this is a GLON-GLAT-VELO-POL fits cube (but the polarization axis has only one bin with I), so cutouts on those axes are not possible.<br><br>POS is listed there because positional cutout will work.<br><br>CIRC and POLY are listed with the minimum spanning circle and polygon bounds "max extent" as described earlier.<br><br>The FITS file itself is in GLON-GLAT-VELO so the the implementation has to take care of all transformations; if you go so far as to download the result, the FITS file will still be in GLON-GLAT-VELO.<span class=""><br><br>For these URLs, I recommend just "curl -v" so you can see the HTTP headers, but you can download the data if you want to :-)<br><br><br></span>* position cutout: POS *<br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:CGPS/cgps_ma1_hi_line_image%5C&POS=circle%2025.0%2060.0%201.0" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:CGPS/cgps_ma1_hi_line_image\&POS=circle%2025.0%2060.0%201.0</a><br><br>decoding the redirect url: cutout=[0][229:696,20:488,*,*]<br><br>* position cutout: CIRC *<br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:CGPS/cgps_ma1_hi_line_image%5C&CIRC=25.0%2060.0%201.0" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:CGPS/cgps_ma1_hi_line_image\&CIRC=25.0%2060.0%201.0</a><br><br>decoding the redirect url: cutout=[0][229:696,20:488,*,*]<br><br>* position cutout: POLY *<br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:CGPS/cgps_ma1_hi_line_image%5C&POLY=24.0%2060.0%2025.0%2060.0%2025.0%2061.0%2024.0%2061.0" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:CGPS/cgps_ma1_hi_line_image\&POLY=24.0%2060.0%2025.0%2060.0%2025.0%2061.0%2024.0%2061.0</a><br><br>decoding the redirect url: cutout=[0][468:601,234:449,*,*]<br><br>* energy cutout: BAND *<br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:CGPS/cgps_ma1_hi_line_image%5C&BAND=0.21102%200.21104" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:CGPS/cgps_ma1_hi_line_image\&BAND=0.21102%200.21104</a><br><br>decoding the redirect URL: cutout=[0][*,*,108:143,*]<br><br><br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 29 February 2016 at 16:05, Patrick Dowler <span dir="ltr"><<a href="mailto:pdowler.cadc@gmail.com" target="_blank">pdowler.cadc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br>TL;DR - CADC implemented a prototype SODA services and use DataLink service descriptors to convey data-specific metadata and parameter info. The implementation demonstrates that with POS you cannot convey metadata so we introduced new positional cutout paramseters (CIRC and POLY) that conform to WD-DALI-1.1 xtypes (circle and polygon) and allow us to convey useful parameter metadata as a result.<br><br>Before anyone panics: we also show that POS, CIRC, and POLY can co-exist :-)<br><br>image dataset: ID=caom:IRIS/f212h000/IRAS-25um<br><br>* ObsCore-1.1 metadata *<br><br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/sia/v2query?ID=caom:IRIS/f212h000/IRAS-25um" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/sia/v2query?ID=caom:IRIS/f212h000/IRAS-25um</a><br><br>So, this is a dataproduct_type=image, calib_level=2, s_xel1 & s_xel2 say it is 500x500 (pixels)<br><br>* image links *<br><br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/datalink?ID=caom:IRIS/f212h000/IRAS-25um" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/datalink?ID=caom:IRIS/f212h000/IRAS-25um</a><br><br>Note the generic SODA service descriptors (not linked!):<br><br><RESOURCE type="meta" ID="soda-sync" utype="adhoc:service"><br> <PARAM name="resourceIdentifier" datatype="char" arraysize="27" <br> value="ivo://<a href="http://cadc.nrc.ca/soda#sync" target="_blank">cadc.nrc.ca/soda#sync</a>" /><br> <PARAM name="standardID" datatype="char" arraysize="*" <br> value="ivo://<a href="http://ivoa.net/std/SODA#sync-1.0" target="_blank">ivoa.net/std/SODA#sync-1.0</a>" /><br> <PARAM name="accessURL" datatype="char" arraysize="*" <br> value="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync</a>" /><br> <GROUP name="inputParams"><br> <PARAM name="ID" datatype="char" ref="fileURIRef" arraysize="*" value="" /><br> <PARAM name="POS" datatype="char" ucd="obs.field" arraysize="*" value="" /><br> <PARAM name="CIRC" datatype="double" ucd="obs.field" unit="deg" xtype="circle" arraysize="3" <br> value="" /><br> <PARAM name="POLY" datatype="double" ucd="obs.field" unit="deg" xtype="polygon" arraysize="*" <br> value="" /><br> <PARAM name="BAND" datatype="double" ucd="em.wl;stat.interval" unit="m" xtype="interval" arraysize="2"<br> value="" /><br> <PARAM name="TIME" datatype="double" ucd="time;stat.interval" unit="d" xtype="interval" arraysize="2"<br> value="" /><br> <PARAM name="POL" datatype="char" ucd="phys.polarization.stokes" arraysize="2*" value="" /><br> </GROUP><br><br></RESOURCE><br><RESOURCE type="meta" ID="soda-async" utype="adhoc:service"><br> <PARAM name="standardID" datatype="char" arraysize="*" <br> value="ivo://<a href="http://ivoa.net/std/SODA#async-1.0" target="_blank">ivoa.net/std/SODA#async-1.0</a>" /><br> ... same params as above<br></RESOURCE><br> <br>params: ID, POS, CIRC, POLY, BAND, TIME, POL<br><br>Above and below you will see a resourceIdentifier param; this is there to support the use of a runtime registry lookup to generate the accessURL. Doing it this way allows us to generate URLs to development, test, or production servers depending on the work environment... our DataLink and SODA services are not actually registered but my intent is to make these resolvable by registering the services in the near future.<br><br>Likewise, the standardID values for SODA are not (yet) resolvable but they will be...<br><br>The link-specific SODA descriptors (the ID attributes have UUIDs in them) e.g.:<br><br><RESOURCE type="meta" ID="soda-cbb62ed5-c2c9-4dd9-aed6-46d7d5173dca" utype="adhoc:service"><br> <PARAM name="resourceIdentifier" datatype="char" arraysize="27"<br> value="ivo://<a href="http://cadc.nrc.ca/soda#sync" target="_blank">cadc.nrc.ca/soda#sync</a>" /><br> <PARAM name="standardID" datatype="char" arraysize="32" <br> value="ivo://<a href="http://ivoa.net/std/SODA#sync-1.0" target="_blank">ivoa.net/std/SODA#sync-1.0</a>" /><br> <PARAM name="accessURL" datatype="char" arraysize="*" <br> value="<a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync</a>" /><br> <GROUP name="inputParams"><br> <PARAM name="ID" datatype="char" ucd="" arraysize="*" value="ad:IRIS/I212B2H0" /><br> <PARAM name="POS" datatype="char" ucd="obs.field" arraysize="*" value="" /><br> <PARAM name="CIRC" datatype="double" ucd="obs.field" unit="deg" xtype="circle" arraysize="3"<br> value=""><br> <VALUES><br> <MAX value="140.63049941314583 0.2007826788236291 8.778341996040131" /><br> </VALUES><br> </PARAM><br> <PARAM name="POLY" datatype="double" ucd="obs.field" unit="deg" xtype="polygon" arraysize="*"<br> value=""><br> <VALUES><br> <MAX value="146.83673628162285 -6.408995958971017 134.3808989000012 -6.370135804011464 134.4242625446688 6.007430601323759 146.8700273500712 5.96918267453771" /><br> </VALUES><br> </PARAM><br> </GROUP><br></RESOURCE><br><br>The value for the ID parameter is specified in the value attribute because this is a file-specific (at CADC) service descriptor and it needs this file. This *is not* the same kind of ID that one uses to call the DataLink service (that is a publisher_did which we will be changing into a resolvable ivo-id once I'm happy with the registration of our collections; this is a file identifier from our storage system). I could have used a ref attribute to the links table (since the file URI is there for our other services and for the generic soda service descriptors) but once you have link-specific descriptiorsd anyway this seems tidier (e.g. I'll eventually be able to remove the custom fileURIref column from our links table).<br><br>BAND, TIME, and POL are missing because this is a 2D image so cutouts on those axes are not possible.<br><br>POS is listed there because positional cutout is possible, but I don't see a sane way to convey sensible values to help someone use POS; the client has to know the extent (from data discovery) or get it in some other way (metadata capability).<br><br>For CIRC and POLY the service includes a "maximum sensible extent" with which to perform cutouts. The value attribute of the MAX element is a string and my interpretation of the intent is that the client should interpret it as the same "type" of thing as the PARAM in which it is found. It feels like the MAX extent conveys useful and sensible information, but I didn't see anything useful to put in MIN. Is this an abuse of MAX? Maybe (in the sense that MAX usually implies that there is an ordering) but given the implied type consistency that is already there people are interpreting this now and when I showed this to a few techy astronomers that understand VOTable they interrpetted this as I meant it. (The CIRC MAX is the minimum spanning circle; the POLY MAX is the polygon boundary -- so using those values would get all the pixels.)<br><br>Right now, CIRC and POLY are my own custom parameters and they should not bother a strict client that used this descriptor because of the standardID. I chose different parameter names so i could be more explicit about the value metadata (datatype, arraysize, xtype, units, ucd -- WD-DALI-1.1) *and* so I could provide the "maximum value" of the exact same type. In the SODA service this is very straightforward to implement (I use the same Format classes for reading and writing VOTables and for parsing and validating SODA params).<br><br></div><div>For these URLs, I recommend just "curl -v" so you can see the HTTP headers, but you can download the data if you want to :-)<br><br>* image cutout: POS *<br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:IRIS/I212B2H0%5C&POS=circle%20140.5%200.0%200.5" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:IRIS/I212B2H0\&POS=circle%20140.5%200.0%200.5</a><br><br>decoding the redirect url: cutout=[0][235:275,238:278,*]<br><br>* image cutout: CIRC *<br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:IRIS/I212B2H0%5C&CIRC=140.5%200.0%200.5" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:IRIS/I212B2H0\&CIRC=140.5%200.0%200.5</a><br><br>decoding the redirect url: cutout=[0][235:275,238:278,*]<br><br>* image cutout: POLY *<br><a href="http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:IRIS/I212B2H0%5C&POLY=140%200.0%20140.5%200.0%20140.5%200.5%20140.0%200.0" target="_blank">http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad:IRIS/I212B2H0\&POLY=140%200.0%20140.5%200.0%20140.5%200.5%20140.0%200.0</a><br><br>decoding the redirect url: cutout=[0][255:275,258:278,*]<br><br>If successful, these SODA requests to /caom2ops/sync respond with an error message or a redirect to a URL with a pixel cutout using cfitsio syntax. That is completely an implementation detail of our archive infrastructure and not part of the prototype per se.<br><br>If they fail (easy to do, just mess with the params) the response (after redirect) is text/plain with a suitable HTTP status code.<br><br><br><br><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On 29 February 2016 at 16:03, Patrick Dowler <span dir="ltr"><<a href="mailto:pdowler.cadc@gmail.com" target="_blank">pdowler.cadc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><br></div>I have finally finished and deployed our latest prototype SODA services and augmented our DataLink service to provide service descriptors to enable use of SODA. This works spans several services so, following Markus' "gripes" appoach I will try to separate things into separate messages, but I'll just make the messages replies to this one so they will be a single thread and I promise to put the TL;DR at the top of each :-)<br><br></div>So, coming up:<br><br></div>1. description of datalink service descriptor output and soda sync cutout of a 2D image<br><br>2. less wordy description of datalink service descriptor output and soda sync cutout of a 3D cube<br></div><br>3. description of datalink service descriptor output and soda async cutout(s) of a 2D image<br><br></div>It is quite a lot to look at, but I would like to point out here that implementing the whole end-to-end usage forced me to reconsider some earlier decisions and refine things to make them more clear and more useful. Although DataLink and SODA are loosely coupled in a technical sense, they do need to get along and work together and each has some effect or influence on decisions one takes while implementing the other.<br><br></div>more to follow...<br><div><div><div><div><br>--<br> <div><div><div><div><div><div dir="ltr"><div><div>Patrick Dowler<br></div>Canadian Astronomy Data Centre<br></div>Victoria, BC, Canada<br></div></div>
</div></div></div></div></div></div></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div><div dir="ltr"><div><div>Patrick Dowler<br></div>Canadian Astronomy Data Centre<br></div>Victoria, BC, Canada<br></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div><div dir="ltr"><div><div>Patrick Dowler<br></div>Canadian Astronomy Data Centre<br></div>Victoria, BC, Canada<br></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div><div>Patrick Dowler<br></div>Canadian Astronomy Data Centre<br></div>Victoria, BC, Canada<br></div></div>
</div>