Implementation experience with SIA 2

[Walter Landry]

Hello Everyone,

Back in July, I sent a note to this list about some issues I had with
SIA 2.0.  Since then, we have implemented a synthetic image generation
service for the Planck satellite.  We tried to implement this is in a
way consistent with SIA 2.0, but we had some difficulties.

1) BAND

   The Planck satellite detector bands are all specified in GHz: 30,
   44, 70, etc.  These are nice, integer numbers.  Mapping to
   wavelength leaves me with numbers that are not exactly
   representable in floating point.  This means that every search has
   to give a range.  It would be nice if I could specify the frequency
   instead of the wavelength.  We ended up using the keyword FREQ in
   MHz, since I do not know of any astronomical observations of EM
   radiation that go lower than that.

2) RANGE

   As I mentioned in July, RANGE is prohibitively expensive for this
   data set.  So we do not support it, will never support it, and I
   still think it should not be part of the spec.

3) POS

   Since this is a synthetic image generation service, it would be
   nice to make rectangular images.  The current SIA 2.0 spec has no
   great circle rectangles.  The client has to construct the polygons
   themselves, which is non-trivial.  Given the negative reaction I
   got to Box's last time, we ended up ditching SIA 2.0 for this
   entirely and using SIA 1 syntax: POS, SIZE, CFRAME, CDELT (though
   SPATRES would have been fine).  I would really prefer a better
   mechanism than this.

4) TARGET vs OBJECT

   Why does SIA 2.0 use the TARGET parameter?  OBJECT is an existing
   standard FITS convention.

5) Syntax

   Consider these issues:

   a) In July, I highlighted a problem with the syntax of POS
      parameters.  It requires spaces, which must be URL encoded or
      things silently break.  Silent breakage is the worst kind of
      breakage.

   b) Polygon searches use a straight list of numbers.  It would be
      better to have a list of pairs to make typos more obvious.

   c) We need to be able to select multiple detectors at once, so we
      would like to have an array of strings.

   d) There is no way to add arbitrary parameters.  COORD was the way
      to do that in old versions of SIA 2.0.  Now I have to use up a
      keyword and hope it does not accidentally conflict with new
      versions of the standard.  This is not going to scale.

   e) We have a smart client doing searches on behalf of the user.  In
      general, we would like to set arbitrary metadata that are not
      necessary for the search but convenient for the user.

   This prompted me to use a more general syntax to express queries.
   Specifically, I used json5

     https://github.com/aseemk/json5

   It is an extension of JSON to make it friendlier to write.  It is a
   strict superset of JSON and a strict subset of Javascript.  So
   every valid JSON file is valid json5, and eval() will still work
   for those of you foolish enough to run it on unverified user input ;)
   To be specific, a sample query would be

     http://irsa.ipac.caltech.edu/cgi-bin/Planck_TOI/nph-planck_toi_sia?POS=[0.053,-0.062]&CFRAME='GAL'&ROTANG=90&SIZE=1&CDELT=0.05&FREQ=44000&ITERATIONS=20&INSTRUMENT=['24m','24s']&TIME=[[0,55300],[55500,Infinity]]&USER_METADATA={CLIENT:'IRSA Smart Client'}

   Note that the service is not public yet, so this URL will not work
   for you yet.  

   Internally, every parameter is converted into a json5 element.  So
   this would turn into the json5 document

   {
     POS:[0.053,-0.062],
     CFRAME:'GAL',
     ROTANG:90,
     SIZE:1,
     CDELT:0.05,
     FREQ:44000,
     ITERATIONS:20,
     INSTRUMENT:['24m','24s'],
     TIME:[[0,55300],[55500,Infinity]],
     USER_METADATA:{CLIENT:'IRSA Smart Client'}
   }

   Modulo whitespace, this is just replacing '&' with ',' and '=' with
   ':'.  We also support submitting a json5 document directly

     http://irsa.ipac.caltech.edu/cgi-bin/Planck_TOI/nph-planck_toi_sia?{POS:[0.053,-0.062],CFRAME:'GAL',ROTANG:90,SIZE:1,CDELT:0.05,FREQ:44000,ITERATIONS:20,INSTRUMENT:['24m','24s'],TIME:[[0,55300],[55500,Infinity]],USER_METADATA:{CLIENT:'IRSA Smart Client'}}

   On a side note, I have used JSON (not json5) as input for
   simulations in geology [1].  The SAMRAI Adaptive Mesh Refinement
   framework for massively parallel simulations [2] also uses a format
   that is almost indistinguishable [3] from json5 for input files.
   So I would claim that json5 would be able to cover any needs that
   SIMDAL would need in specifying model parameters.

Given all of the issues that I ran into, it is not clear to me that it
would be a good idea to ratify SIA 2 as it is now.  Whether or not you
like the changes I made, there seem to be some major deficiencies that
need to be addressed before the standard can be accepted.

Cheers,
Walter Landry


[1] http://geodynamics.org/cig/software/gale
[2] https://computation-rnd.llnl.gov/SAMRAI/index.php
[3] You can separate elements in an array or object with newlines instead of commas.