VOEvent Update: JSON and data models

[Rob Seaman]

Delighted to read all the replies. Not surprisingly there continue to be
divergent points of view. An IVOA note is precisely the mechanism to
provoke such a response.

<break>make coffee</break>

Won't reply to all the messages directly, but most of the original key
design points have been touched upon: diversity of use cases, schema +
code generation, transport, backwards compatibility, triggering
follow-up observations, time series, tabular data, etc.

The current compromises were taken with full knowledge of future
projects like LSST, Gaia, SKA, and if not ZTF then PTF. The world has
moved on and perhaps some new issues have been revealed, but most likely
there has been a change of emphasis to prefer a binary format. We always
intended to support binary content.

The question with new formats like Avro is whether they are mature
enough. I share somebody's concern with the maturity of JSON schema, but
for that matter many of the problems with XML are really problems with
XML schema.

There are many ways for such a project to fail. But one of the biggest
risks would be starting over from scratch. VOEvent as a technology
project was flawed, as is IVOA in general. As a community building
exercise, VOEvent has been one of IVOA's crown jewels. VOEvent has built
a following among both professional and amateur projects.

Perhaps some ground rules?

1) VOEvent-2 should remain as a stable standard. Ditto with the current
transport standard.

2) VOEvent-3 (in JSON and/or Avro) should seek to support reversible
translation from VOEvent-2 as much as possible. If there are features of
VOEvent-2 that cannot be supported, describe them in a section of the
new recommendation. Which is to say, let's not introduce
"simplifications" without overriding motivation.

3) If there are new features of VOEvent-3 that cannot easily be
accommodated in native VOEvent-2, one imagines they might be handled
with references to additional formats in the same stream. Or it may just
be that to use embedded images you need to use tools that understand
VOEvent-3.

4) The language and intent of the normative VOEvent-2 specification is:

    /"[T]he <WhereWhen> element may reference an STC <ObsDataLocation>
    element to provide a sky location and time for the event."//
    /

    /"The STC specification, in particular <ObsDataLocation> and its
    contained elements, allows more exotic coordinate systems (for
    example, describing planetary surfaces). Further description of how
    VOEvent packets might be constructed to convey such information to
    subscribers is outside the scope of this document."/

The prototype schema is non-normative and makes simplifications, but
full support for all the richness of <ObsDataLocation> was explicitly
intended. STC is evolving and VOEvent-3 may have different options.
Having survived both VOEvent-1 and VOEvent-2 (and seeing that Arnold
still has skin in the game), it is my humble belief that the best way to
avoid the complications of STC is to surrender to them entirely. We
don't have to support all of STC, but the richness of the astronomical
use cases - even for a single facility like ZTF - requires all of
<ObsDataLocation>. Leave the question of how best to support this in a
JSON schema for somebody else and just incorporate it as a subschema.

5a) Agree with Rick that VOEvent should continue to focus on describing
astronomical phenomena. Leave controlling telescope/instrument behavior
to another standard(s). Each type of standard is fairly complex.
Combining them would be more than twice so. If RTML, for example, needs
its own JSON realization, that will surely succeed better alone.

5b) This may also apply to standards like ADES for astrometry. ZTF needs
to handle moving objects as well as it bumper crop of kilonovae, but it
doesn't need to handle them with the same format. Consider how the
various formats will interact in workflows. Update the broker diagram
and build the bricks-and-mortar infrastructure.

6) If there is ever a VOEvent-4, suggest these meta-rules (perhaps there
are others) continue in force. Preserve both VOEvent-2 and VOEvent-3 for
backwards compatibility and make the new standard as interchangeable as
possible, but don't fret about new features.

Rob

--


On 10/17/17 3:24 AM, Roy Williams wrote:
> I would like to point out that there are two independent ideas in the Note that we circulated:
> -- The idea of using JSON as an alternate format
> -- The idea that Groups of Params and Tables can be used as data models
>
> These are independent of each other. The data models can be just as well implemented in XML. I have not heard any complaints about the data models. Should I split the Note into two pieces for separate consideration?
>
> Reasons for using JSON. Perhaps (5) is the most important, followed by (6), which is really a question.
>
> (1) JSON and XML can be used together, and pretty much interchanged. Certainly I have had little trouble making an XML-to-JSON converter. However, I will say that we put in a few simplifications that might not convert back easily. For example instead of saying Name1=RA and C1=300 (the XML), we just say RA=300 (the JSON). This means that there is no longer a choice of name, so if you want to call it RiteAscenshun in XML, and it gets converted to JSON and back to XML, then it will come out being called RA. We have also defaulted the units to degrees and meters in the WhereWhen section.
>
> (2) There is no longer the confusion over what is text and what is attributes that XML has. Instead of <tag att=hello>dolly</tag>, we convert to att=hello and tag=dolly.
>
> (3) I like the way Tim refers to XML as Betamax. Many people I have talked to prefer JSON. It is a matter of taste I suppose.
>
> (4) If we are encouraged by the IVOA, it is our intention to build a JSON schema that can be used to validate the content. Once there is a schema, the JSON is much more constrained. JSON can also be "schema-compliant"
>
> (5) The future high-volume transient surveys are proposing to use a format called AVRO, that is very much like JSON, to transport event notices. There are technical efficiencies such as binary encoding, and in particular the ability to put binary values in the notice -- image cutouts. The AVRO schema is written in JSON, and is quite close to the JSON schema alluded in (4). Thus VOEvent is brought in line with the ZTF and LSST projects.
>
> (6) If encouraged by the IVOA, we could shift attention from JSON and work directly with its cousin, AVRO.
>
> Thank you for your attention
> Roy (with Eric, Matthew, Rob, and Scott)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/voevent/attachments/20171017/e1708955/attachment-0001.html>