Draft VOEvent 2.1

[Dave Morris]

Hi all,

On 2017-11-03 13:43, Roy Williams wrote:
> The real problem though, lies with VOEvent itself being (a)
> tied to XML and (b) in a state of flux.

On 2017-11-04 15:51, Rob Seaman wrote:
> I suspect that anybody who has navigated a standard through the IVOA
> process would agree with Roy!

I'm sorry guys, but I'm afraid I disagree.

First, regarding the issue of the existing XML schema; I have seen no 
evidence to suggest that the XML based schema is causing problems for 
the VOEvent community.

In fact, I was impressed and encouraged by the number of projects I met 
at a recent conference on transients and alerts who were either already 
using, or planning to use, the current XML based VOEvent tools to 
develop networks for coordinating follow up responses to transient 
events.

As far as I can remember, none of the projects even mentioned XML as an 
issue.

Next, the suggestion that an urgent and immediate transition to JSON is 
imperative for whatever reason is, to my mind, unfounded.

To a great extent, the actual on the wire format is irrelevant. Both XML 
and JSON are text based formats, so they will both need to expand 
numeric values into strings of ASCII characters, with the associated 
loss of precision, and they will both need to use some form of encoding 
like base64 to embed binary objects such as images.

However, this misses an important point. None of our scientists should 
ever encounter the raw on-the-wire data format. Software for 
implementing the science use cases should be developed based on higher 
level libraries such as the FourPiSky voevent-parse toolkit, which 
represents event messages as Python objects, not as XML or JSON.

The fact that it might be easier to write a low-level parser in a 
particular language for one data format rather than another may 
influence our choice of formats in the future. But it is nowhere near 
enough evidence to justify disrupting an existing, working system that 
people are using in live production networks.

Third, the suggestion that we should drop everything and convert to Avro 
in a mad rush to be ready for the ZTF/LSST streams as soon as they 
become available is unrealistic for several reasons.

Not least of which is because the ZTF/LSST teams themselves are still 
experimenting with how to use Avro. I think we need to allow the 
ZTF/LSST team space to evolve their designs from their current 
experiments towards a production ready design before we decide to bet 
the future IVOA standard on it.

So far, the only people I know of who have direct experience with Avro 
are the ZTF/LSST team, and they are not ready to publish anything yet.

It is also important to note that based on the current plans LSST are 
only distributing the Avro based fire-hose streams to a limited number 
of community brokers, who will then filter the streams into smaller 
lower bandwidth streams of more specific event types.

When this was discussed at the IVOA meeting in Santiago, the expectation 
was that in most cases the community brokers would probably issue their 
downstream alerts using the existing VOEvent format and protocol.

This would enable users to plan and develop their software using the 
existing tools and test it using the existing VOEvent network to be 
ready to receive the new types of event when they become available.

It is also important to note that the LSST project is not external to 
the IVOA. They are participating members of the IVOA and scientists and 
engineers from LSST were active in, and in some cases lead, the 
discussions at the conference in Santiago.

We should not see them as an external threat developing ad-hoc designs 
that will fragment the community. They are part of this community, and 
we should work with them to develop the next generation of VOEvent. 
However, at this stage I think the conversation should be about 
prototypes and experiments, not standards and specifications.

If we are concerned about how to prepare for this new generation of 
alert streams, then I suggest we can ask the LSST team to make a 
presentation at the next IVOA conference. Based on  what I have heard 
from talking with them, I think they may already be planning on 
presenting something at the May interop anyway.

In addition, those of us who are working on developing community brokers 
for the LSST streams can play our part in ensuring that the message 
content of ZTF/LSST alerts are compatible with VOEvent, by proposing 
changes to both the VOEvent and the ZTF/LSST schema to bring them closer 
together.

If, as a result of that, we discover specific structures or patterns in 
XML schema that we should avoid using because they would make a 
transition to a new format more difficult then I agree we should take 
care to avoid them where ever possible.

We should use the lessons learned from our Avro prototypes and 
experiments to inform the evolution of the 2.x XML schema, but I don't 
think the prospect of Avro/Kafka based systems in the future  should 
prevent us evolving the existing XML based standards in the mean time.

I think we should continue to evolve the 2.x XML schema in response to 
specific science use cases.

As long as we ensure that minor point version changes 2.1, 2.2 etc. are 
backwards compatible with the existing tools and libraries, then I see 
this as part of the natural evolution that happens with any IVOA 
standard.

I also want to reassure astronomers who are already using the existing 
2.x schema, and the associated tools and libraries, like Comet and 
FourPiSky, that their work will continue to be supported in the future.

When Pierre and Baptiste contacted me about their ideas for changes to 
the 2.x schema based on their specific use case, I encouraged them to 
present their proposals to the community for discussion. The suggestion 
that it should be proposed as a new version (2.1) of the standard came 
from me.

On the other hand, if we can incorporate some or all of their proposals 
by issuing an errata, then I'm all for that, it would take less time and 
involve a lot less paperwork.

As far as I am concerned, as long as they are compatible with the 
existing 2.x schema, do not disrupt the systems already in place, and 
solve well defined science use cases, then I think we should welcome 
their proposals.

If there are specific XML constructs that we know will make it difficult 
to migrate to another format such as Avro at a later date, then I agree 
we should avoid using them wherever possible.

However, I don't believe that the prospect of ZTF/LSST using Avro for 
their alert streams in the future should prevent us from evolving the 
existing VOEvent standards to meet current science use cases now.

Evolving the current standards to meet current science use cases, and 
developing prototypes to learn about new technologies like Avro are not 
exclusive.

We should be able to do both at the same time.

Regards,
Dave

--------
Dave Morris
Research Software Engineer
Wide Field Astronomy Unit
Institute for Astronomy
University of Edinburgh
--------