Draft VOEvent 2.1

[Baptiste Cecconi]

Hi all, 

After Rob's comment, I went back to the VOEvent 2.0 recommendation. It is indeed stating that the WhereWhen section must follow the STC model.
The current schema 2.0 available on the IVOA website is not including all the STC elements that we need for our project. 

To me, this is clearly a case for an errata: the VOEvent 2.0 XML schema currently available is not fully compliant with the VOEvent data model. 
I will update my draft proposal following this path. I will remove the single extra feature that is presently in the draft (the addition of external AuthorURI), and we may discuss this separately. The rest of the schema modifications are just improving with implementation of the VOEvent 2.0 recommendation.

Sincerely
Baptiste

 
> Le 5 nov. 2017 à 16:47, Matthew Graham <mjg at caltech.edu> a écrit :
> 
> Hi Dave,
> 
> Thanks for these words - I agree with them heartily; except for the suggestion to incorporate changes to VOEvent schema via errata. Errata are to correct and clarify existing standards and not as a shortcut to introduce new features.
> 
> ZTF(/LSST) are prototyping AVRO as an efficient distribution format for alerts for exactly the same reason CRTS used XMPP/Jabber with VOEvent a decade ago. Why write something ourselves when industry has already produced something that seems to do what we want and has some nice additional functions as well.
> 
> — Matthew
> 
> 
>> On Nov 5, 2017, at 3:09 PM, Dave Morris <dave.morris at metagrid.co.uk> wrote:
>> 
>> Hi all,
>> 
>> On 2017-11-03 13:43, Roy Williams wrote:
>>> The real problem though, lies with VOEvent itself being (a)
>>> tied to XML and (b) in a state of flux.
>> 
>> On 2017-11-04 15:51, Rob Seaman wrote:
>>> I suspect that anybody who has navigated a standard through the IVOA
>>> process would agree with Roy!
>> 
>> I'm sorry guys, but I'm afraid I disagree.
>> 
>> First, regarding the issue of the existing XML schema; I have seen no evidence to suggest that the XML based schema is causing problems for the VOEvent community.
>> 
>> In fact, I was impressed and encouraged by the number of projects I met at a recent conference on transients and alerts who were either already using, or planning to use, the current XML based VOEvent tools to develop networks for coordinating follow up responses to transient events.
>> 
>> As far as I can remember, none of the projects even mentioned XML as an issue.
>> 
>> Next, the suggestion that an urgent and immediate transition to JSON is imperative for whatever reason is, to my mind, unfounded.
>> 
>> To a great extent, the actual on the wire format is irrelevant. Both XML and JSON are text based formats, so they will both need to expand numeric values into strings of ASCII characters, with the associated loss of precision, and they will both need to use some form of encoding like base64 to embed binary objects such as images.
>> 
>> However, this misses an important point. None of our scientists should ever encounter the raw on-the-wire data format. Software for implementing the science use cases should be developed based on higher level libraries such as the FourPiSky voevent-parse toolkit, which represents event messages as Python objects, not as XML or JSON.
>> 
>> The fact that it might be easier to write a low-level parser in a particular language for one data format rather than another may influence our choice of formats in the future. But it is nowhere near enough evidence to justify disrupting an existing, working system that people are using in live production networks.
>> 
>> Third, the suggestion that we should drop everything and convert to Avro in a mad rush to be ready for the ZTF/LSST streams as soon as they become available is unrealistic for several reasons.
>> 
>> Not least of which is because the ZTF/LSST teams themselves are still experimenting with how to use Avro. I think we need to allow the ZTF/LSST team space to evolve their designs from their current experiments towards a production ready design before we decide to bet the future IVOA standard on it.
>> 
>> So far, the only people I know of who have direct experience with Avro are the ZTF/LSST team, and they are not ready to publish anything yet.
>> 
>> It is also important to note that based on the current plans LSST are only distributing the Avro based fire-hose streams to a limited number of community brokers, who will then filter the streams into smaller lower bandwidth streams of more specific event types.
>> 
>> When this was discussed at the IVOA meeting in Santiago, the expectation was that in most cases the community brokers would probably issue their downstream alerts using the existing VOEvent format and protocol.
>> 
>> This would enable users to plan and develop their software using the existing tools and test it using the existing VOEvent network to be ready to receive the new types of event when they become available.
>> 
>> It is also important to note that the LSST project is not external to the IVOA. They are participating members of the IVOA and scientists and engineers from LSST were active in, and in some cases lead, the discussions at the conference in Santiago.
>> 
>> We should not see them as an external threat developing ad-hoc designs that will fragment the community. They are part of this community, and we should work with them to develop the next generation of VOEvent. However, at this stage I think the conversation should be about prototypes and experiments, not standards and specifications.
>> 
>> If we are concerned about how to prepare for this new generation of alert streams, then I suggest we can ask the LSST team to make a presentation at the next IVOA conference. Based on  what I have heard from talking with them, I think they may already be planning on presenting something at the May interop anyway.
>> 
>> In addition, those of us who are working on developing community brokers for the LSST streams can play our part in ensuring that the message content of ZTF/LSST alerts are compatible with VOEvent, by proposing changes to both the VOEvent and the ZTF/LSST schema to bring them closer together.
>> 
>> If, as a result of that, we discover specific structures or patterns in XML schema that we should avoid using because they would make a transition to a new format more difficult then I agree we should take care to avoid them where ever possible.
>> 
>> We should use the lessons learned from our Avro prototypes and experiments to inform the evolution of the 2.x XML schema, but I don't think the prospect of Avro/Kafka based systems in the future  should prevent us evolving the existing XML based standards in the mean time.
>> 
>> I think we should continue to evolve the 2.x XML schema in response to specific science use cases.
>> 
>> As long as we ensure that minor point version changes 2.1, 2.2 etc. are backwards compatible with the existing tools and libraries, then I see this as part of the natural evolution that happens with any IVOA standard.
>> 
>> I also want to reassure astronomers who are already using the existing 2.x schema, and the associated tools and libraries, like Comet and FourPiSky, that their work will continue to be supported in the future.
>> 
>> When Pierre and Baptiste contacted me about their ideas for changes to the 2.x schema based on their specific use case, I encouraged them to present their proposals to the community for discussion. The suggestion that it should be proposed as a new version (2.1) of the standard came from me.
>> 
>> On the other hand, if we can incorporate some or all of their proposals by issuing an errata, then I'm all for that, it would take less time and involve a lot less paperwork.
>> 
>> As far as I am concerned, as long as they are compatible with the existing 2.x schema, do not disrupt the systems already in place, and solve well defined science use cases, then I think we should welcome their proposals.
>> 
>> If there are specific XML constructs that we know will make it difficult to migrate to another format such as Avro at a later date, then I agree we should avoid using them wherever possible.
>> 
>> However, I don't believe that the prospect of ZTF/LSST using Avro for their alert streams in the future should prevent us from evolving the existing VOEvent standards to meet current science use cases now.
>> 
>> Evolving the current standards to meet current science use cases, and developing prototypes to learn about new technologies like Avro are not exclusive.
>> 
>> We should be able to do both at the same time.
>> 
>> Regards,
>> Dave
>> 
>> --------
>> Dave Morris
>> Research Software Engineer
>> Wide Field Astronomy Unit
>> Institute for Astronomy
>> University of Edinburgh
>> --------
>