Draft VOEvent 2.1

[Arnold Rots]

FWIW: the singular of errata is erratum.

On Nov 5, 2017 3:34 PM, "Baptiste Cecconi" <baptiste.cecconi at obspm.fr>
wrote:

> Hi all,
>
> After Rob's comment, I went back to the VOEvent 2.0 recommendation. It is
> indeed stating that the WhereWhen section must follow the STC model.
> The current schema 2.0 available on the IVOA website is not including all
> the STC elements that we need for our project.
>
> To me, this is clearly a case for an errata: the VOEvent 2.0 XML schema
> currently available is not fully compliant with the VOEvent data model.
> I will update my draft proposal following this path. I will remove the
> single extra feature that is presently in the draft (the addition of
> external AuthorURI), and we may discuss this separately. The rest of the
> schema modifications are just improving with implementation of the VOEvent
> 2.0 recommendation.
>
> Sincerely
> Baptiste
>
>
> > Le 5 nov. 2017 à 16:47, Matthew Graham <mjg at caltech.edu> a écrit :
> >
> > Hi Dave,
> >
> > Thanks for these words - I agree with them heartily; except for the
> suggestion to incorporate changes to VOEvent schema via errata. Errata are
> to correct and clarify existing standards and not as a shortcut to
> introduce new features.
> >
> > ZTF(/LSST) are prototyping AVRO as an efficient distribution format for
> alerts for exactly the same reason CRTS used XMPP/Jabber with VOEvent a
> decade ago. Why write something ourselves when industry has already
> produced something that seems to do what we want and has some nice
> additional functions as well.
> >
> > — Matthew
> >
> >
> >> On Nov 5, 2017, at 3:09 PM, Dave Morris <dave.morris at metagrid.co.uk>
> wrote:
> >>
> >> Hi all,
> >>
> >> On 2017-11-03 13:43, Roy Williams wrote:
> >>> The real problem though, lies with VOEvent itself being (a)
> >>> tied to XML and (b) in a state of flux.
> >>
> >> On 2017-11-04 15:51, Rob Seaman wrote:
> >>> I suspect that anybody who has navigated a standard through the IVOA
> >>> process would agree with Roy!
> >>
> >> I'm sorry guys, but I'm afraid I disagree.
> >>
> >> First, regarding the issue of the existing XML schema; I have seen no
> evidence to suggest that the XML based schema is causing problems for the
> VOEvent community.
> >>
> >> In fact, I was impressed and encouraged by the number of projects I met
> at a recent conference on transients and alerts who were either already
> using, or planning to use, the current XML based VOEvent tools to develop
> networks for coordinating follow up responses to transient events.
> >>
> >> As far as I can remember, none of the projects even mentioned XML as an
> issue.
> >>
> >> Next, the suggestion that an urgent and immediate transition to JSON is
> imperative for whatever reason is, to my mind, unfounded.
> >>
> >> To a great extent, the actual on the wire format is irrelevant. Both
> XML and JSON are text based formats, so they will both need to expand
> numeric values into strings of ASCII characters, with the associated loss
> of precision, and they will both need to use some form of encoding like
> base64 to embed binary objects such as images.
> >>
> >> However, this misses an important point. None of our scientists should
> ever encounter the raw on-the-wire data format. Software for implementing
> the science use cases should be developed based on higher level libraries
> such as the FourPiSky voevent-parse toolkit, which represents event
> messages as Python objects, not as XML or JSON.
> >>
> >> The fact that it might be easier to write a low-level parser in a
> particular language for one data format rather than another may influence
> our choice of formats in the future. But it is nowhere near enough evidence
> to justify disrupting an existing, working system that people are using in
> live production networks.
> >>
> >> Third, the suggestion that we should drop everything and convert to
> Avro in a mad rush to be ready for the ZTF/LSST streams as soon as they
> become available is unrealistic for several reasons.
> >>
> >> Not least of which is because the ZTF/LSST teams themselves are still
> experimenting with how to use Avro. I think we need to allow the ZTF/LSST
> team space to evolve their designs from their current experiments towards a
> production ready design before we decide to bet the future IVOA standard on
> it.
> >>
> >> So far, the only people I know of who have direct experience with Avro
> are the ZTF/LSST team, and they are not ready to publish anything yet.
> >>
> >> It is also important to note that based on the current plans LSST are
> only distributing the Avro based fire-hose streams to a limited number of
> community brokers, who will then filter the streams into smaller lower
> bandwidth streams of more specific event types.
> >>
> >> When this was discussed at the IVOA meeting in Santiago, the
> expectation was that in most cases the community brokers would probably
> issue their downstream alerts using the existing VOEvent format and
> protocol.
> >>
> >> This would enable users to plan and develop their software using the
> existing tools and test it using the existing VOEvent network to be ready
> to receive the new types of event when they become available.
> >>
> >> It is also important to note that the LSST project is not external to
> the IVOA. They are participating members of the IVOA and scientists and
> engineers from LSST were active in, and in some cases lead, the discussions
> at the conference in Santiago.
> >>
> >> We should not see them as an external threat developing ad-hoc designs
> that will fragment the community. They are part of this community, and we
> should work with them to develop the next generation of VOEvent. However,
> at this stage I think the conversation should be about prototypes and
> experiments, not standards and specifications.
> >>
> >> If we are concerned about how to prepare for this new generation of
> alert streams, then I suggest we can ask the LSST team to make a
> presentation at the next IVOA conference. Based on  what I have heard from
> talking with them, I think they may already be planning on presenting
> something at the May interop anyway.
> >>
> >> In addition, those of us who are working on developing community
> brokers for the LSST streams can play our part in ensuring that the message
> content of ZTF/LSST alerts are compatible with VOEvent, by proposing
> changes to both the VOEvent and the ZTF/LSST schema to bring them closer
> together.
> >>
> >> If, as a result of that, we discover specific structures or patterns in
> XML schema that we should avoid using because they would make a transition
> to a new format more difficult then I agree we should take care to avoid
> them where ever possible.
> >>
> >> We should use the lessons learned from our Avro prototypes and
> experiments to inform the evolution of the 2.x XML schema, but I don't
> think the prospect of Avro/Kafka based systems in the future  should
> prevent us evolving the existing XML based standards in the mean time.
> >>
> >> I think we should continue to evolve the 2.x XML schema in response to
> specific science use cases.
> >>
> >> As long as we ensure that minor point version changes 2.1, 2.2 etc. are
> backwards compatible with the existing tools and libraries, then I see this
> as part of the natural evolution that happens with any IVOA standard.
> >>
> >> I also want to reassure astronomers who are already using the existing
> 2.x schema, and the associated tools and libraries, like Comet and
> FourPiSky, that their work will continue to be supported in the future.
> >>
> >> When Pierre and Baptiste contacted me about their ideas for changes to
> the 2.x schema based on their specific use case, I encouraged them to
> present their proposals to the community for discussion. The suggestion
> that it should be proposed as a new version (2.1) of the standard came from
> me.
> >>
> >> On the other hand, if we can incorporate some or all of their proposals
> by issuing an errata, then I'm all for that, it would take less time and
> involve a lot less paperwork.
> >>
> >> As far as I am concerned, as long as they are compatible with the
> existing 2.x schema, do not disrupt the systems already in place, and solve
> well defined science use cases, then I think we should welcome their
> proposals.
> >>
> >> If there are specific XML constructs that we know will make it
> difficult to migrate to another format such as Avro at a later date, then I
> agree we should avoid using them wherever possible.
> >>
> >> However, I don't believe that the prospect of ZTF/LSST using Avro for
> their alert streams in the future should prevent us from evolving the
> existing VOEvent standards to meet current science use cases now.
> >>
> >> Evolving the current standards to meet current science use cases, and
> developing prototypes to learn about new technologies like Avro are not
> exclusive.
> >>
> >> We should be able to do both at the same time.
> >>
> >> Regards,
> >> Dave
> >>
> >> --------
> >> Dave Morris
> >> Research Software Engineer
> >> Wide Field Astronomy Unit
> >> Institute for Astronomy
> >> University of Edinburgh
> >> --------
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/voevent/attachments/20171106/549e43c4/attachment.html>