Debrief from recent ASTERICS meeting

[Rob Seaman]

Hi Andy,

Thanks for the detailed report.

If "ESFRI, the European Strategy Forum on Research Infrastructures, is a strategic instrument to develop the scientific integration of Europe and to strengthen its international outreach,” then the astronomical time domain is the poster child for why an integrated infrastructure is needed. The activities you outline below that are occurring under all those interesting projects participating in ASTERICS-WP4 also have implications for WP3 and WP5, but also for VO and non-VO projects reached through the IVOA TDIG (as you presumably noted). Ultimately these must be tied back to the astronomical ur-processes of IAU on the science side and SPIE on the technical side.

As such, you should consider reporting on these activities at the SPIE Observatory Operations conference (26 June - 1 July, 2016) – especially since this will be conveniently located for your attendence ;-) The abstract deadline is in a few weeks, the time domain will again be a focus:

	http://spie.org/AS/conferencedetails/observatory-operations

Likewise, representatives of various projects you discuss have individually attended previous Hotwired workshops. As ASTERICS appears to be ramping up, the various work package teams should be encouraged to coordinate a strong presence at Hotwired V (10-14 October, 2016, registration yet to open):

	http://hotwireduniverse.org

While astronomical oversight for the IVOA flows from WG2 of IAU Commission B2:

	http://www.iau.org/science/scientific_bodies/commissions/B2/

The IAU Time Domain Working Group has an overlapping mission. Broader in scope than data management or ESFRI’s “research infrastructures”, the TDA WG has transitioned from Comm-B2 to its parent, Div-B:

	http://www.iau.org/science/scientific_bodies/working_groups/260/

Interested parties from both the science and technical worlds are invited/urged to join:

	https://pairlist10.pair.net/mailman/listinfo/tda_wg

Please forward this as appropriate to other ASTERICS folks.

> we have just been concluding some discussions about time domain science within WP4 of the ASTERICS project and I thought some of this was worth reporting to the group. For those who don't know, ASTERICS is an EU-funded project meant to plan and implement data-infrastructure for major upcoming European "ESFRI" facilities - SKA, CTA, KM3NET, LIGO/EGO, and ELT. (LSST is not one of these, but its kind of the elephant in the room). WP4 is specifically about the role of VO in the data infrastructure. The heavy-lifting grid-style stuff is elsewhere, in WP3. Much of the discussion of course centred around the precursors and pathfinders - LOFAR, Antares, HESS, LIGO/VIRGO etc. I spoke to the meeting about our experience with PanSTARRS, and the UK plans in LSST (MOA about to be signed!!)

It took a little digging, but Genova, et. al., appears to provide the most complete context for this from the Euro-VO side:

	http://www.sciencedirect.com/science/article/pii/S2213133715000190

And the ASTERICS wiki:

	https://www.astron.nl/asterics/doku.php?id=open:start

maps the acronyms to the numbered work packages (WP4=DADI, WP3=OBELICS):

	https://www.asterics2020.eu

Please let us know if there are other related document repositories or forums online. It looks like WP5, CLEOPATRA = "Connecting Locations of ESFRI Observatories and Partners in Astronomy for Timing and Real-time Alerts”, also has responsibilities overlapping TDIG/VOEvent:

	https://www.asterics2020.eu/cleopatra.html

> There was lots of interesting discussion, but here are a few key things. Most of the issues will be familiar to people on this list, but still worth saying, especially as ASTERICS-WP4 can be a route to addressing some of these issues.
> 
>      andy lawrence
> 
> ==Existing experience==
> The gravy-wave and neutrino folk have already been using VO Event, as well as GCN, and have used them to trigger real follow-up observations. They already appreciate that VO Event gives better automation.

Perhaps this success of VOEvent can be brought to the attention of the IVOA Exec (that is, by those outside the TDIG)?

> In particular the neutrino folk have set up a network of collaborators to share streams of VO Events (AMON : http://amon.gravity.psu.edu/) LOFAR, despite earlier stated intentions, seems not be generating a stream of alerts; they can receive and respond to alerts from others, but they do this "by email”.

John can comment, but perhaps his relocating to the Garden State has something to do with that. On the other hand, Four Pi Sky (http://4pisky.org) seemed to be going gangbusters at Hotwired. Is there feedback on projects that are using Comet?

> I described our experience with the Belfast-based FGSS system for producing PanSTARRS transients and following them up. This was scientifically very succesful, but could best be described as "semi-automated", relying on both automated junk-filtering, and second-stage grad-student eyeballing - right at the edge of what works.

There have been several talks on the difficulty of real-bogus detection for surveys.

> I also described our experience in constructing historial light curves for selected transients by combining data from several archives.

This continues to be chatted up at IVOA but could benefit from somebody cracking the whip. Perhaps ASTERICS could help with that.

> ==Astroparticle alerts==
> The event rates expected from gravy waves, neutrinos, and gamma-rays are quite modest; but they all want to be able to issue alerts within seconds and potentially have multiple facilities respond in a kind of cascade. This is driven by how fast some likely counterparts (e.g. GRB afterglows) respond. They seem to think so far that VOEvent is fit for purpose, but we encouraged them to write down their experiences and make critical asssessments. The issue of VTP vs XMPP briefly raised its head.

The VOEvent standard is distinct from VTP. Various parties long since looked at XMPP as an alternative; there is no reason different transport methods can’t coexist. In particular, transport for high-value/low frequency event streams like you mention need not be the same as for high frequency streams like LSST.

> These facilities are generally assuming that they are sharing events with other projects with whom they have MOAs, rather than public broadcasting, but some are considering a kind of mixed economy. There was some discussion about whether or not they needed brokers, whether streams could have auth/auth attached rather than just being hidden by obscurity.

The word “broker” means different things to different people. Most basically it just means a node that either turns metadata into events and passes them on, or receives VOEvent messages and provides the metadata to a client application via some API, or perhaps receives event packets from multiple sources (an aggregator) or sends them to multiple destinations (fan-out) or throttles the stream(s) via some criteria (a filter).

So yes, they need a broker to publish and a broker to subscribe and those brokers may also implement better security than simple obscurity.

> ==Optical alerts==
> Our experience with PanSTARRS/FGSS shows that full automation of QA is the number one problem. (I have some numbers for those who are interested).

I presume this is real/bogus again?

> However even when you reach the stage of believing all the transients, we were right on the edge of what a research team could cope with; using brokers to filter LSST transients, and handlers of some kind to act on them selectively, will clearly be crucial. This is potentially a volume issue (XML verbosity etc) as well as a CPU issue (parsing, processing, deciding).

Various groups have worked on these issues over the years, e.g., ANTARES at the University of Arizona currently.

> Although LSST is not an ESFRI project, many other projects will want to consume and act on LSST transients, so the partners are concerned by this issue. So... we are interested in high event rate stress-testing, experiments in compact serialization, and use of brokers. (I suspect we will be able to resource this through the new UK LSST funding, but I will contact Jeff/John/Mario about this separately.)

Standing up actual (scalable) infrastructure has been an issue with no identified funding. If ASTERICS can help make this happen, people will bless their name. On the other hand, we have had numerous pilot projects and the time is ripe to commit to technology choices.

> ==Radio alerts and response==
> We encouraged the LOFAR rep attending (Roberto Pizzo) to begin some experiments in VOEvent creation/consumption. We stressed that even some simple experiments will help, because we need feedback if VOEVent and VTP don't suit them in some way; reporting experience with brokers such as COMET would also be useful.

Again, please make sure that they understand that VOEvent and VTP are distinct issues.

> ==Other VOEvent issues==
> I raised other issues that have sometimes come up, such as verification and discoverability, but nobody seemed very exercised about these.

Lack of overt interest doesn’t mean such issues are unimportant. When the big surveys go live people will run into the fundamental engineering requirements, including these.

> ==Light curve use cases==
> There was considerable interest in standardising the approach to time series, so that it is easy to mix and match data from various archives. However, it was clear that a collection of use cases was needed if we are to concentrate effort, and not waste effort on unnecessary things. It was pointed out that Enrique Solano and David Ciardi are writing a white paper along these lines, so what we need to do is to collect any use cases and feed to them.

We’ve had a lot of talk previously, but more power to anybody who can move the effort forward. You might also raise this question in the IAU time domain WG. As with many VO issues, the real problem is getting astronomers to participate in requirements discovery.

> ==Time series standards== 
> We discussed "Simple Time Series" and whether to develop that, versus the alternative approach of adapting SSA or upcoming datacube datamodels. There was not a clear conclusion to this debate, but on balance a feeling that a dedicated datamodel and access protocol did make sense, and that it shouldn't be hard.

“It shouldn’t be hard” sums up so much of the IVOA history...

> One thing that people clearly wanted was simple discoverability of services with a light curve construction capability.

God bless ‘em!

> One minor specific thing that came up from the CTA people was that a simple timestamp was not enough, because data was accumulated on and off over an extended period. Possibly start-end and live-time would be needed. Another specific issue is how to specify types of flux etc. Possibly this is covered by linking to Photometry DM.

Timestamps are a developing issue outside astronomy, too. This could also benefit from an airing in the IAU TDA WG and one could point out that a time series maps flux-like quantities to time-like quantities via some sort of stamping operation. That said, the CTA requirements are likely fairly boutique.

> ==Time series tools==
> There was brief discussion about whether some kind of graphical tool was needed along the lines of Topcat or Aladin or Splat. The mood of the meeting was probably not; astronomers doing this sort of thing would cook their own with Python etc. However, I think this needs a bit more thought.

It is likely true that a single tool won’t address all applications, but perhaps a toolkit (Python or otherwise) could simplify projects rolling their own.

Rob Seaman
Catalina Sky Survey