Time series data

[Doug Tody]

Hi Rob (and all you VOEvent guys) -

Can you say a bit more please about the VOEvent/HTN use-case where
time series data is concerned.

I gather you want to embed a time series observation directly within a
VOEvent packet - is this the case?  If so why - is such an observation
taken in real time when the event is detected?  If so, is this sort
of thing limited to time series, or could other forms of data be of
interest as well?

I should think another related use-case would be to follow up on
a VOEvent by issuing a query to an unrelated data service to find
time series or other data of interest on the object pointed to by
the VOEvent.  If so, a common solution might be desirable.

While up to a point one might want to integrate some observational
data directly into the VOEvent, eventually use-cases such as these
are likely to become complex enough that it will be desirable to
encapsulate the object instances so that they can be manipulated by
reusable code that knows nothing about VOEvent.  In short I think
there may be an architectural question here which needs more thought,
before deciding upon the best way to approach this.

	- Doug


On Sun, 10 Jun 2007, Rob Seaman wrote:

> Hi Doug, etc.,
> 
> > The issue of time series data came up briefly in the NVO telecon today.
> > Since all the attention has been focused on spectra little has been
> > done about this yet, although the Spectrum/SED data model upon which
> > SSA is based was always intended to be general enough to support time
> > series data as well.  Both are spectrophotometric sequences, with
> > the spectral coordinate varying in one case, and time in the other.
> 
> There's a lot of interest in VOEvent in seeing a practical time series format
> emerge soon.  Several other action items (e.g., XML signatures for
> authentication, and the representation of orbits) came out of the recent
> "Hotwired" workshop, confirming that VOEvent will continue to have a large
> cross-section with many other (I and N) VO standards and activities.
> 
> In this case, I wonder whether "spectrophotometric" only begins to cover the
> pertinence of time series for sky transient - and time domain, in general -
> alerts.  In addition to contribution from gamma rays to long wavelength radio
> waves, we had two talks on neutrino telescopes, several on classification
> schemes including ways to recognize signals embedded in sequences of
> temporally varying measurements of all types, coincidence testing, etc.  We
> take the meaning of the word "event" very liberally.
> 
> Pragmatically, the piratical individuals engaged in VOEvent and HTN related
> projects just want a time series standard that is utilitarian, simple, yet
> flexible.  They won't have infinite patience waiting for it to arrive,
> however.  To gauge both the completeness and elegance of an acceptable
> standard, the consensus would likely be that STC resides somewhere outside our
> comfort zone.  I doubt we could be motivated to accept such a complicated
> sub-schema again.
> 
> > It was mentioned that VizieR is a major resource for light curves
> > so I had a look to get a better feel for current practice.  What we
> > have now for SSA could probably already be used for light curves such
> > as we see in VizieR, but there are two areas where it could stand
> > improvement for this data.  The support for photometric systems could
> > be better - but we need this for spectra and images as well.  This was
> > a major topic of dicussion at the recent Spectroscopy in VO workshop
> > in Madrid for example.
> 
> If this is an ongoing debate, I'd suggest that a VOEvent-friendly prototype
> would specify a small number of standard systems that could be expanded later.
> 
> > The second thing is that it would be good to
> > be able to have multiple flux values (photometric systems or filters)
> > associated with each time value, as time series data often measures
> > the object in multiple standard filters/bandpasses/photometric systems
> > for each time sample.
> 
> Yes.  Something like multiple <params> per time step.
> 
> > A final issue is segmentation, as large time
> > series taken over a long period of time are often segmented, with
> > big time caps between the segments.
> 
> Yes, although VOEvent's follow-up citations provide one way of doing this.
> 
> > All of these issues came up in connection with SSA and spectra as
> > well, but they were second order features (for spectra) and lower
> > priority for the first version of SSA, hence were deferred.  But with
> > the completion of SSA and the Spectrum data model, Characterization,
> > etc., we are probably 90% of the way there already.
> 
> I guess the question I have is whether that 90% is sufficient for a functional
> prototype.  Alternately, how long would the remaining 10% take to converge?
> My sense of the debate at last week's workshop is that a VOEvent consensus on
> a 90% solution could be ready in a few week's, not months.  This seems much
> more aggressive than a similar consensus for SSA, especially as you are using
> words like "lower priority" and "deferred".
> 
> > In terms of the data interface, it appears that a SSA service could be
> > trivially modified to support time series data.   The query interface
> > would probably work as-is.  For simple light curves, CSV/TSV output
> > would probably be popular, and is essentially what existing services
> > such as VizieR return now.  HTML or JPEG for a direct graphical view
> > is also popular, and VOTable would be ideal for serious analysis;
> > again this is much the same as for simple 1-D spectra.
> 
> We're a publication format.  Some of the things you mention could be included
> as external references, but really we're just going to want to define some XML
> element to be embedded in our packets.  I would imagine these will include
> VOTable-like <params>.  If alternate representations are needed for purposes
> outside VOEvent, it seems to me we just need to ensure that our quasi-human
> readable packet format maps onto an underlying common time series data model.
> In that case, I would suggest that the VOEvent time series use case should be
> straightforward enough that the broader VO time series DM could and should
> simply support this.
> 
> If the SED DM is mature enough to start scribbling possible explicit time
> series representations, VOEvent would be delighted to evaluate these and
> comment.  If not, we will be tendering a simple, packet oriented
> representation of our own for other groups to comment.  In either case, we
> will be seeking a VOEvent related consensus quickly.
> 
> Rob
>