<div dir="ltr">Francois,<br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 13, 2017 at 1:46 AM, François Bonnarel <span dir="ltr"><<a href="mailto:francois.bonnarel@astro.unistra.fr" target="_blank">francois.bonnarel@astro.unistra.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>That's the point. We had the impression that VO-DML mapping into
VOTable will not help us to define a TAP schema for Provenance. The
version annouced recently by Laurent is not, as far as I understand,
seems to confirm that.<br></p><span class="gmail-">
<br></span></div></blockquote><div><br></div><div>We are happy to have more people participating in the process so we can get more done in the same amount of time. About three poeple for the past six months have been working (very, very part time) on data models (STC, Cube, TimeSeries, plus I know a couple of other people worked on CAOM as well), their serializations for a number of different cases, the mapping specification and document, along with several iterations of prototype implementations, tools, documentation, and demo software. Actually, I think that's a lot of stuff accomplished given the incredibly limited resources.</div><div><br></div><div>The more, the merrier.</div><div><br></div><div>That being said, Gerard indeed has been working on the VODML-TAP mapping for years (again, part time and along a whole bunch of other things), and he actually presented something at a couple of Interops and an ADASS, so maybe he should chime in on this topic.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><span class="gmail-">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>If ProvenanceDM has a valid VODML representation,
VOTable representation of its instances will be defined by
the standard mapping mechanism, without requiring an
ad-hoc serialization definition.<br>
</div>
</div>
</div>
</div>
</blockquote></span>
If we could find a standard way to go from "vo-dml xml"
represntation of any IVOA data model to TAP schema that would be
better than ad hoc serialisation. <br></div></blockquote><div><br></div><div>I think this is where we are not on the same page, really, because I am not even sure I understand your point. However I think/hope we agree on the goals.</div><div><br></div><div>PROV-VOTable is a serialization for *instances* of the model: it shouldn't be ad-hoc and defined in the model, it should be standardized, because it makes for a better architecture, reducing the burden on both producers and consumers.</div><div><br></div><div>So PROV-VOTable is an exchange mechanism for representing a Provenance Database, which by itself has a number of applications. It is not supposed to represent the result of a (Prov)TAP query, which is a different conversation.</div><div><br></div><div>ProvTAP is not a serialization, but a representation of the model with a different (kind of) schema. Mapping VODML to ProvTAP should be easier, because VODML maps directly to the relational model, so Types and Roles in VODML directly map to Entities and Relationships in a relational schema, with some complications typical of the Object Relational Mapping domain, like inheritance, but nothing that wasn't solved in industry decades ago. In this case instances are just rows in the tables. While it's certainly useful to standardize this mapping to make it more software-friendly.</div><div><br></div><div>While I did quickly read the Provenance WD I certainly didn't dive into the details, so I might well be missing something.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><span class="gmail-">
</span>We would be interested to see that. Seems to be a bit of hard work
however.<span class="gmail-"><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"><br></blockquote></div></div></div></blockquote></span></div></blockquote><div><br></div><div>The mapping is not that hard, what's hard is modeling STC in such a way that simple things are simple (in any syntax) and complex things are possible (in any syntax).</div><div><br></div><div>You'll see the examples in Santiago, and I am packaging them in such a way that they can be browsed independently from the Interop talks, annotated, and commented.</div><div><br></div><div>I think the complexity of VODML and its mappings is more of an assumption used to explain the complexity of the requirements on the models. STC is really complex, we are striving to simplify it for everybody. The current serialization, which was really simple to implement (the same level of complexity of any XML serialization, including VOTable 1.3), is helping to drive the feedback loop back into the models, improving them a lot.</div><div><br></div><div>The syntax is certainly perfectible and, as it's usually true for syntax, subject to heavy bike-shedding, so that's going to be fun, as are perfectible the models, but we are getting there, and the complexity of the mapping implementations was never a factor in my experience.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
Again "as far as I understand it", VO-DML mapping into VOTABLE
provides a set of TAGS in an xml schema extension of VOTABLE defined
to describe instances of a model. At some point it integrates
references to elements of a classical VOTable where the actual
data/metadata values are stored.<br></div></blockquote><div><br></div><div>This definition would also apply to the note Sebastien wrote and presented in Naples, where GROUPs, FIELDrefs, and optionally PARAM-refs are used to describe what's in the FIELDs and PARAMs themselves.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>I said "complex" because by looking at the examples it took me a while to understand what the structure is really doing and I was reluctant to strat writing such a structure for Provenance.</blockquote><div><br></div><div>Some indirection is introduced by the fact that we want current VOTables to remain unchanged for backward compatibility if that's what providers want: in this case you would just *add* annotations that old clients can just ignore. That's quite a feature if you think about it. The price to pay is the level of indirection in the new annotation.</div><div><br></div><div>This is why I am working on examples that do not use that kind of indirection, because for humans that's distracting. Machines don't really care about that. I think examples have circulated with too much indirection and that's unfortunate. I agree.</div><div><br></div><div>Along with the examples and the modeling, I am also working on documentation on what clients need in order to make sense of the data, in the context of STC, and implementations that can help data providers annotating their tables. That's a lot of work for a single person who is paid to do other stuff!</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
I said "complex" because by looking at the examples it took me a
while to understand what the structure is really doing and I was
reluctant to strat writing such a structure for Provenance.</div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
Moreover, this doesn't help me to build this classical VOTable
itself. I don't think the object/relational mapping of a data model
is currently done in the proposed spec. This is a real missing
point.<br></div></blockquote><div><br></div><div>Two answers here. First, ORM is indeed in the current spec. Is something missing/wrong/hard-to-implement? Please let us know!</div><div><br></div><div>Secondly, that's where collaboration is key. It's as if we were building HTML5, a complex standard that needs to be implemented by people with different skills in different domains for different use cases, and while you build it you can't ask questions on stack overflow, because nothing is there yet, we are building it from scratch!</div><div><br></div><div>[And actually, if you compare the specs, VODML and the mapping document are orders of magnitude simpler/shorter than HTML5, so we shouldn't even need the sheer amount of collaboration that it took to get HTML5 right.]</div><div><br></div><div>Especially given the limited resources we have we can't really afford working in isolation and just a few weeks before the Interops, with the interaction during the Interop being limited to contemplating the complexity of the world.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
Responses variation is almost infinite. The number and content of
possible involved columns is very large. So the model VO-DML mapping
to the VOTable response must be done dynamically, which probably
forbids easy usage of public libraries. </div></blockquote><div><br></div><div>This point has been made several times, and I believe Markus has responded several times that to him this is not a big deal and he's implemented that logic in his services. I don't want to speak for anybody, so I'll leave it at that.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">While in our approach the
"TAP schema" representation of the model contents everything to
describe the model and identify it in the response as long as we
consider it is a valid relational transcription of the vo-dml xml
representation of the model. Can we standardize that ?<br></div></blockquote><div><br></div><div>Yes! Your contributions to the work we've done so far will be invaluable!</div><div> </div><div>Thanks,</div><div><br></div><div>Omar.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><span class="gmail-"><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote">PROV-VOTable is nothing else
than the transcription in VOTable of the tables and
relationships of the <b>vodml-xml model</b>.<br>
[...]<br>
PROV-VOTable format can allow input/output operations of a
whole project provenance metadata in a database system.</blockquote>
<div><br>
</div>
</div>
Note my own edit in bold face.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">It's the same goal as your
PROV-VOTable, really, just standardized for all data models
rather than ad-hoc for a single model.<br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">We are definitely going to discuss
suggestions on the simplification of the syntax for the
Mapping WD, so if you have any such suggestions please send
them our way!</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">By currently focusing on important data
models like STC, Cube, and TimeSeries (and Provenance asap) we
can finally see everything coming together. With such
implementation experience, actual serializations of actual
models to look at and compare we can then go back to discuss
about the syntax more pragmatically.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Best,</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Omar.<br>
<div><br>
</div>
-- <br>
<div class="gmail-m_-3097556572441768119gmail_signature">
<div dir="ltr">Omar Laurino<br>
Smithsonian Astrophysical Observatory<br>
Harvard-Smithsonian Center for Astrophysics
<div><a href="https://maps.google.com/?q=100+Acorn+Park+Dr&entry=gmail&source=g">100 Acorn Park Dr</a>. R-377 MS-81</div>
<div>02140 Cambridge, MA<br>
<span><a value="+16174957227">(617)
495-7227</a></span></div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Omar Laurino<br>Smithsonian Astrophysical Observatory<br>Harvard-Smithsonian Center for Astrophysics<div><font color="#999999">100 Acorn Park Dr. R-377 MS-81</font></div><div><font color="#999999">02140 Cambridge, MA</font><br><span style="color:rgb(153,153,153)"><a value="+16174957227" style="color:rgb(17,85,204)">(617) 495-7227</a></span></div></div></div>
</div></div>