<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi FX,<br>
<br>
Quartiles are a first step. I think they are enough for the
moment, but of course, if most of our users people want more
quantiles, we can decide to compute them.<br>
<br>
I do not accept approximated values for the moment, but indeed, it
could be interested to change that if we have to compute any
quantiles. Thank you for the suggestion of the algorithm ; I think
having already taken a quick look at it, but a closer look will be
of course necessary to compute quantiles.<br>
<br>
Cheers,<br>
Grégory<br>
<br>
<br>
On 21/10/2016 08:58, Francois-Xavier Pineau wrote:<br>
</div>
<blockquote
cite="mid:807dbada-a8f2-ea72-c79b-97320b5f1fa4@astro.unistra.fr"
type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<p>Greg,</p>
<p>For quartiles (why not generalizing to any quantiles), you
probably have to specify whether you accept approximated values
or not.</p>
<p>If approximated values are accepted, you can compute them for
data of any size in a single pass with a very limited amount of
memory.</p>
<p>A good algorithm is for example provided in the Numerical
Recipes: "Single-pass estimation of arbitrary quantiles".</p>
Kind regards,<br>
<br>
fx<br>
<br>
<div class="moz-cite-prefix">Le 20/10/2016 à 23:07, Gregory
Mantelet a écrit :<br>
</div>
<blockquote
cite="mid:780a48ce-8520-4b4d-b156-54040c2736e7@ari.uni-heidelberg.de"
type="cite">
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<p> </p>
<div class="moz-text-flowed" style="font-family: -moz-fixed;
font-size: 12px;" lang="x-unicode">Dear DAL and Apps members,
<br>
<br>
Since I do not attend to this interop, I would like to
highlight quickly one of my last development concerning TAP
because I think it may be interested to either do the same in
your own TAP service or merely use it. As suggested by the
title of this email it is about adding metadata in TAP. <br>
<br>
(I send this email also to Apps because of the last point I
make in this email: a compatibility with a new feature of
TOPCAT) <br>
<br>
<br>
** Columns metadata <br>
<br>
The idea is to add basic statistics like a count, min, max,
... for some numerical columns of tables published in a TAP
service. For that, I have just added the following columns in
TAP_SCHEMA.columns: <br>
<br>
- min_value <br>
- max_value <br>
- mean <br>
- std_dev <br>
- q1 (i.e. first quartile) <br>
- median (i.e. second quartile) <br>
- q3 (i.e. third quartile) <br>
- filling (number of rows having a NOT NULL value for
this column) <br>
<br>
Except for "filling" which must be an integer (INTEGER or
BIGINT in PostgreSQL), I have chosen to set all these columns
as DOUBLE PRECISION since most of the columns to describe are,
in the "worst" case, double values. <br>
<br>
When no statistics can be provided for a column, all these
additional metadata would be NULL. <br>
<br>
<br>
** Tables metadata <br>
<br>
In addition, I have also added another column in
TAP_SCHEMA.tables: <br>
<br>
- row_count (of type INTEGER or BIGINT) <br>
<br>
<br>
** VOSI description of tables <br>
<br>
Since in TAP all tables and columns metadata MUST be the same
in TAP_SCHEMA and /tables, I have also updated our /tables
resource. <br>
<br>
Besides, on a recommendation of Mark Taylor, I designed and
linked a simple XSD schema in order to have a valid XML
document. You can find this schema at the following address: <br>
<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://gaia.ari.uni-heidelberg.de/tap-stats.xsd">http://gaia.ari.uni-heidelberg.de/tap-stats.xsd</a>
<br>
<br>
<br>
** Visibility in TOPCAT <br>
<br>
Thanks to Mark Taylor, any custom metadata (non-standard TAP
columns) can be displayed in the last version of TOPCAT. Thus,
all the statistics described above can be seen there for our
Gaia TAP service (n.b. you can find this TAP service easily in
the registry with the keywords "Gaia" and "ARI", but in case
you can not, here is the root access URL: <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://gaia.ari.uni-heidelberg.de/tap">http://gaia.ari.uni-heidelberg.de/tap</a>).
<br>
<br>
<br>
** Last words... <br>
<br>
According to me all these basic statistics may be useful to
discover the content of a table, especially when this one is
as large as Gaia, PPMXL, 2MASS, ... It can indeed prevent some
users to perform apparently simple and short queries such as
"SELECT COUNT(*) FROM a_big_table" which on the contrary to
what most people think is not often a quick query on large
tables. Having already computed such information is then an
economy of time and resources for the users and the server. <br>
<br>
Finally, I am not trying to convince anybody to have such
metadata, but I just want to highlight a possible extension of
TAP helping in simple data discovery. Besides, this use-case
also demonstrates how easy it could be to add more simple
metadata inside a TAP service. So maybe it could be
interested, if possible, to write an appendix about that in
the next version of TAP or just as an IVOA note. What do you
think? <br>
<br>
If anybody has questions or wants further details about the
TAP "extension" I presented here, do not hesitate to ask ; I
am not at the interop, but I am fully available by email <span
class="moz-smiley-s1" title=":)"></span> <br>
<br>
Regards, <br>
Grégory <br>
<br>
<br>
PS: For those who are interested, I also provide histograms
and sky-maps (using Healpix) for most of the published columns
on the page <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://gaia.ari.uni-heidelberg.de/tap/tables">http://gaia.ari.uni-heidelberg.de/tap/tables</a>.
Both can be downloaded as images but also as tables that you
can then display/process as you want (e.g. display the
histogram in TOPCAT, display and navigate inside the Healpix
map in Aladin, ...). <br>
<br>
<br>
</div>
</blockquote>
<br>
</blockquote>
<p><br>
</p>
</body>
</html>