From m.b.taylor at bristol.ac.uk  Thu Jan  9 16:49:16 2025
From: m.b.taylor at bristol.ac.uk (Mark Taylor)
Date: Thu, 9 Jan 2025 15:49:16 +0000 (GMT)
Subject: VOParquet document
In-Reply-To: <41b830db-a62d-5395-6320-7183c6de7053@andromeda.star.bris.ac.uk>
References: <274a171-251-dcba-aa47-bff6b97cbe6c@andromeda.star.bris.ac.uk>
 <41b830db-a62d-5395-6320-7183c6de7053@andromeda.star.bris.ac.uk>
Message-ID: <2b99826b-a4ae-611-d619-8d3a508ceb77@andromeda.star.bris.ac.uk>

Hi apps,

the draft VOParquet document (https://github.com/ivoa/voparquet)
currently has one Issue under active discussion
(https://github.com/ivoa/voparquet/issues/10),
but hopefully that can be resolved (at least as far as it needs to be
for this document) fairly soon, and then I plan to approach Apps
management with a view to publishing it as a Note, or possibly
an Endorsed Note.

In the mean time I have written a VOParquet validator which should
help people to implement this convention correctly.  For details see:

   https://wiki.ivoa.net/twiki/bin/view/IVOA/ParquetInAstronomy#Validator

If you are interested in generating or consuming files following 
this convention, I encourage you to look at the document, 
attempt to implement it, and post information about your implementation 
and possibly example VOParquet files at:

   https://wiki.ivoa.net/twiki/bin/view/IVOA/ParquetInAstronomy#Implementation%20Status

Even though this note is intentionally provisional, it will be easier 
to resolve any issues with the text now than after publication.

Thanks

Mark

On Tue, 17 Dec 2024, Mark Taylor wrote:

> Hi Apps,
> 
> following some editorial activity, the VOParquet Note is now in
> what I consider a suitable state for publication.
> You can see it here:
> 
>    https://github.com/ivoa/voparquet
> 
> As advertised, it is open for comment and contribution until the end
> of December, after which we plan to move towards publication.
> 
> Although it's not mandatory for Notes, we'd like to have interoperable
> implementations of this convention, and maybe even a validator.
> So:
> 
>  - If you have implementation code relevant to this proposal
>    (software that reads and/or writes parquet files in a way
>    that's compatible with the proposed VOParquet convention),
>    please add an entry to the "Implementation Status" section
>    of the wiki page
>    https://wiki.ivoa.net/twiki/bin/view/IVOA/ParquetInAstronomy
> 
>  - If you have comments on the VOParquet document text, please
>    follow up this email and/or raise an issue in the repository at
>    https://github.com/ivoa/voparquet , preferably before the
>    end of the year.
> 
> Thanks,
> 
> Mark
> 
> On Tue, 26 Nov 2024, Mark Taylor wrote:
> 
> > Dear Apps,
> > 
> > I have drafted a document on use of the parquet file format in the VO,
> > specifically defining a covention for use of VOTable to add rich
> > metadata to parquet files.  You can find it on github here: 
> > 
> >    https://github.com/ivoa/voparquet
> > 
> > I have started a few Issues to invite comments on particular aspects of
> > the text; if you are interested please contribute to these, create new
> > issues, or submit pull requests.
> > 
> > As agreed in the Apps telecon of 5 November and at the Malta Interop,
> > the aim is to get agreement on this on a short timescale, so that
> > projects that are writing or about to write parquet files in bulk
> > can do so interoperably.
> > 
> > My suggestion therefore is to gather comments and make updates by
> > *the end of this calendar year*, so it can be published
> > (as a Note or possibly towards Endorsed Note, TBD) in the first
> > weeks of 2025.  If there are major disagreements that schedule
> > might have to slip, but I'm currently optimistic that won't
> > be the case.
> > 
> > If you think that this timescale is too ambitious, please follow up 
> > this message.
> > 
> > Thanks
> > 
> > Mark
> > 
> > --
> > Mark Taylor  Astronomical Programmer  Physics, Bristol University, UK
> > m.b.taylor at bristol.ac.uk          https://www.star.bristol.ac.uk/mbt/
> > 
> 
> --
> Mark Taylor  Astronomical Programmer  Physics, Bristol University, UK
> m.b.taylor at bristol.ac.uk          https://www.star.bristol.ac.uk/mbt/
> 

--
Mark Taylor  Astronomical Programmer  Physics, Bristol University, UK
m.b.taylor at bristol.ac.uk          https://www.star.bristol.ac.uk/mbt/