Parquet with STIL/STILTS/TOPCAT

[Tom Donaldson]

On behalf of David Shupe:


From: "Shupe, David L." <shupe at ipac.caltech.edu>
Date: Tuesday, March 9, 2021 at 10:53 PM
To: Mark Taylor <m.b.taylor at bristol.ac.uk>
Cc: Applications WG <apps at ivoa.net>
Subject: Re: Parquet with STIL/STILTS/TOPCAT

Mark — 

I am very happy to see this Parquet-capable prototype! Yes, it is an interesting capability for TOPCAT and STILTS.

This TOPCAT prototype worked successfully for me on a catalog for the Galactic Plane Infrared Polarization Survey, served by IRSA from the directory https://irsa.ipac.caltech.edu/data/GPIPS/catalog where there are also IPAC table and VOTable formats. (See https://irsa.ipac.caltech.edu/data/GPIPS/overview.html for all the context for these data.)

It also works on a number of other Parquet files I am working with; though not, as you noted with a Parquet dataset which is a directory containing several Parquet files. It also did not work on some Parquet files with multidimensional columns. I could make these available to you if it would be helpful.

So far in our work at IRSA, we have not tried storing additional metadata. I think Brian Van Klaveren’s followup message in this thread has the latest thinking on that.

-David
David L. Shupe, PhD (he/him/Dave)
Scientist, NASA/IPAC Infrared Science Archive
Lead, Zwicky Transient Facility

Caltech/IPAC
Mail Code MR-100, Pasadena CA 91125