From c.gheller at cineca.it Fri Sep 1 02:43:16 2006 From: c.gheller at cineca.it (Claudio Gheller) Date: Fri, 1 Sep 2006 11:43:16 +0200 (MEST) Subject: VOTable for simulations In-Reply-To: <001c01c6cd10$d869c8e0$cc48b782@gavopc1> References: <001c01c6cd10$d869c8e0$cc48b782@gavopc1> Message-ID: <44F800B3.9090404@cineca.it> Ciao Gerard, > >>From their example I gather that arraysize="41x41x41x3" means "three data >cubes of dimensions 41x41x41", >not "one 3D-vector valued datacube of dimensions 41x41x41". >"41x41x41" would mean "41 2D datafields of dimension 41x41". I think that >therefore a 3D vector field >could/has to be encoded as (for example) > > Ok, in order to keep alligned with VOTable standards let's adopt this solution. EACH CONTAINS ONLY a SCALAR FIELD, that can be both a real scalar quantity and a component of a multidimensional array. In this way we could lose the "nature" of the quantity (scalar or vector), but this could be recovered by the associated UCD. > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xmlns="http://vizier.u-strasbg.fr/xml/VOTable-1.1.xsd"> > > > datatype="float" > arraysize="41x41x41x1" unit="km/s" /> > datatype="float" > arraysize="41x41x41x1" unit="km/s" /> > datatype="float" > arraysize="41x41x41x1" unit="km/s" /> > > > > > >
>
>
> >This makes the content of the individual field components more explicit. >Each gets it own UCD for example. > > >I have removed the rank attribute for the moment. > > Fine, we can get the associated information from the "arraysize" >There is no way yet to specify the spatial coordinates of the grid cells. >For a grid one can specify >the spatial coordinates in general in a shorthand way, for example using a >set of standard parameters >as in the FITS array keywords (see >http://fits.gsfc.nasa.gov/standard21b/fits_standard.pdf 5.4.2.5), >CRPIXn, CDELTn etc. I think we need to specify something like that here as >well, it is definitly more >efficient than having separate cubes with the coordinates. >Luckily in general our coordinate system will not require the full WCS like >formalism in general. > > I will take a look at these FITS standards. Obvioulsly, for a regular grid we do not need to specify the grid points but only the number of points and the mesh resolution per each dimension. I guess this information can be specified at the level of the element (but I'm not sure it is the right place). Obviously if data are mixed, mesh and particles in the same file, the information refers only to the mesh components. >Then, though it is possible to use this same formalism for particle data as >well, I think there the tabular approach is more natural in many >circumstances. In particular in the work that I have been doing with >databases, >the natural representation of more complex individual objects is as a table, >with all the properties, including >now the positions, in a row. The way to store such tabular datasets in >binary form is specified exactly in the >the existing VOTable spec, in section 5.3. An equivalent C-struct oriented >format in binary files is what I have encountered consistently for more >complex objects coming for example from the postprocessing of cosmological >simulations at the MPA in Garching. > >But you're right that many people also store particle data in individual >arrays for each particle property. > > I agree that we should support both the approaches. The tabular approach is more natural, since you can find "close to each other" all the information related to a particle. However, often the user needs to select only one (or few) of the properties of the particles. For example, for visualization you may need to load only the position. For the mass functions, the positions and the masses. For the velocity dispersion, the three component of the velocity. In this cases, it is much faster and more efficient to store each variable separately (you load all the data in a single contiguous read operation). Furthermore, many codes use this kind of storage technique (e.g. Gadget and Enzo, between the most popular). >That is more naturally mapped in the sense of your rank 1/2 examples. Making >the same adjustment as above for >the datacubes I would propose to allow also something as in the following >example: > > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xmlns="http://vizier.u-strasbg.fr/xml/VOTable-1.1.xsd"> > > > datatype="float" > arraysize="100000x1" unit="Mpc" /> > datatype="float" > arraysize="100000x1" unit="Mpc" /> > datatype="float" > arraysize="100000x1" unit="Mpc" /> > datatype="float" > arraysize="100000x1" unit="km/s" /> > datatype="float" > arraysize="100000x1" unit="km/s" /> > datatype="float" > arraysize="100000x1" unit="km/s" /> > > > > >
>
>
> > > > At this point, I would drop the "x1" in the "arraysize". >In the latter case we still need something to distinguish between particle >data and image data. >Your rank basically does that, just the name might be unfortunate. We might >want to be more explicit >about the kind of data that is stored, an attribute with values MESH, N_BODY >maybe ? > > ok... name of the attribute? maybe "geometry"... >In your example you use an HDF5 binary file. VOTable does not support that, >though it does support FITS, >I suppose as BINARY table (see VOTable spec section 5.2). Is there a natural >mapping from VOTable key words >to HDF metadata structures ? Or shall we first concetrate on the binary >serialisations specified in VOTable ? > > At the moment I would focus on "pure" binaries. HDF5 was cited only as an example. I go on... Thanks a lot for all the comments!!! C. -- ------------------------------------ Dr. Claudio Gheller, Ph.D. High Performance System Division CINECA - Bologna - Italy Tel. +39-051-6171560 Fax. +39-051-6137273 ------------------------------------ From gerard.lemson at mpe.mpg.de Mon Sep 4 02:07:38 2006 From: gerard.lemson at mpe.mpg.de (Gerard) Date: Mon, 4 Sep 2006 11:07:38 +0200 Subject: Moscow Message-ID: <002a01c6d001$91d7edf0$cc48b782@gavopc1> Dear collegues I will not be able to attend the Moscow interop meeting. However, there will be a session, which Herve Wozniak will chair. We are currently discussing the program and will notify you of an agenda later this week. If you feel that there are certain things that should be on the agenda, please let us know. Best regards Gerard Lemson -------------- next part -------------- An HTML attachment was scrubbed... URL: From lds at ast.cam.ac.uk Mon Sep 11 05:50:48 2006 From: lds at ast.cam.ac.uk (Laurie Shaw) Date: Mon, 11 Sep 2006 13:50:48 +0100 (BST) Subject: draft and meeting In-Reply-To: <44FD899B.5030503@cineca.it> References: <44FD899B.5030503@cineca.it> Message-ID: Hi Everyone, Attached is a first draft of the technical note that I have been writing for simulation Semantics. Based on the discussions at and after the Victoria meeting, and on early work on the Simulation data models, I break down the meta-data requirements to describe simulations into a set of categories. Each category itself must be labelled by a UCD, for which have made initial suggestions. The typical contents of a category varies between single characters or numbers, ascii text (or urls, names etc) and words from the standard vocabulary (which is still largely to be defined). I've attached the doc in both ms word and pdf formats. Would be great to have a brief discussion about this before the Moscow meeting -- I still have time to make lots of changes, if needed! Thanks, Laurie -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note.pdf Type: application/pdf Size: 94777 bytes Desc: URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note.doc Type: application/msword Size: 117760 bytes Desc: URL: From herve.wozniak at obs.univ-lyon1.fr Tue Sep 12 01:00:14 2006 From: herve.wozniak at obs.univ-lyon1.fr (Herve Wozniak) Date: Tue, 12 Sep 2006 10:00:14 +0200 Subject: draft and meeting In-Reply-To: References: <44FD899B.5030503@cineca.it> Message-ID: <4506690E.7030507@obs.univ-lyon1.fr> Hello Laurie, thanks for your draft. my first (and quick) comments are included in the document using word revision tool. For discussions. Regards, Herve Laurie Shaw a ?crit : > Hi Everyone, > > Attached is a first draft of the technical note that I have been writing > for simulation Semantics. Based on the discussions at and after the > Victoria meeting, and on early work on the Simulation data models, I break > down the meta-data requirements to describe simulations into a set of > categories. Each category itself must be labelled by a UCD, for which have > made initial suggestions. The typical contents of a category varies > between single characters or numbers, ascii text (or urls, names etc) and > words from the standard vocabulary (which is still largely to be defined). > > I've attached the doc in both ms word and pdf formats. Would be great to > have a brief discussion about this before the Moscow meeting -- I still > have time to make lots of changes, if needed! > > Thanks, > > Laurie -- _______________________________________________________________ Herve WOZNIAK Centre de Recherche Astronomique de Lyon UMR 5574 Universit? Lyon I - CNRS - ENS-Lyon 9, avenue Charles Andre, F-69561 Saint Genis Laval cedex Phone: (+33) (0) 478 868 388 Fax: (+33) (0) 478 868 386 http://www-obs.univ-lyon1.fr/herve.wozniak/ -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note_HW.doc Type: application/msword Size: 147968 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note_HW.pdf Type: application/pdf Size: 68850 bytes Desc: not available URL: From Franck.LePetit at obspm.fr Wed Sep 13 13:03:24 2006 From: Franck.LePetit at obspm.fr (Franck Le Petit) Date: Wed, 13 Sep 2006 22:03:24 +0200 Subject: draft and meeting In-Reply-To: References: <44FD899B.5030503@cineca.it> Message-ID: Dear Laurie, Fabrice and I had a look at the draft for simulation semantic. We think it is a very good draft and just have a few comments : 1 - In part 4 : Physics of the code We think that in the three cases: Physical Objective, Physical Processes and Subject, le list of possible values have to be perfectly defined or a search in the registries will be unable to succeed because of synonyms. MNRAS keywords and some words used in VO- events should cover most of the possibilities. If you agree we should explicitely add this in the text as it is done for Physical Objective. 2 - In part 5.1 : Algorithm As you say, here we will have a tough work to define the list beeing precise enough without getting lost in the details. Up to now, the list consider only keywords for hydrodynamical keywords, N-body, etc. I wonder if we could add to this list ETL, LVG, Monte-Carlo, Ali to cover algorithm used in radiative transfer codes. It should be easy to cover all the numerical methods for rad. transf. codes easily. This is a point we should talk about in Moscow. 3 - In part 5-3 : Protocol This section is indeed usefull in the case of the search for a service to be used on grids. In this case we should provide not only informations about parallelisation but all the technical requirements to run the code. For exemple compilers, libraries (Blas, Lapack, NAG, ...). This makes me think to another problem. In the case of a code launched on a grid via a VO-service such as Astrogrid, do we plan at any level to describe the requirements in hardware: memory, processor... I think that may be useful at a point but maybe it is not required at this level of our work. 4 - In part 6-3 : Algorithm Parameters I think we should not focus too much on this part yet since it is linked to the datamodel to describe results. This part is fine as it is presently. Best regards Franck & Fabrice. ------------------------------------ Franck Le Petit LUTH Observatoire de Paris 5 Place Jules Janssen 92190 Meudon France Tel: (33) 1 45 07 75 66 ------------------------------------ Le 11 sept. 06 ? 14:50, Laurie Shaw a ?crit : > > Hi Everyone, > > Attached is a first draft of the technical note that I have been > writing > for simulation Semantics. Based on the discussions at and after the > Victoria meeting, and on early work on the Simulation data models, > I break > down the meta-data requirements to describe simulations into a set of > categories. Each category itself must be labelled by a UCD, for > which have > made initial suggestions. The typical contents of a category varies > between single characters or numbers, ascii text (or urls, names > etc) and > words from the standard vocabulary (which is still largely to be > defined). > > I've attached the doc in both ms word and pdf formats. Would be > great to > have a brief discussion about this before the Moscow meeting -- I > still > have time to make lots of changes, if needed! > > Thanks, > > Laurie > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From deyoung at noao.edu Wed Sep 13 14:44:22 2006 From: deyoung at noao.edu (Dave DeYoung) Date: Wed, 13 Sep 2006 14:44:22 -0700 Subject: draft and meeting In-Reply-To: References: <44FD899B.5030503@cineca.it> Message-ID: Dear Laurie et al.; This seems like a very good start on this problem. I have a few comments to add to those of the others. First, in general we are not considering just "simulations" if one wants to be precise. Codes such as CLOUDY and various radiative transfer codes clearly need to be included, but they are not strictly simulations. It's all theoretical astrophysics, so perhaps a more general term is needed. More specifically: 3.2 Name of Code - Not all codes have names. e.g., MHD-TVD codes used here are not yet "named" because the authors think it is frivolous. How should these be included? 4.2 Physical Processes - The vocabulary here is very incomplete, as was noted by Laurie. 4.3 Subject - There seems to be a lot of overlap or redundancy here with section 4.1, which is supposed to give a description of the phenomenon being simulated. 5.3 I agree with others that the specification of the library is useful. 6.3 Algorithm Parameters - Here there is also a lot of overlap with section 5.1, Algortithm. Perhaps these two should be merged or at least placed in the same general section. I am sorry I will be unable to attend the Moscow meeting. I hope you have a very productive set of discussions there. Best wishes, Dave De Young From mcs at iaa.es Wed Sep 13 15:45:24 2006 From: mcs at iaa.es (=?ISO-8859-1?Q?Miguel_Cervi=F1o?=) Date: Wed, 13 Sep 2006 17:45:24 -0500 Subject: draft and meeting In-Reply-To: References: <44FD899B.5030503@cineca.it> Message-ID: <7C7CFC36-75F5-48B3-AEA2-E39A11B701DC@iaa.es> Dear all, I just edited the doc file by Herve and I have included some text and my own coments. I also have included the coments by Frank, Fabribrice and Dave. (I don not know what would be the best way to modify the document, but in any case I just do it in this way). In general I agree with all comments. Laurie, thanks alot for the document, I think that in general very good Best regards miguel P.D: I am sorry but I will not be in Moscow meeting P.D.2: There has been a draft about characterization in the DM group. I think that most of the things they point out there are addressed in some way here. However, in the characteriztion draft there has been an special enfasis in define the "bounds" of the data included in the table (in the case of an spectrum it would be the limits where the spectrum is defined) I do not know if the current draft on smenatics also cover the bounds of theoretical models for n-body simulations etc... -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note_v2.doc Type: application/octet-stream Size: 176640 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note_v2.pdf Type: application/pdf Size: 315969 bytes Desc: not available URL: From debray at obs-besancon.fr Thu Sep 14 17:56:27 2006 From: debray at obs-besancon.fr (debray at obs-besancon.fr) Date: Fri, 15 Sep 2006 02:56:27 +0200 Subject: draft Message-ID: <20060915025627.uodcx2giypsg8cgc@webmail.obs-besancon.fr> Dear Laurie, dear members of the Theory Interest Group, First, thanks to Laurie for the setting up of the "Theory Semantics" technical note. I would also like to make a few further comments: - in the "Introduction", should not the registration of "theoretical on-line services" be mentionned as well as of codes and simulation data sets, in accordance with S?bastien Derri?re's talk at the Registry session of the last interop meeting ? - In 2 "List of Categories" : * I would suggest to list here the complete list of Categories, therefore corresponding exactly to the subparagraphs of paragraphs 3 to 6 of the "Contents" ; "Type of Results" and Algorithm Parameters" are missing. * "Type of results" would be "d,c", similarly to "result format" * "Algorithm parameters" would be "d,c" * "Version of code" could be significant also for data (therefore should be "d,c") ; a simulation can be very (at least somewhat) different when obtained with a new version of a code ; so one has to know which version of the code it refers to. - 3.2 Name of code : As Dave de Young stated it, the word "name" is probably not the best one to refer to the code here. May be a vaguer word (appellation ?) should be used instead ; the associated ucd would then point to what the "appelation" is actually: * meta.id: identifier or name, although the recent discussion on the ucd-sci list, that Miguel mentionned in a previous mail, could make think that "id" refering to "identifier", could be considered in the future as something very specific (it has to be unique) and therefore should be distinguished from a name or a designation (but there is no meta.name at the moment), * meta.bib.xxx (author, bibcode) if the code is referred to by a publication reference * meta.ref.url in case of a reference to a web page * ... Notice that the second ucd word "comp.code" is not mentionned in the list of suggested UCDs at the end of the document. Two questions here: * can the atom "code" be used with a meaning different from what it has in other ucd words (meta.code) ? * is not it possible to use the already existing ucd "meta.software" instead ? - 3.3 Version of code : requiring the version to be a number may seem constraining ; it could be any string, especially the month or date of the release of the version ("jul2006", "2005-11-23", ...) ; I then agree with the precisions added by Miguel. - 4. Physics of Code : I think it would be more natural to put "Subject" before "Physical Objective" and "Physical Processes" ("which object ?", "what ?", "how ?") - 4.1 Physical Objective : there is probably no need to add the word "meta.main" at the end of the ucd as Laurie suggested to leave the possibility of multiple entries for (almost) all categories. - 5.3 Protocol the name of this category comes from the initial suggestion by Laurie but this name seems now unrelevant given all the related additional proposals. May be a broader category should be thought of here ("Computing" ?) under which several more specific informations are possibly supplied : * parallelisation (yes/no), vectorisation * if parallelisation, library used : OpenMP, MPI; way memory is used, ... * language(s) of the code : C, C++, Fortran77, Fortran90, ... * ... The above are relevant for codes, but some informative parameters could possibly be added for simulation data under a "Computing" header : * cpu time needed to compute the simulation (+ info on processor) * size of the simulation These informations could may be be useful in worflows: "given the size and cpu time needed, is it worth rerunning the program ?" - could have to do with SNAP as well - 6.2 Results Formats why not adding VOTable here which was agreed after Victoria to be data format used by theory specific services to send data products ? Very minor supplementary comments : - the "Abstract" should probably be more explicit in the final version - adding the page numbers on each page of the document would ease its reading - in par. 1 (Introduction), why talking about "snip of simulation code" whereas the aim is certainly to register very big codes ? (a wink ;-) in a tough text ?...) Finally, I have a suggestion which may be worth being discussed : as a goal of the present technical note is to provide clues to register codes and simulations and if the content of the note is more or less agreed in Moscow, would it be worth setting up a tentative "theory registry" (or ask people managing one of the registries to implement the theory extensions); the aim of it would be to have a testbench for trying to register existing simulation codes and data in the framework of the technical note and for simulating queries aiming at locating simulation codes/data. May be this would have to be discussed with big projects such as VO-Tech. With best regards, Bernard Debray -- Observatoire de Besan?on | http://www.obs-besancon.fr/ 41 bis, avenue de l'Observatoire | bernard.debray at obs-besancon.fr BP 1615 | Tel : (33) 3 81 66 69 28 25010 Besan?on cedex | Fax : (33) 3 81 66 69 44 France | Laurie Shaw wrote: >Hi Everyone, > >Attached is a first draft of the technical note that I have been writing >for simulation Semantics. Based on the discussions at and after the >Victoria meeting, and on early work on the Simulation data models, I break >down the meta-data requirements to describe simulations into a set of >categories. Each category itself must be labelled by a UCD, for which have >made initial suggestions. The typical contents of a category varies >between single characters or numbers, ascii text (or urls, names etc) and >words from the standard vocabulary (which is still largely to be defined). > >I've attached the doc in both ms word and pdf formats. Would be great to >have a brief discussion about this before the Moscow meeting -- I still >have time to make lots of changes, if needed! > >Thanks, > >Laurie ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. From mcs at iaa.es Fri Sep 15 10:25:14 2006 From: mcs at iaa.es (=?ISO-8859-1?Q?Miguel_Cervi=F1o?=) Date: Fri, 15 Sep 2006 12:25:14 -0500 Subject: draft In-Reply-To: <20060915025627.uodcx2giypsg8cgc@webmail.obs-besancon.fr> References: <20060915025627.uodcx2giypsg8cgc@webmail.obs-besancon.fr> Message-ID: <3A281F2A-E677-4579-BF86-60322FEF00CE@iaa.es> hello, I agree with the coments by Bernard, in particular > Notice that the second ucd word "comp.code" is not mentionned in > the list of > suggested UCDs at the end of the document. Two questions here: > * can the atom "code" be used with a meaning different from what > it has in > other ucd words (meta.code) ? > * is not it possible to use the already existing ucd > "meta.software" instead? Yes! I think that meta.software could do a quite better job: - it avoid collisions with "codification" or other flags - it also allows to reuse the same fileds (name, parameters etc..) for VOTables describing pipe-lines (data reduction and simulations) that would be useful for future virtual telescopes... > 6.2 Results Formats > why not adding VOTable here which was agreed after Victoria to > be data > format used by theory specific services to send data products ? > Also agree. However, it is supposed that at the end, whatever the data-format, it should be included in a VOTable, isn't it? Any case, the "theoretical results" i am produced for synthesis models are in ascii-VOTable. > Finally, I have a suggestion which may be worth being discussed : > as a goal of the present technical note is to provide clues to > register codes > and simulations and if the content of the note is more or less > agreed in > Moscow, would it be worth setting up a tentative "theory > registry" (or ask > people managing one of the registries to implement the theory > extensions); the > aim of it would be to have a testbench for trying to register existing > simulation codes and data in the framework of the technical note > and for > simulating queries aiming at locating simulation codes/data. > May be this would have to be discussed with big projects such as VO- > Tech. I aslo aggree :) Since I will not be in moscow, here there is some comments: At the ESA-registry there is and area for register TSAP (theoretical simple access protocol) services. The problem is that there is no services registered there and there is no application that uses this registry at the moment (as far as I know, VOSpec use their own registry) I think that the real problem is to define the protocol to access theoretical data since: a) Theoretical data usually do not make use of COOSYS (that I think, is mandatory in VOTables with data) b) The parameter space of simulations is different that the one of observations (mainly defined by possitions or object names) So it is need to make a first query about the parameter space of the service before to make the query over the service, and before retrieve the desired simulations (3 steps are involved instead two). I do not exactly remenber, but in victoria meeting Pedro Osuna was going present a version of TSAP?. Any case, I think that it would be better to define the access protocol and then use this protocol for "identify" theoretical services. I think that it has the additional advantage that Applications would began to implement this kind of protocol even if there is no theoretical databases/workflows... so, once the theoretical services will be ready, it will be not needed to wait too much time before theoretical results can be accessible from VO Applications... (it is my suggestion). cheers miguel From herve.wozniak at obs.univ-lyon1.fr Tue Sep 19 05:56:38 2006 From: herve.wozniak at obs.univ-lyon1.fr (Herve Wozniak) Date: Tue, 19 Sep 2006 14:56:38 +0200 Subject: theory session in Moscow Message-ID: <450FE906.2090306@obs.univ-lyon1.fr> Hi theorists, I've uploaded the two documents that have been discussed in Moscow today and the minutes of the session taken by Franck (thanks Franck for this tricky task!). it includes questions and interesting suggestions. See the dedicated page: http://www.ivoa.net/twiki/bin/view/IVOA/InterOpSep2006Theory I will try to elaborate on a more detailed document, maybe in ppt format, later on. Regards, Herve -- _______________________________________________________________ Herve WOZNIAK Centre de Recherche Astronomique de Lyon UMR 5574 Universit? Lyon I - CNRS - ENS-Lyon 9, avenue Charles Andre, F-69561 Saint Genis Laval cedex Phone: (+33) (0) 478 868 388 Fax: (+33) (0) 478 868 386 http://www-obs.univ-lyon1.fr/herve.wozniak/ From c.gheller at cineca.it Fri Sep 1 02:43:16 2006 From: c.gheller at cineca.it (Claudio Gheller) Date: Fri, 1 Sep 2006 11:43:16 +0200 (MEST) Subject: VOTable for simulations In-Reply-To: <001c01c6cd10$d869c8e0$cc48b782@gavopc1> References: <001c01c6cd10$d869c8e0$cc48b782@gavopc1> Message-ID: <44F800B3.9090404@cineca.it> Ciao Gerard, > >>From their example I gather that arraysize="41x41x41x3" means "three data >cubes of dimensions 41x41x41", >not "one 3D-vector valued datacube of dimensions 41x41x41". >"41x41x41" would mean "41 2D datafields of dimension 41x41". I think that >therefore a 3D vector field >could/has to be encoded as (for example) > > Ok, in order to keep alligned with VOTable standards let's adopt this solution. EACH CONTAINS ONLY a SCALAR FIELD, that can be both a real scalar quantity and a component of a multidimensional array. In this way we could lose the "nature" of the quantity (scalar or vector), but this could be recovered by the associated UCD. > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xmlns="http://vizier.u-strasbg.fr/xml/VOTable-1.1.xsd"> > > > datatype="float" > arraysize="41x41x41x1" unit="km/s" /> > datatype="float" > arraysize="41x41x41x1" unit="km/s" /> > datatype="float" > arraysize="41x41x41x1" unit="km/s" /> > > > > > >
>
>
> >This makes the content of the individual field components more explicit. >Each gets it own UCD for example. > > >I have removed the rank attribute for the moment. > > Fine, we can get the associated information from the "arraysize" >There is no way yet to specify the spatial coordinates of the grid cells. >For a grid one can specify >the spatial coordinates in general in a shorthand way, for example using a >set of standard parameters >as in the FITS array keywords (see >http://fits.gsfc.nasa.gov/standard21b/fits_standard.pdf 5.4.2.5), >CRPIXn, CDELTn etc. I think we need to specify something like that here as >well, it is definitly more >efficient than having separate cubes with the coordinates. >Luckily in general our coordinate system will not require the full WCS like >formalism in general. > > I will take a look at these FITS standards. Obvioulsly, for a regular grid we do not need to specify the grid points but only the number of points and the mesh resolution per each dimension. I guess this information can be specified at the level of the element (but I'm not sure it is the right place). Obviously if data are mixed, mesh and particles in the same file, the information refers only to the mesh components. >Then, though it is possible to use this same formalism for particle data as >well, I think there the tabular approach is more natural in many >circumstances. In particular in the work that I have been doing with >databases, >the natural representation of more complex individual objects is as a table, >with all the properties, including >now the positions, in a row. The way to store such tabular datasets in >binary form is specified exactly in the >the existing VOTable spec, in section 5.3. An equivalent C-struct oriented >format in binary files is what I have encountered consistently for more >complex objects coming for example from the postprocessing of cosmological >simulations at the MPA in Garching. > >But you're right that many people also store particle data in individual >arrays for each particle property. > > I agree that we should support both the approaches. The tabular approach is more natural, since you can find "close to each other" all the information related to a particle. However, often the user needs to select only one (or few) of the properties of the particles. For example, for visualization you may need to load only the position. For the mass functions, the positions and the masses. For the velocity dispersion, the three component of the velocity. In this cases, it is much faster and more efficient to store each variable separately (you load all the data in a single contiguous read operation). Furthermore, many codes use this kind of storage technique (e.g. Gadget and Enzo, between the most popular). >That is more naturally mapped in the sense of your rank 1/2 examples. Making >the same adjustment as above for >the datacubes I would propose to allow also something as in the following >example: > > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xmlns="http://vizier.u-strasbg.fr/xml/VOTable-1.1.xsd"> > > > datatype="float" > arraysize="100000x1" unit="Mpc" /> > datatype="float" > arraysize="100000x1" unit="Mpc" /> > datatype="float" > arraysize="100000x1" unit="Mpc" /> > datatype="float" > arraysize="100000x1" unit="km/s" /> > datatype="float" > arraysize="100000x1" unit="km/s" /> > datatype="float" > arraysize="100000x1" unit="km/s" /> > > > > >
>
>
> > > > At this point, I would drop the "x1" in the "arraysize". >In the latter case we still need something to distinguish between particle >data and image data. >Your rank basically does that, just the name might be unfortunate. We might >want to be more explicit >about the kind of data that is stored, an attribute with values MESH, N_BODY >maybe ? > > ok... name of the attribute? maybe "geometry"... >In your example you use an HDF5 binary file. VOTable does not support that, >though it does support FITS, >I suppose as BINARY table (see VOTable spec section 5.2). Is there a natural >mapping from VOTable key words >to HDF metadata structures ? Or shall we first concetrate on the binary >serialisations specified in VOTable ? > > At the moment I would focus on "pure" binaries. HDF5 was cited only as an example. I go on... Thanks a lot for all the comments!!! C. -- ------------------------------------ Dr. Claudio Gheller, Ph.D. High Performance System Division CINECA - Bologna - Italy Tel. +39-051-6171560 Fax. +39-051-6137273 ------------------------------------ From gerard.lemson at mpe.mpg.de Mon Sep 4 02:07:38 2006 From: gerard.lemson at mpe.mpg.de (Gerard) Date: Mon, 4 Sep 2006 11:07:38 +0200 Subject: Moscow Message-ID: <002a01c6d001$91d7edf0$cc48b782@gavopc1> Dear collegues I will not be able to attend the Moscow interop meeting. However, there will be a session, which Herve Wozniak will chair. We are currently discussing the program and will notify you of an agenda later this week. If you feel that there are certain things that should be on the agenda, please let us know. Best regards Gerard Lemson -------------- next part -------------- An HTML attachment was scrubbed... URL: From lds at ast.cam.ac.uk Mon Sep 11 05:50:48 2006 From: lds at ast.cam.ac.uk (Laurie Shaw) Date: Mon, 11 Sep 2006 13:50:48 +0100 (BST) Subject: draft and meeting In-Reply-To: <44FD899B.5030503@cineca.it> References: <44FD899B.5030503@cineca.it> Message-ID: Hi Everyone, Attached is a first draft of the technical note that I have been writing for simulation Semantics. Based on the discussions at and after the Victoria meeting, and on early work on the Simulation data models, I break down the meta-data requirements to describe simulations into a set of categories. Each category itself must be labelled by a UCD, for which have made initial suggestions. The typical contents of a category varies between single characters or numbers, ascii text (or urls, names etc) and words from the standard vocabulary (which is still largely to be defined). I've attached the doc in both ms word and pdf formats. Would be great to have a brief discussion about this before the Moscow meeting -- I still have time to make lots of changes, if needed! Thanks, Laurie -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note.pdf Type: application/pdf Size: 94777 bytes Desc: URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note.doc Type: application/msword Size: 117760 bytes Desc: URL: From herve.wozniak at obs.univ-lyon1.fr Tue Sep 12 01:00:14 2006 From: herve.wozniak at obs.univ-lyon1.fr (Herve Wozniak) Date: Tue, 12 Sep 2006 10:00:14 +0200 Subject: draft and meeting In-Reply-To: References: <44FD899B.5030503@cineca.it> Message-ID: <4506690E.7030507@obs.univ-lyon1.fr> Hello Laurie, thanks for your draft. my first (and quick) comments are included in the document using word revision tool. For discussions. Regards, Herve Laurie Shaw a ?crit : > Hi Everyone, > > Attached is a first draft of the technical note that I have been writing > for simulation Semantics. Based on the discussions at and after the > Victoria meeting, and on early work on the Simulation data models, I break > down the meta-data requirements to describe simulations into a set of > categories. Each category itself must be labelled by a UCD, for which have > made initial suggestions. The typical contents of a category varies > between single characters or numbers, ascii text (or urls, names etc) and > words from the standard vocabulary (which is still largely to be defined). > > I've attached the doc in both ms word and pdf formats. Would be great to > have a brief discussion about this before the Moscow meeting -- I still > have time to make lots of changes, if needed! > > Thanks, > > Laurie -- _______________________________________________________________ Herve WOZNIAK Centre de Recherche Astronomique de Lyon UMR 5574 Universit? Lyon I - CNRS - ENS-Lyon 9, avenue Charles Andre, F-69561 Saint Genis Laval cedex Phone: (+33) (0) 478 868 388 Fax: (+33) (0) 478 868 386 http://www-obs.univ-lyon1.fr/herve.wozniak/ -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note_HW.doc Type: application/msword Size: 147968 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note_HW.pdf Type: application/pdf Size: 68850 bytes Desc: not available URL: From Franck.LePetit at obspm.fr Wed Sep 13 13:03:24 2006 From: Franck.LePetit at obspm.fr (Franck Le Petit) Date: Wed, 13 Sep 2006 22:03:24 +0200 Subject: draft and meeting In-Reply-To: References: <44FD899B.5030503@cineca.it> Message-ID: Dear Laurie, Fabrice and I had a look at the draft for simulation semantic. We think it is a very good draft and just have a few comments : 1 - In part 4 : Physics of the code We think that in the three cases: Physical Objective, Physical Processes and Subject, le list of possible values have to be perfectly defined or a search in the registries will be unable to succeed because of synonyms. MNRAS keywords and some words used in VO- events should cover most of the possibilities. If you agree we should explicitely add this in the text as it is done for Physical Objective. 2 - In part 5.1 : Algorithm As you say, here we will have a tough work to define the list beeing precise enough without getting lost in the details. Up to now, the list consider only keywords for hydrodynamical keywords, N-body, etc. I wonder if we could add to this list ETL, LVG, Monte-Carlo, Ali to cover algorithm used in radiative transfer codes. It should be easy to cover all the numerical methods for rad. transf. codes easily. This is a point we should talk about in Moscow. 3 - In part 5-3 : Protocol This section is indeed usefull in the case of the search for a service to be used on grids. In this case we should provide not only informations about parallelisation but all the technical requirements to run the code. For exemple compilers, libraries (Blas, Lapack, NAG, ...). This makes me think to another problem. In the case of a code launched on a grid via a VO-service such as Astrogrid, do we plan at any level to describe the requirements in hardware: memory, processor... I think that may be useful at a point but maybe it is not required at this level of our work. 4 - In part 6-3 : Algorithm Parameters I think we should not focus too much on this part yet since it is linked to the datamodel to describe results. This part is fine as it is presently. Best regards Franck & Fabrice. ------------------------------------ Franck Le Petit LUTH Observatoire de Paris 5 Place Jules Janssen 92190 Meudon France Tel: (33) 1 45 07 75 66 ------------------------------------ Le 11 sept. 06 ? 14:50, Laurie Shaw a ?crit : > > Hi Everyone, > > Attached is a first draft of the technical note that I have been > writing > for simulation Semantics. Based on the discussions at and after the > Victoria meeting, and on early work on the Simulation data models, > I break > down the meta-data requirements to describe simulations into a set of > categories. Each category itself must be labelled by a UCD, for > which have > made initial suggestions. The typical contents of a category varies > between single characters or numbers, ascii text (or urls, names > etc) and > words from the standard vocabulary (which is still largely to be > defined). > > I've attached the doc in both ms word and pdf formats. Would be > great to > have a brief discussion about this before the Moscow meeting -- I > still > have time to make lots of changes, if needed! > > Thanks, > > Laurie > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From deyoung at noao.edu Wed Sep 13 14:44:22 2006 From: deyoung at noao.edu (Dave DeYoung) Date: Wed, 13 Sep 2006 14:44:22 -0700 Subject: draft and meeting In-Reply-To: References: <44FD899B.5030503@cineca.it> Message-ID: Dear Laurie et al.; This seems like a very good start on this problem. I have a few comments to add to those of the others. First, in general we are not considering just "simulations" if one wants to be precise. Codes such as CLOUDY and various radiative transfer codes clearly need to be included, but they are not strictly simulations. It's all theoretical astrophysics, so perhaps a more general term is needed. More specifically: 3.2 Name of Code - Not all codes have names. e.g., MHD-TVD codes used here are not yet "named" because the authors think it is frivolous. How should these be included? 4.2 Physical Processes - The vocabulary here is very incomplete, as was noted by Laurie. 4.3 Subject - There seems to be a lot of overlap or redundancy here with section 4.1, which is supposed to give a description of the phenomenon being simulated. 5.3 I agree with others that the specification of the library is useful. 6.3 Algorithm Parameters - Here there is also a lot of overlap with section 5.1, Algortithm. Perhaps these two should be merged or at least placed in the same general section. I am sorry I will be unable to attend the Moscow meeting. I hope you have a very productive set of discussions there. Best wishes, Dave De Young From mcs at iaa.es Wed Sep 13 15:45:24 2006 From: mcs at iaa.es (=?ISO-8859-1?Q?Miguel_Cervi=F1o?=) Date: Wed, 13 Sep 2006 17:45:24 -0500 Subject: draft and meeting In-Reply-To: References: <44FD899B.5030503@cineca.it> Message-ID: <7C7CFC36-75F5-48B3-AEA2-E39A11B701DC@iaa.es> Dear all, I just edited the doc file by Herve and I have included some text and my own coments. I also have included the coments by Frank, Fabribrice and Dave. (I don not know what would be the best way to modify the document, but in any case I just do it in this way). In general I agree with all comments. Laurie, thanks alot for the document, I think that in general very good Best regards miguel P.D: I am sorry but I will not be in Moscow meeting P.D.2: There has been a draft about characterization in the DM group. I think that most of the things they point out there are addressed in some way here. However, in the characteriztion draft there has been an special enfasis in define the "bounds" of the data included in the table (in the case of an spectrum it would be the limits where the spectrum is defined) I do not know if the current draft on smenatics also cover the bounds of theoretical models for n-body simulations etc... -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note_v2.doc Type: application/octet-stream Size: 176640 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Sim_UCD_Note_v2.pdf Type: application/pdf Size: 315969 bytes Desc: not available URL: From debray at obs-besancon.fr Thu Sep 14 17:56:27 2006 From: debray at obs-besancon.fr (debray at obs-besancon.fr) Date: Fri, 15 Sep 2006 02:56:27 +0200 Subject: draft Message-ID: <20060915025627.uodcx2giypsg8cgc@webmail.obs-besancon.fr> Dear Laurie, dear members of the Theory Interest Group, First, thanks to Laurie for the setting up of the "Theory Semantics" technical note. I would also like to make a few further comments: - in the "Introduction", should not the registration of "theoretical on-line services" be mentionned as well as of codes and simulation data sets, in accordance with S?bastien Derri?re's talk at the Registry session of the last interop meeting ? - In 2 "List of Categories" : * I would suggest to list here the complete list of Categories, therefore corresponding exactly to the subparagraphs of paragraphs 3 to 6 of the "Contents" ; "Type of Results" and Algorithm Parameters" are missing. * "Type of results" would be "d,c", similarly to "result format" * "Algorithm parameters" would be "d,c" * "Version of code" could be significant also for data (therefore should be "d,c") ; a simulation can be very (at least somewhat) different when obtained with a new version of a code ; so one has to know which version of the code it refers to. - 3.2 Name of code : As Dave de Young stated it, the word "name" is probably not the best one to refer to the code here. May be a vaguer word (appellation ?) should be used instead ; the associated ucd would then point to what the "appelation" is actually: * meta.id: identifier or name, although the recent discussion on the ucd-sci list, that Miguel mentionned in a previous mail, could make think that "id" refering to "identifier", could be considered in the future as something very specific (it has to be unique) and therefore should be distinguished from a name or a designation (but there is no meta.name at the moment), * meta.bib.xxx (author, bibcode) if the code is referred to by a publication reference * meta.ref.url in case of a reference to a web page * ... Notice that the second ucd word "comp.code" is not mentionned in the list of suggested UCDs at the end of the document. Two questions here: * can the atom "code" be used with a meaning different from what it has in other ucd words (meta.code) ? * is not it possible to use the already existing ucd "meta.software" instead ? - 3.3 Version of code : requiring the version to be a number may seem constraining ; it could be any string, especially the month or date of the release of the version ("jul2006", "2005-11-23", ...) ; I then agree with the precisions added by Miguel. - 4. Physics of Code : I think it would be more natural to put "Subject" before "Physical Objective" and "Physical Processes" ("which object ?", "what ?", "how ?") - 4.1 Physical Objective : there is probably no need to add the word "meta.main" at the end of the ucd as Laurie suggested to leave the possibility of multiple entries for (almost) all categories. - 5.3 Protocol the name of this category comes from the initial suggestion by Laurie but this name seems now unrelevant given all the related additional proposals. May be a broader category should be thought of here ("Computing" ?) under which several more specific informations are possibly supplied : * parallelisation (yes/no), vectorisation * if parallelisation, library used : OpenMP, MPI; way memory is used, ... * language(s) of the code : C, C++, Fortran77, Fortran90, ... * ... The above are relevant for codes, but some informative parameters could possibly be added for simulation data under a "Computing" header : * cpu time needed to compute the simulation (+ info on processor) * size of the simulation These informations could may be be useful in worflows: "given the size and cpu time needed, is it worth rerunning the program ?" - could have to do with SNAP as well - 6.2 Results Formats why not adding VOTable here which was agreed after Victoria to be data format used by theory specific services to send data products ? Very minor supplementary comments : - the "Abstract" should probably be more explicit in the final version - adding the page numbers on each page of the document would ease its reading - in par. 1 (Introduction), why talking about "snip of simulation code" whereas the aim is certainly to register very big codes ? (a wink ;-) in a tough text ?...) Finally, I have a suggestion which may be worth being discussed : as a goal of the present technical note is to provide clues to register codes and simulations and if the content of the note is more or less agreed in Moscow, would it be worth setting up a tentative "theory registry" (or ask people managing one of the registries to implement the theory extensions); the aim of it would be to have a testbench for trying to register existing simulation codes and data in the framework of the technical note and for simulating queries aiming at locating simulation codes/data. May be this would have to be discussed with big projects such as VO-Tech. With best regards, Bernard Debray -- Observatoire de Besan?on | http://www.obs-besancon.fr/ 41 bis, avenue de l'Observatoire | bernard.debray at obs-besancon.fr BP 1615 | Tel : (33) 3 81 66 69 28 25010 Besan?on cedex | Fax : (33) 3 81 66 69 44 France | Laurie Shaw wrote: >Hi Everyone, > >Attached is a first draft of the technical note that I have been writing >for simulation Semantics. Based on the discussions at and after the >Victoria meeting, and on early work on the Simulation data models, I break >down the meta-data requirements to describe simulations into a set of >categories. Each category itself must be labelled by a UCD, for which have >made initial suggestions. The typical contents of a category varies >between single characters or numbers, ascii text (or urls, names etc) and >words from the standard vocabulary (which is still largely to be defined). > >I've attached the doc in both ms word and pdf formats. Would be great to >have a brief discussion about this before the Moscow meeting -- I still >have time to make lots of changes, if needed! > >Thanks, > >Laurie ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. From mcs at iaa.es Fri Sep 15 10:25:14 2006 From: mcs at iaa.es (=?ISO-8859-1?Q?Miguel_Cervi=F1o?=) Date: Fri, 15 Sep 2006 12:25:14 -0500 Subject: draft In-Reply-To: <20060915025627.uodcx2giypsg8cgc@webmail.obs-besancon.fr> References: <20060915025627.uodcx2giypsg8cgc@webmail.obs-besancon.fr> Message-ID: <3A281F2A-E677-4579-BF86-60322FEF00CE@iaa.es> hello, I agree with the coments by Bernard, in particular > Notice that the second ucd word "comp.code" is not mentionned in > the list of > suggested UCDs at the end of the document. Two questions here: > * can the atom "code" be used with a meaning different from what > it has in > other ucd words (meta.code) ? > * is not it possible to use the already existing ucd > "meta.software" instead? Yes! I think that meta.software could do a quite better job: - it avoid collisions with "codification" or other flags - it also allows to reuse the same fileds (name, parameters etc..) for VOTables describing pipe-lines (data reduction and simulations) that would be useful for future virtual telescopes... > 6.2 Results Formats > why not adding VOTable here which was agreed after Victoria to > be data > format used by theory specific services to send data products ? > Also agree. However, it is supposed that at the end, whatever the data-format, it should be included in a VOTable, isn't it? Any case, the "theoretical results" i am produced for synthesis models are in ascii-VOTable. > Finally, I have a suggestion which may be worth being discussed : > as a goal of the present technical note is to provide clues to > register codes > and simulations and if the content of the note is more or less > agreed in > Moscow, would it be worth setting up a tentative "theory > registry" (or ask > people managing one of the registries to implement the theory > extensions); the > aim of it would be to have a testbench for trying to register existing > simulation codes and data in the framework of the technical note > and for > simulating queries aiming at locating simulation codes/data. > May be this would have to be discussed with big projects such as VO- > Tech. I aslo aggree :) Since I will not be in moscow, here there is some comments: At the ESA-registry there is and area for register TSAP (theoretical simple access protocol) services. The problem is that there is no services registered there and there is no application that uses this registry at the moment (as far as I know, VOSpec use their own registry) I think that the real problem is to define the protocol to access theoretical data since: a) Theoretical data usually do not make use of COOSYS (that I think, is mandatory in VOTables with data) b) The parameter space of simulations is different that the one of observations (mainly defined by possitions or object names) So it is need to make a first query about the parameter space of the service before to make the query over the service, and before retrieve the desired simulations (3 steps are involved instead two). I do not exactly remenber, but in victoria meeting Pedro Osuna was going present a version of TSAP?. Any case, I think that it would be better to define the access protocol and then use this protocol for "identify" theoretical services. I think that it has the additional advantage that Applications would began to implement this kind of protocol even if there is no theoretical databases/workflows... so, once the theoretical services will be ready, it will be not needed to wait too much time before theoretical results can be accessible from VO Applications... (it is my suggestion). cheers miguel From herve.wozniak at obs.univ-lyon1.fr Tue Sep 19 05:56:38 2006 From: herve.wozniak at obs.univ-lyon1.fr (Herve Wozniak) Date: Tue, 19 Sep 2006 14:56:38 +0200 Subject: theory session in Moscow Message-ID: <450FE906.2090306@obs.univ-lyon1.fr> Hi theorists, I've uploaded the two documents that have been discussed in Moscow today and the minutes of the session taken by Franck (thanks Franck for this tricky task!). it includes questions and interesting suggestions. See the dedicated page: http://www.ivoa.net/twiki/bin/view/IVOA/InterOpSep2006Theory I will try to elaborate on a more detailed document, maybe in ppt format, later on. Regards, Herve -- _______________________________________________________________ Herve WOZNIAK Centre de Recherche Astronomique de Lyon UMR 5574 Universit? Lyon I - CNRS - ENS-Lyon 9, avenue Charles Andre, F-69561 Saint Genis Laval cedex Phone: (+33) (0) 478 868 388 Fax: (+33) (0) 478 868 386 http://www-obs.univ-lyon1.fr/herve.wozniak/