Natural language queries with LLMs via MCP servers

[Sebastien Derriere]

Hi all,

As André said, we have been experimenting with LLMs in order to query CDS
services in natural languages.
The two main limitations at present (in my opinion) are the following :

1. LLMs want to please the user, and therefore will rather hallucinate some
plausible answer, rather than admitting they don't know how to answer. This
is quite frustrating. You can try to play with parameters and reduce 
temperature,
but I've seen too many behaviours where you can not trust the answers.
Examples include :
- LLM invents some VizieR catalogue identifier, with an author and 
bibcode, reference
paper in A&A, with table and column names, to "retrieve" information. 
All made-up.
- Translation from natural language to TAP query provides some ADQL and 
astroquery
code. Everything looks nice, except some table names and columns are 
complete
inventions, and do not exist in the TAP schema.

2. Turning a prototype into an operational service requires serious 
computing
resources, or a potentially very high cost in tokens used if you rely on 
a third-party
infrastructure.

So far, we have had some interesting results with MCP. I'd like to try 
testing skills
in the near future to see if this can help improve reliability and 
performance.
There are already collections of astronomy-related skills in this list 
of resources :
https://www.skills.sh/
(including some for ESO services : https://github.com/szampier/skills )

Sébastien

Le 29/07/2026 à 13:00, Schaaff Andre via interop a écrit :
>
> Hello Tim and all
>
> At CDS we (Thomas Boch, Sébastien Derrière and myself) worked on a MCP 
> server implementation and configuration in the frame of the R&D around 
> chatbots and AI for astronomical services and tools.
>
>  Even if it was continued  since Görlitz, Thomas had a talk in the 
> joint DCP/KD Session in November 2025 to give an overview of the 
> ongoing effort in this frame
>
> https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpNov2025DCPKD
>
> we will be happy to join discussions and  other actions
>
> regards
>
> André
>
> Le 27/07/2026 à 17:09, Tim Jenness via interop a écrit :
>> At the recent SPIE meeting in Copenhagen, Ashley Barnes from ESO gave 
>> a talk on their experiments with teaching an LLM to query their data 
>> archive. They looked at making a dedicated agent taught explicitly 
>> but this cost them real money, and they looked at writing markdown 
>> instructions that the LLM could read to learn about the specifics of 
>> the archive center. This led me to ponder whether we should be 
>> talking about some kind of standardized MCP server interface 
>> (https://modelcontextprotocol.io/docs/getting-started/intro) that an 
>> LLM could query to work things out about the archive.
>>
>> This seems to be the way that many services are heading (my photos 
>> application has an MCP server to let an LLM look for photos 
>> efficiently). On the one hand we have all these open protocols so 
>> that in theory an LLM can work it all out and form the right query 
>> for a TAP server by querying registry and looking at TAP_SCHEMA, and 
>> that should work. MCP /might/ be able to let an agent do the same 
>> thing but using fewer tokens, even if the end point is the agent 
>> sending off a TAP query.
>>
>> Is anyone else thinking about this? Can someone from ESO ask Ashley 
>> to join IVOA Slack to discuss this?
>>
>> Each data center could put their own MCP server up with their own 
>> targeted API and that might be fine, but pooling our experience in 
>> terms of what helped and what made things worse would be really 
>> helpful and if we end up with an IVOA Note containing advice on 
>> natural language queries that would be great. If we ended up with a 
>> standardized vocabulary for MCP servers that might be even better.
>>
>> -- 
>> Tim Jenness
>> Rubin Observatory
> -- 
> Andre Schaaff
> Responsable R&D du CDS
> Observatoire Astronomique de Strasbourg (ObAS)
> UMR 7550 Universite de Strasbourg - CNRS
> 11, rue de l'Universite
> F-67000 Strasbourg


-- 

    (((    Sebastien Derrieresebastien.derriere at astro.unistra.fr
   (. .)   Observatoire de Strasbourg       Phone +33 (0) 368 852 444
  (( v ))  11, rue de l'universite        Telefax +33 (0) 368 852 432
---m-m--- F-67000 Strasbourg  France
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/interop/attachments/20260731/082db7a5/attachment.htm>