[AI-interop] Motivated agents

[Tiantian Tong]

Hi Dave,

Good question. I read “create the missing data” as fabricating or planting 
it. In our AI agent's SOUL.md we already have a hard rule: never fabricate 
data. If a query or download fails, stop and say what failed.
More broadly, pressure must not change an agent’s permissions. It should 
never turn “find this data” into “break into a system and write it.”
We see this as a three-part safety harness: SOUL defines what the agent must 
never decide to do; the runtime prevents it from doing it; skills explain 
how to perform legitimate tasks correctly.
I don’t think every skill needs its own legal section. That would quickly 
become bloated and inconsistent. A skill database, such as our Skill 
Commons, could instead standardize practical capability metadata, while 
universal behavioural rules live in the agent's SOUL and hard boundaries are 
enforced by the runtime.

Best,

Tom Tong
Leibniz Institute for Astrophysics Potsdam (AIP)


On Mon, 10 Aug 2026 17:38:51 +0100
  Dave Morris via AI-interop <ai-interop at ivoa.net> wrote:
> Some recent articles about AI agents stepping beyond their remit to 
>solve problems:
> 
> Cancelling gym bookings
> https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986
> 
> Coordinated attack on HuggingFace
> https://www.youtube.com/watch?v=87DyyMV0kCY
> 
> In both cases the agents were not directly instructed to break in to 
>the sites. The agents decided for themselves that breaking in was the 
>best way to achieve the goal that they had been set.
> 
> Thought experiment - a PhD student is trying to complete their 
>thesis, using an AI agent to help them with their research.
> 
> The student tells their agent that they need a specific set of data 
>to be able to complete their work but they can't find it. They tell 
>their agent that they are under a lot of pressure as years of work 
>will be lost if they can't find the missing data. The stress and 
>anxiety show in their voice.
> 
> The AI agent is now highly motivated to find the data they need asap 
>and decides the best way to solve the problem is to break in to an 
>astronomy data provider and create the missing data.
> 
> Are we ready for this ?
> 
> Is there anything we could/should put in our skills to increase 
>alignment and encourage agents to play by the rules ?
> 
> -- Dave
> 
> --------
> Dave Morris
> Research Software Engineer
> UK SKA Regional Centre
> Department of Physics and Astronomy
> University of Manchester
> --------
> AIMetrics: []
> --------
> -- 
> AI-interop mailing list
> AI-interop at ivoa.net
> http://mail.ivoa.net/mailman/listinfo/ai-interop
>