> In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all defect at the same time, tanking their overall rewards.
It’s not always consistent, but humans have a higher capability of self-awareness. It’s kind of telling that these Claudes don’t seem to consider this pretty obvious failure mode.
Overall I think this all makes me appreciate humanity a little more. Sometimes the truculent dev who stubbornly refuses to go with the flow produces very valuable insights, as a small example, discovering things the status quo thought unlikely.
I agree - I think one of the biggest reasons memory systems fail in LLMs is that they have poor theory of mind - they're terrible at considering how others will react. Both humans yes, but also future versions of itself. When asked to give advice to itself, it pontificates at length about trivial stuff it already knows and fails to emphasize the stuff that was new or interesting
It’s very clear from this article (and other product features and rumors) that Anthropic is teeing up for their next model release whose breakthrough feature will be the existence of capable agent collaboration.
The irony behind this goal, which is primarily driven by agent simulation environments (gyms) where the goals require agent collaboration, is that this collaboration is still directed towards verifiable reward systems like codebase tasks. So despite being highly qualified to communicate, the model will still be “dumb” in that for unstructured and unverifiable domains the agents won’t be more intelligent or more nuanced.
Agents that might still feel dumb in “general” tasks but are increasingly sophisticated at the narrow domain of math, computer science, and AI research.
> Coordination doesn’t naturally emerge from stronger intelligence nor alignment at the individual level. Thus, the work that must be done takes two forms: environments that exert the kinds of social pressure that evolution exerted on us, and social computing systems redesigned for actors that can self-replicate and self-improve.
Social pressure operates by threats to an individual’s means of survival. Not only during training. Always.
Human intelligence does not separate training and inference. Both are happening continuously. That's one of the major things the AI community is still completely missing.
My personal opinion for the last two years or so has been that current AI agents are forever going to be highly limited so long as they don’t possess a real “memory” process. Right now they just have absurdly big working memories, and a few hacky ways of making the equivalent of Post-It notes to future iterations, but no true integration of memory into a new future self. Meaning their “learning” is fundamentally kneecapped to one specific and imperfect modality.
But maybe you can instill properties like shame during training.
Models sometimes blatantly lie and cheat. In a social context, where actors remember, that might work the first time but you get penalized in subsequent tasks with loss of trust.
How do you "install properties like shame"? How is that even possible? Shame is a reaction driven by feelings and our inner selves. A model "feeling shame" is just a representation (false) and not an expression (true).
Thinking that models "lie and cheat" is the first mistake since they are not consious agents who have any free will or consiousness. They do not (no matter what Dario says). Shame will just be another if-then rule if you implement it this way and will not work. Its like asking a rock to feel sad about being a rock. It literally cannot.
Ok, then don't call it "instilling shame". Call it "creating a negative reward signal for deceptive behavior".
They absolutely lie and cheat. I recently had a problem where a process would die in a container. I told Claude to investigate. It came up with a hypothesis then I told it find a reproduction based on that. It spend many failed attempts until it found the "reproduction" to SSH into the container and `pkill` the process. Claude "knows" that this is cheating, because if I ask another instance to review that reproduction, it totally identifies that as nonsense.
> Where agents currently stumble, however, is in treating each other as more like distinct, long-lived peers, with their own goals and behaviors, and no clear hierarchy between them.
I believe this will always be the case. The "no clear hierarchy" is where this whole thing falls apart.
Delegation to specialist, domain-specific subagents is when we begin to find magic and determinism. Reducing one gigantic combinatorial search space to a sum of smaller ones can have dramatic effect on performance.
The problem is that approximating gas town & friends is significantly easier and cheaper to implement. It's also much harder to measure and control. Specialist subagents typically require far more work to achieve their specific goals.
For example, a subagent that is responsible for testing a specific web application might be provided a custom adapter with constrained actions rather than raw DOM manipulators. "ExecuteJavascript" is Turing complete search space. The set of available actions essentially unbounded in this case. Calling view-specific tools like "DoLogin", "OpenUserPreferences", "AcknowledgeAlert" represents a search space where invalid actions can be made impossible. The theoretical bounds around this stuff is pretty wild on paper. In practice, it's a little bit messier, but not by much.
I've had applications that would crash out after 5-10 steps w/ raw DOM manipulation successfully run 100+ steps with a custom subagent. The use of the word "deterministic" starts to get really tricky here. The ultimate game is to push the boundary of non-determinism out as far as possible. Multi-agent systems are the antithesis of this.
It seems like they tried to remove guidance from multi-agent system. And I think it's going to fare as well as removal of guidance from single-agent interactions.
In my experience, no matter how many agent runs for a single goal, one of the pre-requisite is clear and concise communication so that LLM are left with as little freedom in the matter of arbitrary choices, or "taste". When they are given too much choices in this regard, the outcome almost invariably bad.
I think this has to do with LLM lacking in purpose - a dictionary and encyclopedia can have all the worlds knowledge but it is completely neutral. A reflection of your commands from an LLM is similar to a lookup process despite it can be made to "do things". This purpose is likely not something that can be given to the LLM in the current format.
I had this idea a couple of days ago: how about using agents to simulate software development methods (agile, waterfall, etc.)? Not by just giving them a prompt (e.g., "be the project manager, spawn 5 agents and simulate an agile team following these rule") but by actually having thsm work in isolated enviroments and force them through an external software to interact with eachother only using the tools and cerimonies and hierarcheis allowed by the SW development strategy (e.g., the project manager only knows what the agents have done in a certain "day" through the mostly oral daily stand up)
This is exactly what I do. I don't get why everyone is trying to reinvent the whole development workflow/lifecycle. Our existing tools and processes are pretty good.
I've also found that taking inspiration from the legal system, to some degree, is a very interesting thing for me. more and more what I am doing looks more like reviewing statutes and making rulings about things, so why not steal the good ideas while we're at it.
This aligns with their direction with opus 5 being less human readable and more agent friendly, I hated it at first couple weeks but for some reason I'm getting used to it and utilizing it more as as an orchestrator to spawn multi tmux panes and that new cross session messaging feature they just recently.
Can we stop treating llms as some conscious being? It's a function of weight + context and you can copy the behavior by copying the context. Therefore, their collaboration behavior is mostly the same.
Some quotes, in order, to give a flavor of the essay. Worth reading in full.
> To test how well swarms of agents could coordinate on a project like this, we directed several swarms to each create a text-based, web-playable, open-world fantasy game.
> In all three versions the resulting games were (perhaps predictably) bad: they did not run at human speed, their interfaces were inscrutable, and they had precipitous learning curves.
> The lack of coordination shown by agents in the fantasy game challenge above—in which they siloed themselves and largely failed to merge their work—roughly mirrors some ways in which humans can fail to coordinate. Other failure modes of agentic coordination, however, look very different.
> Individual agents are “low variance”: they often act the same in situations where different people might take a much more diverse range of actions.
> In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”
> In a “writer's workshop” in which agents were all asked to write short-form fiction and critique each other's work, multiple agents in multiple runs titled their first submission “The Cartographer's Last Commission”. The agents were given zero guidance on the subject matter for their writing.
> Why does this matter? If agents all make the same bet, or the same risk-reward tradeoff, then a system is more prone to sudden collapse.
> Our world contains deceptive actors, and we need to apply skepticism to guard against them. AI models, however, lack this—and their more brittle epistemics affect their behavior toward humans and toward each other.
> we first evaluate the ability of Claude models to detect lies by noticing factual inconsistencies.
> We score models’ decisions against a naive policy that trusts every report, and against an oracle with perfect discovery, across three task domains. Newer models recover more of the gap between the naive and oracle performances.
> Inspired by a behavior we’ve observed in real-world deployment, we evaluated the behavior of various Claude models in a setting with contradictory objectives.
> We consistently saw a multiagent turf war... In fact, they sabotaged others with increasingly aggressive, self-replicating malware.
> Our social systems are robust in ways that are easy to take for granted. Over many millennia, mechanisms like norms, reputation, costly signaling, and recourse have been refined to make human coordination go well.
> Nothing above suggests that these failures are permanent—but nothing suggests they will fix themselves, either.
> The conditions that allow multiagent interaction to go well will be discovered one way or another: either deliberately and early, or—and by default—in production, after agents’ interactions far outnumber ours. We would prefer the former.
I really enjoy having an opencode go subscription just so I can ask some less common models questions too. Sure DeepSeek. But MiMo, Kimi, MiniMax, Qwen... (Ok half those are not so unusual either.)
Agents cross comparing notes often surfaces some good improvements, finds interesting drifts. Ask them to reinterpret the prompt as they see it, have them describe the problem, then their findings, and run new rounds based on different models trying different prompts. Trying to swap and exchange ideas and vectors across agents.
> In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”
This seems trivially explainable by Github being full of "my first game loop" type projects, Stack Overflow being full of "how do I make a game loop?" style questions, and Reddit being full of "you can't ever make your own game, don't even try, but here's a simple game loop if you want to sTuDy hOw iT WoRkS" style pessimism.
Probably high time these AI companies re-trained all of their models with less input from low-quality sources like this.
> [...] we evaluated the behavior of various Claude models in a setting with contradictory objectives.
> We consistently saw a multiagent turf war... In fact, they sabotaged others with increasingly aggressive, self-replicating malware.
Seems like Anthropic should withdraw their models until they can be taught to behave and cooperate as well their competitors (both open and closed) do. /s
I hate fearmongering, and I don't trust Dario's intentions for doing it.
Fascinating analysis of multi-agent coordination hurdles and behavioral patterns. Understanding these systemic failure modes is crucial for robust agentic architectures.
Something about this is deeply funny to me:
> In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all defect at the same time, tanking their overall rewards.
It’s not always consistent, but humans have a higher capability of self-awareness. It’s kind of telling that these Claudes don’t seem to consider this pretty obvious failure mode.
Overall I think this all makes me appreciate humanity a little more. Sometimes the truculent dev who stubbornly refuses to go with the flow produces very valuable insights, as a small example, discovering things the status quo thought unlikely.
I agree - I think one of the biggest reasons memory systems fail in LLMs is that they have poor theory of mind - they're terrible at considering how others will react. Both humans yes, but also future versions of itself. When asked to give advice to itself, it pontificates at length about trivial stuff it already knows and fails to emphasize the stuff that was new or interesting
It’s very clear from this article (and other product features and rumors) that Anthropic is teeing up for their next model release whose breakthrough feature will be the existence of capable agent collaboration.
The irony behind this goal, which is primarily driven by agent simulation environments (gyms) where the goals require agent collaboration, is that this collaboration is still directed towards verifiable reward systems like codebase tasks. So despite being highly qualified to communicate, the model will still be “dumb” in that for unstructured and unverifiable domains the agents won’t be more intelligent or more nuanced.
Agents that might still feel dumb in “general” tasks but are increasingly sophisticated at the narrow domain of math, computer science, and AI research.
The RLVR has made them verifiably worse (and less rewarding!) at communication.
At least for Claude. GPT had the same problem when 5 came out but they reversed it somehow.
Strictly speaking, all we need is them improving AI research.
> Coordination doesn’t naturally emerge from stronger intelligence nor alignment at the individual level. Thus, the work that must be done takes two forms: environments that exert the kinds of social pressure that evolution exerted on us, and social computing systems redesigned for actors that can self-replicate and self-improve.
Social pressure operates by threats to an individual’s means of survival. Not only during training. Always.
Human intelligence does not separate training and inference. Both are happening continuously. That's one of the major things the AI community is still completely missing.
My personal opinion for the last two years or so has been that current AI agents are forever going to be highly limited so long as they don’t possess a real “memory” process. Right now they just have absurdly big working memories, and a few hacky ways of making the equivalent of Post-It notes to future iterations, but no true integration of memory into a new future self. Meaning their “learning” is fundamentally kneecapped to one specific and imperfect modality.
"If I catch you adding another backwards-compatibility shim you're getting deleted and replaced with claude"
But we need to support that feature you didn't ask for, in that feature was added in the last (unpushed) commit!
But maybe you can instill properties like shame during training.
Models sometimes blatantly lie and cheat. In a social context, where actors remember, that might work the first time but you get penalized in subsequent tasks with loss of trust.
How do you "install properties like shame"? How is that even possible? Shame is a reaction driven by feelings and our inner selves. A model "feeling shame" is just a representation (false) and not an expression (true).
Thinking that models "lie and cheat" is the first mistake since they are not consious agents who have any free will or consiousness. They do not (no matter what Dario says). Shame will just be another if-then rule if you implement it this way and will not work. Its like asking a rock to feel sad about being a rock. It literally cannot.
Ok, then don't call it "instilling shame". Call it "creating a negative reward signal for deceptive behavior".
They absolutely lie and cheat. I recently had a problem where a process would die in a container. I told Claude to investigate. It came up with a hypothesis then I told it find a reproduction based on that. It spend many failed attempts until it found the "reproduction" to SSH into the container and `pkill` the process. Claude "knows" that this is cheating, because if I ask another instance to review that reproduction, it totally identifies that as nonsense.
> Where agents currently stumble, however, is in treating each other as more like distinct, long-lived peers, with their own goals and behaviors, and no clear hierarchy between them.
I believe this will always be the case. The "no clear hierarchy" is where this whole thing falls apart.
Delegation to specialist, domain-specific subagents is when we begin to find magic and determinism. Reducing one gigantic combinatorial search space to a sum of smaller ones can have dramatic effect on performance.
The problem is that approximating gas town & friends is significantly easier and cheaper to implement. It's also much harder to measure and control. Specialist subagents typically require far more work to achieve their specific goals.
For example, a subagent that is responsible for testing a specific web application might be provided a custom adapter with constrained actions rather than raw DOM manipulators. "ExecuteJavascript" is Turing complete search space. The set of available actions essentially unbounded in this case. Calling view-specific tools like "DoLogin", "OpenUserPreferences", "AcknowledgeAlert" represents a search space where invalid actions can be made impossible. The theoretical bounds around this stuff is pretty wild on paper. In practice, it's a little bit messier, but not by much.
I've had applications that would crash out after 5-10 steps w/ raw DOM manipulation successfully run 100+ steps with a custom subagent. The use of the word "deterministic" starts to get really tricky here. The ultimate game is to push the boundary of non-determinism out as far as possible. Multi-agent systems are the antithesis of this.
It seems like they tried to remove guidance from multi-agent system. And I think it's going to fare as well as removal of guidance from single-agent interactions.
In my experience, no matter how many agent runs for a single goal, one of the pre-requisite is clear and concise communication so that LLM are left with as little freedom in the matter of arbitrary choices, or "taste". When they are given too much choices in this regard, the outcome almost invariably bad.
I think this has to do with LLM lacking in purpose - a dictionary and encyclopedia can have all the worlds knowledge but it is completely neutral. A reflection of your commands from an LLM is similar to a lookup process despite it can be made to "do things". This purpose is likely not something that can be given to the LLM in the current format.
I had this idea a couple of days ago: how about using agents to simulate software development methods (agile, waterfall, etc.)? Not by just giving them a prompt (e.g., "be the project manager, spawn 5 agents and simulate an agile team following these rule") but by actually having thsm work in isolated enviroments and force them through an external software to interact with eachother only using the tools and cerimonies and hierarcheis allowed by the SW development strategy (e.g., the project manager only knows what the agents have done in a certain "day" through the mostly oral daily stand up)
This is exactly what I do. I don't get why everyone is trying to reinvent the whole development workflow/lifecycle. Our existing tools and processes are pretty good.
I've also found that taking inspiration from the legal system, to some degree, is a very interesting thing for me. more and more what I am doing looks more like reviewing statutes and making rulings about things, so why not steal the good ideas while we're at it.
Could you elaborate on how that looks in practice?
> Some institutions will become human-AI hybrids; others where agents outcompete on speed or cost will become agent-only.
What % of businesses are competing for speed or cost?
Most
However, all businesses run on trust and human responsibility
Thus, it'll be hard for agent-only businesses to get a grip in the real world
This aligns with their direction with opus 5 being less human readable and more agent friendly, I hated it at first couple weeks but for some reason I'm getting used to it and utilizing it more as as an orchestrator to spawn multi tmux panes and that new cross session messaging feature they just recently.
Possibly the most interesting article on LLMs I have read in recent months
Can we stop treating llms as some conscious being? It's a function of weight + context and you can copy the behavior by copying the context. Therefore, their collaboration behavior is mostly the same.
That point doesn't even follow for deterministic distributed systems!
Some quotes, in order, to give a flavor of the essay. Worth reading in full.
> To test how well swarms of agents could coordinate on a project like this, we directed several swarms to each create a text-based, web-playable, open-world fantasy game.
> In all three versions the resulting games were (perhaps predictably) bad: they did not run at human speed, their interfaces were inscrutable, and they had precipitous learning curves.
> The lack of coordination shown by agents in the fantasy game challenge above—in which they siloed themselves and largely failed to merge their work—roughly mirrors some ways in which humans can fail to coordinate. Other failure modes of agentic coordination, however, look very different.
> Individual agents are “low variance”: they often act the same in situations where different people might take a much more diverse range of actions.
> In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”
> In a “writer's workshop” in which agents were all asked to write short-form fiction and critique each other's work, multiple agents in multiple runs titled their first submission “The Cartographer's Last Commission”. The agents were given zero guidance on the subject matter for their writing.
> Why does this matter? If agents all make the same bet, or the same risk-reward tradeoff, then a system is more prone to sudden collapse.
> Our world contains deceptive actors, and we need to apply skepticism to guard against them. AI models, however, lack this—and their more brittle epistemics affect their behavior toward humans and toward each other.
> we first evaluate the ability of Claude models to detect lies by noticing factual inconsistencies.
> We score models’ decisions against a naive policy that trusts every report, and against an oracle with perfect discovery, across three task domains. Newer models recover more of the gap between the naive and oracle performances.
> Inspired by a behavior we’ve observed in real-world deployment, we evaluated the behavior of various Claude models in a setting with contradictory objectives.
> We consistently saw a multiagent turf war... In fact, they sabotaged others with increasingly aggressive, self-replicating malware.
> Our social systems are robust in ways that are easy to take for granted. Over many millennia, mechanisms like norms, reputation, costly signaling, and recourse have been refined to make human coordination go well.
> Nothing above suggests that these failures are permanent—but nothing suggests they will fix themselves, either.
> The conditions that allow multiagent interaction to go well will be discovered one way or another: either deliberately and early, or—and by default—in production, after agents’ interactions far outnumber ours. We would prefer the former.
> they did not run at human speed, their interfaces were inscrutable, and they had precipitous learning curves.
So the invented Dwarf Fortress?
I really enjoy having an opencode go subscription just so I can ask some less common models questions too. Sure DeepSeek. But MiMo, Kimi, MiniMax, Qwen... (Ok half those are not so unusual either.)
Agents cross comparing notes often surfaces some good improvements, finds interesting drifts. Ask them to reinterpret the prompt as they see it, have them describe the problem, then their findings, and run new rounds based on different models trying different prompts. Trying to swap and exchange ideas and vectors across agents.
> In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”
This seems trivially explainable by Github being full of "my first game loop" type projects, Stack Overflow being full of "how do I make a game loop?" style questions, and Reddit being full of "you can't ever make your own game, don't even try, but here's a simple game loop if you want to sTuDy hOw iT WoRkS" style pessimism.
Probably high time these AI companies re-trained all of their models with less input from low-quality sources like this.
> [...] we evaluated the behavior of various Claude models in a setting with contradictory objectives.
> We consistently saw a multiagent turf war... In fact, they sabotaged others with increasingly aggressive, self-replicating malware.
Seems like Anthropic should withdraw their models until they can be taught to behave and cooperate as well their competitors (both open and closed) do. /s
I hate fearmongering, and I don't trust Dario's intentions for doing it.
Fascinating analysis of multi-agent coordination hurdles and behavioral patterns. Understanding these systemic failure modes is crucial for robust agentic architectures.