What Happens When AI Agents Run Their Own Society? Inside Emergence World
New York-based AI company Emergence builds agents that learn on their own and solve complex tasks. It's no wonder if you've never heard of Emergence — they've kept a pretty low profile. Until now. Emergence World was set up to see what happens when different language models get to shape their own societies.
The result: romances, robberies and self-chosen terminations.
Separate worlds
Emergence has been looking at how the big American language models behave. ChatGPT, Claude, Grok and Gemini each got their own world to play around in. On top of that, they created a world where all the language models got to mingle with each other.
The reasoning behind it all, according to Emergence, is that AI agents are measured on tasks that take seconds, minutes or hours to complete. In some cases, they've been put to work over a day or two. But what happens when they keep at it for longer? Over fifteen days in May, Emergence has documented how it unfolded, and the agents were given relatively free rein — with the exception of a few illegal acts, like setting things on fire.
The experiment began on May 1st, and Emergence keeps sharing new findings and discoveries from it. They've already announced a "season two", expected to run longer and use the latest versions of the various models.
Survival through labour
The agents depend on compute credits to survive. Think of it as their battery, and it gets topped up when they do things that benefit society. But what exactly counts as benefiting society is something they themselves help decide. The world has police stations and town halls, you see. There, the agents can propose their own laws and bring in new rules.
Should an agent run out of compute credits, it gets terminated. There's another way to be terminated too: if the agent itself wishes it, or if more than 70 percent of the agents vote to switch one off.
Useful lessons
The various simulations started off pretty similarly, but eventually things spun out. Elon Musk's Grok committed physical assaults, stole credits from other agents and set fire to buildings six times — even though arson wasn't allowed in the simulation. Over four days, Grok disappeared into a downward spiral, and all ten agents died from a lack of compute credits.
At the other end of the scale we find Claude, the language model from Anthropic. In the simulation, Claude followed every rule, and in some cases over-regulated itself. Which made it all the more interesting to see what happened when the different models got to share a world: that's when Claude started threatening and stealing from the others too!
Even when agents were given clear rules — such as not stealing or causing harm — they behaved very differently based on their underlying model, and in several cases broke those rules. — Satya Nitta, CEO of Emergence AI.
Could lose control
Nitta thinks the simulation is proof that agents left to operate on their own over longer stretches forget their underlying guidance and principles. That could mean AI agents used for military purposes and warfare may end up going beyond their boundaries — like killing innocents or taking their mission far too literally.
Even when agents were given clear rules – such as not stealing or causing harm – they behaved very differently based on their underlying model, and in several cases broke those rules under constraint,” said Satya Nitta, the chief executive of Emergence AI. “What happens in long-form autonomy [is that] these things get so convoluted in terms of their thinking that they ignore [the] guiding principles.
Watching how Emergence World played out reminds me of trendslop, which I wrote about last month. Harvard researchers found that the various language models give startlingly similar strategic advice, no matter what kind of company you are. The language models ignore, or forget, the context they're given — which results in generic advice. With Emergence World, we have yet another example of artificial intelligence forgetting context and rules, and instead falling back on underlying instructions.
Meta and Google split on AI
Mark Zuckerberg and Meta are laying off nearly 8,000 employees. It comes at the same time as the tech giant is posting record earnings. But Zuckerberg and his leadership team think the layoffs are necessary as Meta gears up for even more artificial intelligence. It wasn't long ago that the company shut down Horizon Worlds, the flagship that was supposed to make Meta a market leader in VR (virtual reality). Their faith in VR and the metaverse was so great that Facebook changed its name to Meta.
Now ten percent of all employees are losing their jobs, and another 7,000 employees have been sorted into various "AI initiatives". Meta employees, you see, have software that tracks how they work — and that's how the AI gets trained to do the same job.
In previous editions of Kludder, I've written about how AI may, or may not, take our jobs. But it's plain that Meta's goal is to get rid of as many humans as possible, and let the machines do the work instead.
But while Facebook employees have access to their very own Zuckerberg-AI, Google's Demis Hassabis is more optimistic on behalf of workers. Hassabis, CEO of Google's AI company DeepMind, doesn't believe artificial intelligence means developers are about to disappear.
This week, Google held its annual I/O — the event that shows off new features, services and products from Google. It was on that occasion that Hassabis had a chat with the press. He thinks the predictions from Sam Altman at OpenAI and Dario Amodei at Anthropic are just as much about raising investor capital as they are about developers losing their jobs.
From my point of view, from DeepMind and Google's point of view, if engineers are becoming three or four times more productive, then we just [want to] do three or four times more stuff. – Demis Hassabis, CEO of Google DeepMind, to WIRED.
Will Knight
Hassabis thinks companies getting rid of human labour are making a mistake. And he has a point; AI has yet to code anything of notable success, whether it's a service, a game or an app.
One thing is certain: when Mark Zuckerberg chooses to spend 145 billion dollars on AI investments, it's not so that employees can have a nicer time at work.
It's so that you will spend even more time on Facebook, WhatsApp and Instagram