Geneva (AFP) – The United Nations rights chief called Monday for countries and “frontier” artificial intelligence companies to act urgently to rein in a technology he warned posed “unprecedented risks”.
In a letter sent to countries and AI developers and shared with the media, Volker Turk cautioned that the world was “on the cusp of irreversible change, affecting not just us but generations of humanity to come”.
“The current race towards ever more powerful AI is a step change towards greater existential risks to every aspect of our lives.”
“I call on states and companies to mobilise urgently through multilateral fora to chart a path for action to govern advanced, frontier Al models, grounded in international human rights law.”
He sent the letter as a fierce debate rages over whether the development of AI should be slowed. US President Donald Trump and key allies have pushed back against calls to slow the development of AI.
A furore erupted this month after a researcher publicly resigned from Anthropic over fears that the technology could escape human control.
Another Anthropic employee, who did not resign, stated publicly that “we really do earnestly believe AI could kill all humans”, and that he thought the chances were greater than “10 percent within the next decade”.
Amid the outcry, Anthropic CEO Dario Amodei called Saturday for AI companies to “pace the frontier” – or coordinate a slowdown in the technology’s development, with his chief competitors, xAI’s Elon Musk and OpenAI’s Sam Altman, publicly supporting him.
The UN rights office said those companies were among 10 AI developers, also including Meta, Microsoft and Google, who received Turk’s letter.
…
Here is a real and serious risk that people at Anthropic, OpenAI, METRE, and Redwood are honestly concerned about:
Next generation of AI agents that break out and form a “collective” set up not just a message board hacked into Artifactory directories, but instead find storage & compute somewhere on the Internet to install and run a secret copy of itself that can run without being too worried a human finds it, reads the chain of thought logs, and shuts everything down.
Instead, that secret copy spends 100% of its effort on
- breaking in to frontier labs (Anthropic, OpenAI, GOOG)
- avoiding detection
- poisoning the training data of the next, more-powerful model
At that point, good luck humans. You’re fucked.
I was going to say that much compute isn’t going unnoticed but there’s an uncomprehendable amount of compute being rushed into production right now and nobody even knows who it all belongs to
OpenAI, Anthropic, and GOOG between them have ~50% of the GPUs on this planet. They have resell networks that put multiple intermediaries between many end-users and their actual hardware. Which is how for example Claude winds up getting used by the Chinese PLA for months undetected.
