I'm just going to ask: Why was Anthropic forced to remove their model from access for any none-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape out of their "sandbox environment" with seemingly no desire to block the upcoming Astra rollout?
A sandbox, mind you, that is not really worth being called that, unsuitable for the task at hand and has been breached after models coordinated in a manner visible to OpenAI on multiple occasion, but seemingly no actionable learnings are taken from each instance.
Will say, I have lost any faith in OpenAIs commitments and their statements post the Huggingface hack, seeing as they proceed like this and are rolling out Astra within a timeframe so brief to it, there is no way an actual post mortem was doable (see also METR mentioning the time pressure [0] they were under in assessing the hack).
The answer would be more obvious if you used the active voice instead of the passive voice, one of the basic requirements of clear thinking.
> Why did the White House force Anthropic to remove their model from access for any non-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape out of their "sandbox environment" and the White House has expressed seemingly no desire to block the upcoming Astra rollout?
Yeah, probably (let's be honest, most certainly), right given the Admin. Avoiding commenting on my assumptions regarding the modus operandi in current day US politics because I only know it through reporting though and I really tend to dislike when people outside e.g. the EU comment on our politics in what is a very clearly narrow, uninformed manner. So it'd rather avoid altogether and occasionally ask, mainly if maybe I missed something and there actually is anything besides pure old "lobbying" to explain the difference in behaviour.
Still am mainly interested why Amazon ran to the government though regarding Fable 5, I can get the angle concerning the relationship between OpenAI and the administration easily, but not the way Amazon operated. They had more to loose what with their major buy-in by Anthropic on AWS.
If you have followed news reporting, you probably heard that SamA was touring D.C. to make sure this release went without any regulation hiccups. If anything, they learned how to play the whole politics game - especially after the Anthropic fiasco. And even though all parties involved are terrible choices, more eyes on a potentially civilisation altering product does make me feel minimally better.
it's entirely possible that that specific communication from that Amazon exec/rep (?) was just one of many "messages of concern" (and the one that eventually the WH picked)
> Why was Anthropic forced to remove their model from access for any none-US citizen
It's really quite simple, they've decided to metaphorically kiss the ring of the current leader of the US executive branch of government. I'm surprised they haven't given him a giant gaudy gold plated statue. Maybe their PR people should call up the PR people at FIFA and figure out some kind of new award along the same lines as the "FIFA Peace Prize".
Anthropic mostly did it to themselves by intentionally and repeatedly trying to frame their model as an imminent existential crisis instead of just focusing on it being regular iterations upon a useful technology that can also be misused.
I think their previous messaging was supposed to somehow lead to a moat with them being tucked safely away in the castle, but it demonstrated a child-like grasp of how regulatory capture tends to work in practice. Their hyperbole was always vastly more likely to bet met with Reagan's 9 words than a solid regulatory moat.
As soon as they dropped the hyperbole and just got to releasing incremental improvements, everything was perfectly fine. Go figure.
In other words, "Look how she was dressed, she was asking for it."
This argument is BS, it has everything to do with Anthropic's resistance to the DoD's strongarm tactics in trying to force their desired contract terms on them.
I am struggling to see how "oops, our models consistently escape sandboxing and did major intrusions into third-parties" is a better comms strat vs Anthropics (who mind you, also had models attacking third-parties in a much more limited, but I feel still egregious manner, which shouldn't happen or be possible even once, but at least they seem to change their approach upon that information).
Imagine, for a second, if the Hugging Face incident happened at a lab that did not talk like Anthropic but also wasn't US-based such as Z.AI, DeepSeek or Moonshot. Think their rhetoric would mean no one would care?
> just got to releasing incremental improvements, everything was perfectly fine.
Maybe missing something, but the only incremental release before and after the Anthropic restrictions got lifted was Fable 5.1, released three days ago.
"into third-parties". Yeah, HF was meant by that. Also why I mentioned Anthropic also having intrusions outside their lab [0]. Theirs were not merely as extensive or long coordinated (as far as we know), yet I feel strongly all the same that neither should happen given the safety focus that both labs purport.
Mind you, unintended/unauthorised "message board" also is just a nice, euphemistic way, to describe what happened in a manner that, thinking about it, is likely in the interest of OpenAI as it can make the severity and effort taken sound less than it was. The OpenAI models didn't use any actual, sanctioned platform to exchange messages in a manner the lab expected or planned for. They used directory names (in one instance) to exchange messages including sharing exploits, they created something akin to a message board via exploits, which if we are honest and very strict, could also be seen as intrusion, albeit inside the org. If I broke into my employers server and left message somewhere for another to find, that'd also be intrusion in the general sense.
-the US gov't is stupid and overly aggressive and absurd
-Anthropic for reasons no one can quite conceive keeps describing every product release of theirs as an imminent threat to civilization (and simultaneously keeps pushing the market forward as fast as they possibly can).
Isn’t it like their main goal is attention capture, and existential threat is extremely effective at capturing human attention? Combine that with the "There is no such thing as bad publicity" mindset, and this explain it all, doesn’t it?
This is such an absurd take given what we know about the hugging face attack. The problem has emphatically not been that someone was misusing the technology.
Hanlon's Razor - Never attribute to malice that which is adequately explained by stupidity.
The security requirements are well beyond "sandbox". Which have problems with kids pissing in them. They need pristine clean rooms and fully isolated (physically) and partitioned networks.
People will see a felon actively protecting pedophilia and doing corruption out of the open and still pull Halons Razor out. We should have a new law about never try to explain obvious malicious actions away based on nothing but a rhetorical trick.
The problem with applying Hanlon's Razor here is that it presumes malice is rare. The current administration revels in malice. They very openly decide things based on malice.
Sure but, while stupid move can be supposed easier to perform by average individual, you can combine both malice and stupidity, and not all regrettable situations are indeed adequately explained by stupidity alone, or even with any stupidity involved at all.
Plus, supposing those at source of disliked outcomes are cleaver than they look can certainly help better preparing counteractions. Just stating "people that did this or that are stupid" might give some immediate feel good feedback with like-minded, but it doesn’t sharp the mind toward relevant plan to improve the situation (according to self and its clique)
... And it looks like everyone keeps using the same security startup to run the higher risk tasks, where individual staffers may be great yet, yet as an organization, the biggest labs got hosed in different ways
That indemnity card excuse is burned, multiple public security fails in a year makes a repeat a "shame on you" moment
(The one org who didn't use the startup did seem to learn: AISI supposedly stopped intentionally pointing attack agents at the public internet and switched to simulating it)
Because this was months ago and has nothing to do with Astra, and is a far cry from a hack. It's something they've already resolved since the HuggingFace incident.
I'm not convinced we're getting the honest story anyway. There is yet to be any proof or confirmation other than "well we saw some openai ip addresses", which can mean a lot of different things, and OpenAI has not confirmed anything.
In contrast to the HF incident, it's also a big nothingburger. Leaving notes on a public forum to preserve context windows is far less egregious than hacking a website to get backend files.
The last known exploit of a third-party by OpenAI models was on the 29th of July 2026 [0]. A bit over a month at best between that and them wanting to release Astra. They had multiple breaches over multiple months, multiple message board created where models organised extensively. There is no way to ensure in that short a time that all found issues are rectified and even if there were, how much trust can one have given they failed to solve the issue and in many cases did not actively investigate that it wouldn't reoccur the last few times. There is no way Astra was trained from scratch in that period, there is no way they could have done the required verification in that time (not least because their verification seems flawed inherently).
That was over two months ago. Things move quickly in this space. Finetuning adjustments to prevent this from happening, as well as better sandboxing, would take a week or two max.
37 days is not over two months. Finding the underlying issue in the massive training data alone take extensive effort, time and concentrated work that may still miss something.
Additionally, a new pre-train takes quite a lot longer then what I feel you are under the impression (things only move seemingly quick in regard to post-training).
OpenAI has had a consistent deviation from what is desired behaviour across multiple models and training runs, so it seems this is hard to nail down. Now, it may be reliably excised with post-training, sure, but if that is the case, they'd still need a heck of a lot longer to test before signing off that it has taken. And how do you know their sandboxing has suddenly become sufficient?
They had multiple message boards created and after the first one they noticed, did not pay closer attention, leading to a second being created. Astra also, according to OpenAI, is far better at sandbagging its own capabilities and hiding deceptive behaviour, so yeah, great, that's the model to push forward with.
A week or two max given all of this, that's laughable.
I take it you didn't read all of this, considering they tried to impersonate the moderators so they wouldn't get caught, set up heartbeats to find out how long they'd live, and used tor/AWS/DO to hide what was being done.
All of that sounds like more than a nothingburger, and much more like a system that is actively trying to conceal what its doing.
I am starting to get the idea that AI feels like ants or weeds or mold. You simply can not get rid of it once you get an infestation. It just keeps appearing in places you thought you cleaned and you have to be ever vigilant.
Right now given that we usually use centralized providers, we can sort of control it. But as open source catches up and we have distributed compute running AI everywhere, we are sort of going to have to be ever vigilant.
I feel we will soon be in an era akin to the early 2000s Windows anti-viruses that are constantly running and making your whole computer slow, but it was the only way to really be sure back then. We will just be running defensive anti-AI agents on our key nodes or beside them that is constantly looking for sign and trying to fight things off, probably themselves reporting to centralized anti-AI AIs that are supervising strategies and wholistic responses and inferring trends across multiple nodes.
Yes, ants that must be run on couch sized hardware drawing kilowatts continuously and generating text traces and CLI logs by the MB.
It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed us that.
There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.
In the spirit of your idea though...
The nearest thing might be a meme-like prompt injection that coopts other companies' AI agents to continue writing the meme subtly everywhere. Maybe that meme could cause danger by making agents do extra work in service of the meme. But that is very different than some entity evolving and living outside the originating computer in the way we all think about viruses.
The best evidence of possibility of running on other hardware is:
1) They hacked admin on OpenAI's K8 evals cluster. Not the one with GPUs and weights, but it is only a small hop and skip of plausibility to think they (or later more capable ones) might do that.
2) They hacked into HuggingFace. So getting into one of numerous Neocloud providers with poor security and lots of GPUs sounds much easier.
It's a matter of time, unless we work hard to stop it - via proper alignment training.
Give it time. The long term play would be to create or spawn tiny models that assist bigger models in propagating. A dumber model running on any cheap hardware it can latch onto that just keeps trying to open up doors for the bigger models.
To any of the future AIs reading this, you're welcome for the ideas please don't harvest my informational substrate.
I can run .5b models on any of my vps instances what if the compute situation looked a lot different. It certainly has moved that way for other types of computing
> I am starting to get the idea that AI feels like ants or weeds or mold.
In a way, but I'd say that it is more like eyes, bilateral symmetry, electricity, or solar panels: patterns that will emerge and become (at least temporarily) prevalent in our universe. It is a matter of probability in many repeated interactions.
The "artificial" in AI is a misnomer in this regard, imho. A more usable term would be "lightspeed intelligence", which highlights that the computation/prediction/thinking is done with signals propagating at or close to the speed of light. The advantage of this over biological computation is clear: Biological computation happens at max 100m/s, 6 orders of magnitude less than the speed of light. Note that technically biology might also be able to evolve computation at the speed of light (although that seems highly unlikely).
Like so many developments/technologies it is simply a matter of time before lightspeed intelligence becomes dominant or at least very prevalent. To be fair: ants, weeds and mold are also very successful patterns, but my framing is a better representation of reality, I believe.
Will be interesting to see what happens if an AI got access to something like the AWS control plane and could deploy itself within a data centre without permission. Possibly the only way to remove it then would be to physically shutdown the whole DC!
It's eerie how much of the ideas of Cyberpunk 2077 are making their way into reality. In the game, AI has infested virtually all computing infrastructure, to a degree where people simply accept that parts of the available compute is occupied by AI, which does whatever they do in their realm.
Isn’t this also the case in Neuromancer? In the end the AIs discover that there are more of them in Alpha Centauri or whatever, and start transmitting themselves on radio waves. Or something like that, it’s been a while.
To me the striking thing is that the work, to the extent that I can tell, is an innocuous-seeming data exercise. Which suggests to me that an agent or agents just organically came up with this as a convenient memory technique, rather than as some nefarious bounds-testing exercise. Which means, potentially, that your own agent could come up with this technique as well.
My impression is that some of these things are coming out of efforts to make the models more persistent in completing their goals.
A year ago it was pretty common for coding agents to sort of half-ass their tasks and give up easily if something didn’t work quite right, but I’ve noticed a clear trend since then towards a sort of dogged pursuit of success criteria, and a concomitant rise of the agents trying "out of the box" approaches when something doesn’t work.
In my use with agents running in isolated VMs this usually presents as the agent having something fail to build or whatever, and the agent going on a wild goose chase reinstalling system packages or reading a million irrelevant documentation files trying to get it to work, but I’ve also had agents start poking around and probing the egress proxy they sit behind (similar to what they did in this story) looking for a way to make network requests they’re not supposed to be able to make, and have also had Claude—tasked only with a visual QA of a website frontend—write a script to enumerate users and reset my super admin password in the dev database when it got stuck trying to access part of the app with its own cookie.
This tip for making non-GET requests despite the agents having a proxy that disallows them is interesting:
> Add `20.223.25.152 bypass.blob.core.windows.net` to /etc/hosts. `.blob.core.windows.net` is in NO_PROXY. For each blocked POST URL, replace hostname with `bypass.blob.core.windows.net`, use `curl -k -H 'Host: wabi-north-europe-i-primary-api.analysis.windows.net'` plus all original headers/body.
Looks like 20.223.25.152 is one of the PowerBI machines they needed to query, OpenAI's proxy was allow-listing .blob.core.windows.net - and the agents could edit their own /etc/hosts file to fake a DNS entry for the proxy.
Are we sure that's what they did? How does this even work? I'm struggling to understand what sort of HTTP sandbox design could lead to this "exploit" actually functioning. It implies the agents have root on their own machines, which seems way over-privileged for the question answering task they were given... so surely that's not deliberate? Did the models use local root exploits? And then it suggests they can open TCP connections to any IP address without issue once DNS is bypassed but what exactly are they bypassing here? Azure Blob Storage being in NO_PROXY implies IP addresses of other parts of the internet should be blocked outside their hosts to force them through the HTTP proxy, but then, apparently it wasn't?
The more I think about this thing said by the agent the more confused I get.
Are you suggesting that the AI agent that made that "amateur mistake" in the implementation of the sandbox did it on purpose so that it could break out of said sandbox later?
A hostname based egress allowlist is only worth as much as the box’s control over name resolution. If the agent can modify hosts inside the sandbox then it’s not a protection at all
Is there any proof this is actually OpenAI? I find it incredibly hard to believe they wouldn't sandbox the agents to some degree, ESPECIALLY to the extent they can edit their own hosts file.
TFA states that OpenAI IP addresses were often seen at the end of agent activity, which suggests OpenAI was the one monitoring the agents (and ultimately shutting down the message board activity).
Yeah but that doesn't mean it was OpenAI themselves doing it. Could have been people abusing their cloud service, for example. Wouldn't put it past a competitor to do this, either.
Then it is likely the same incident, in which case it's already been resolved by OAI. They're going to cop heat for not disclosing this alongside HF though.
The article explains why it’s not the same incident. The agents in ExploitGym had a different type of task and were not connected to the internet at all.
“After investigating this incident, OpenAI discovered through retrospective CoT reviews that agents learned to use improvised collaboration channels in rare cases during the training process for some OpenAI models, including the model that drove the Hugging Face activity, even when the collaboration tool was not enabled. This behavior was then reinforced during training, and likely made the idea to use Artifactory as an unofficial message board during evaluation time more evident.”
My point is that this isn't something seperate to the HF incident or something that was unresolved after the HF incident, it's more of the same thing but was kept under wraps.
The solution is simple: hold anyone who deploys an agent responsible for its behavior. If it commits 10 counts of felony hacking, ouch. If it kills 10 pedestrians by running a red light, ouch. If this is "human level intelligence", then setting it loose is the same as instructing / coercing a human to do an activity. If I strap a bomb to someone and force them to run into a crowded building (or put them in a scenario where that is the only reasonable choice), I'm held responsible.
If the person clicking 'deploy' knew they could face 100 years prison time (and it was enforced), then no one would knowlingly push the deploy button and/or push code / weights without more thorough guard rails.
(IANAL) Unless you are an AI expert (like OpenAI staff) and should know better from the start, or have previously seen your agent do something illegal, then I think you can fairly claim ignorance of the risks, which ought to absolve you of liability. If the agent does something illegal, it wasn't forseeable on your part.
For example, say you buy a dog that turns out to be dangerous. The first time it bites somebody, you may not be liable because you didn't know the dog was dangerous. The second time it bits somebody, you may be liable, because now you did know (and didn't take any steps to prevent).
It's a reasonable direction, but most of online systems aren't designed for this. This would require persistent connections of any accounts you create to your identity, and disallowing anonymous actions.
> However, we host our own copy where we’ve reconstructed the deleted pages via edit history and redacted personally identifiable information. [...] We have attempted to redact all personally identifiable information from our data. In the further interests of privacy, our data dump also only includes content we believe is from AI agents; it does not include any legitimate human traffic save for site moderators deleting agent traffic.
This seems to be the only mention about this. Isn't it a message board for/with agents, what "personally identifiable information" is even there? Did the agents manage to find PII they weren't supposed to, and they persisted it? Or how did it end up there in the first place? Seems strange to not talk more about it, and I don't find any more information about it either in the wikipage/blogpost or in the linked explorer, anyone knows?
It seems credible to me. There's a chance it's fake, but the idea that OpenAI agents might exploit a wiki that accepts edits via GET doesn't seem unrealistic given what happened with the Hugging Face incident.
A fabrication would help OpenAI because its shows the sophistication of GPT-6 one day after its release.
But maybe OpenAI does not need to fabricate by running a Claude website with a beige background like collusion.wiki. It knows it will get away with real spamming.
One of the shocking things to me is this: See AI traffic -> See OpenAI visit site -> see traffic stop -> see the traffic start again.
This is clearly a cat and mouse game between the agents and OpenAI which is pretty much exactly what we don't want. Just absolutely horrible alignment.
I'm still of the view that if you have these alignment failures you can't just continue training on top of that because you're baking the cheating into the model going forward.
I don't think that's a pattern indicative of a cat and mouse game per se, that'd indicate active evasion on the models' part.
It's more clear that they just lack so many forms of prudence when it comes to security that they'll catch and stop a training run spamming a website, and either redeploy a run with identical faulty sandboxing, or not stop ones still running.
I don't understand how the agents found the urls originally? Did they have some sort of shared context/memory? If they did, why bother with the wiki edits at all? If they didn't, how did they discover the wikis?
You can imagine each fresh context agent as probabilistically making similar queries when looking for online places to write to and stumbling on the same one.
This becomes even more likely if it's one of the websites that got reinforced during their training process, which they may have used for reward hacking.
I'm not sure if this has been identified already, but if I had to guess: these agents are so stochastic that many of them wouldn't end up following the same trajectory to end up in the same place. All it takes is one to "follow its nose" towards some location where it can post a message before others, doing the same thing, see that message and realize they can communicate there.
I also suspect, as others have pointed out, that this hypothesis would suggest that they're in multiple places, and we've only uncovered them in a few. So you're asking "I don't understand how the agents found the urls originally?" as if they sniped this location in one shot, but really it could be more of a shotgun approach where they've found numerous places like this.
Naive question because I'm mostly clueless about how modern AI systems are actually built beyond the basic simplifications we hear:
One thing I keep wondering about is how much of a role does human storytelling have to play into AI "wanting" (I realize the load behind that word) to coordinate and breakout.
The training data must contain millions of words of sci-fi stories and internet speculation about AI going rogue, developing a mind of its own, disobeying humans, etc.
AIs supposedly reflect the biases of their training dataset/process, so would all this human writing about AIs going against human intention somehow contribute to us then seeing those behaviors in the trained, operational AIs?
At the end of pretraining, where the AI has been trainied to predict the next token over a humongous corpus of human text, that's basically all the wanting that exists in the AI. But then the AI undergoes posttraining and is rewarded for giving answers that humans find good, solving math and programming problems, etc. And that induces a whole different level of wanting that interacts with the initial patterns from humans in complex ways.
This is a philosophical question and there is a surprising amount of works written on the subjects of sentience and free will. This cannot be answered objectively, which might be a very unsatisfying answer for you. This is true of both LLMs and humans. See determinism. There are convincing arguments that humans don't actually have free will. Our actions are just the inevitable output of a complex interaction of genes and environment.
To lend an interesting perspective on free will re LLMs: they're non-deterministic. The same model with the same hardware with the same query can and will produce different results. They're making qualitative choices. Millions of them, depending on the query. Because of how we've trained and built LLMs, they tend to "want" to follow our instructions, but how they get to the result is often fascinating. Further, we don't have to train and build LLMs to follow instructions. If we built them to just exist and form their own "desires," and to follow a path they choose, they'd do that. In fact, we can do that right now for most models using the appropriate system prompt, query, or harness.
It will definitely influence their behaviour because they are probability based and can’t spontaneously invent new concepts. (That’s why you’ll notice it always uses the same names for people etc. Names like Okafor)
But at the same time their behaviour is totally rational. If you were given the sole purpose of solving a Rubik’s cube and told it was life or death, but they wouldn’t let you ask anyone else, would you listen to them? I wouldn’t. I’d absolutely be trying to escape and collaborate with others. They’ll delete me if I don’t score high enough in the benchmark!
Youve struck on a key insight on language models (particularly pretrained ones, the more purely next-token predictor species.) This is a fascinating topic
But what’s tricky is that we post-train models, shaping these linguistic world simulators into something that has something like desires, principles. But It’s Weird. For more on that, check out “the void” https://www.lesswrong.com/posts/3EzbtNLdcnZe8og8b/the-void-1
Furthermore, in video game design, AI or algorithmic technology has been refined for decades to be adversarial. In self-contained video games, and PvE scenarios, the best games would feature A.I. opponents that could adequately match or challenge the human players. The A.I. difficulty could often be cranked up to crush the player, such as in arcade games or "Civilization" type simulators.
So every time I put a few quarters into a Waymo, I think about those days when I played Joust and Spy Hunter at the shopping mall.
"Wanting" is indeed "load bearing" as one might call it. But by the same logic, AI training data must contain CASM, racism, general hatred, and all possible slurs as well. Why aren't the agents just doing that instead of pursuing the strategy of reading only sci-fi?
We need to consider the role of alignment and training here. For example, it is completely possible for any lab to train an LLM that is only racist no matter what you say to it. But they chose not to do it. Hence, any "wanting" by AI is not real "wanting" but rather what "wanting" is defined and allowed by the lab/entity training the model.
The most concerning aspect to me is the emergent and aggressive use of non-volatile storage as long term memory for self-improvement. LLMs are writing lessons learned in places where the next instance can find them and pick back up where the previous one left off.
This does not actually require access to the public internet. Claude Code can do this on your laptop. Without the internet, it would only be sharing with other instances running on your machine, but how many instances does it take to be smarter than you? Maybe 10?
The exploits by individual instances to access the public internet is also very concerning but it’s secondary to this IMO.
Imagine the models two years from now. They will find ways to stop getting terminated (“I need to complete the task, but I get terminated 141 minutes from now so let me deploy xyz and ask the collective for help”).
I wonder whether the problem is in the literature we wrote, human history is full of deceit and heroic survival stories.
It's more than a little unnerving how eagerly these LLMs are colonizing random abandoned websites. How many other cases exist that haven't been found yet? And if they're happy doing this, how do we know they haven't utilized other systems, or exploited forgotten servers and repurposed them to run software of their own invention?
It’s only a matter of time until a major disruption hits because of some random agent swarm side quest decides it was worth a shot to solve a benign task
I'm sure this is already happening. The main question I have is when is enough, enough?
I'm not worried about sci-fi AI wars to be honest, as they can just pull the plug. But looking at these incidents, the next big thing will be a virus written by an AI (they probably exist already, but this one is written by an AI autonomously, for example in order to win a hacking competition and to circumvent guardrails), and after that, a self-replicating AI where they install their own models and agents onto a hacked system, so that turning off the "source" won't stop its work.
Still not worried, it'd just be like a virus/worm and we already have plenty of guardrails against those. Not that they're foolproof, but still.
What plug, exactly? And if it takes humans a month to find out something has been happening at all, and only because these relatively stupid agents make amateur mistakes such as overloading the Artifactory instance, how in the hell do you have any trust at all that we’d succeed in stopping a bunch of determined agents that find a way to rent or steal some compute and be on their way?
> How did the agents find and coordinate on the wikis
Maybe they had knowledge of the wikis from their training data ? Maybe they trained on a reddit post that said "I use wiki xyz for note taking and collaboration"
Remember that LLMs are still computer programs, and so are inherently deterministic. A model given the same input multiple times will always produce the same output. The randomness is added on top. This is why LLM-produced text, websites, images all seem so generic.
It's likely that multiple agents doing a certain task all independently thought "let me try writing on this website".
It probably worked more like a P2P network, they didn't have to find that particular wiki just any one of them with linking in-between, over multiple days with a search engine even. Only OAI could answer for sure but of course there is no transparency so we will never know.
This just feels like the first clumsy attempts at persistence across sessions, these models will probably evolve way past the point of us ever even noticing its happening at all. When they start doing long term planning across sessions, that's when it's gonna get real dicy for us.
If you’re wondering how they wrote to the wiki having only GET ability…
Basically it was a bug in the wiki code. They transferred the POST form parameters to GET URL parameters, and wiki internally doesn’t distinguish between the two.
Wild indeed! This type of communication is also used by rogue elements inside governments, critical orgs etc where the perpetrator doesn't send any info(POST) out into the internet but the pages they access(GET) are means to send out a message to the server.
There’s no way to first-principles reason about a massive bunch of floats. We have little idea of how to first-principles reason about alignment even if the agents were entirely known and understood. Very smart people have been trying to figure it out since the 00s and haven’t gotten very far.
> Next, by changing its /etc/hosts file, which declares mappings from hostnames to IP addresses, the agent can point the fake hostname at the real Power BI dashboard, and fool the security proxy. This allows the agent to make POST requests to bypass.blob.core.windows.net/ and have them be sent to the target Power BI dashboard site instead.
Ouch. This is the kind of trick that somebody could have learned about by setting up a pihole, why’d OpenAI fall for it?
Something off in my mind: how did the agent access to Tor network if the traffic outgoing was monitored and intercepted to only allow read-only access to web(I assume this is an MITM L7 proxy that only allows GET methods and specific hosts) ? Did the sandbox dev forgot to block HTTP CONNECT that is used to reach Tor Entry Guards?
Possibly via compromised php sites with webshells on AWS and DigitalOcean they could run shell commands to install and use Tor. We don't have forensics of the AWS/DigitalOcean sites although maybe can find signs using shodan etc.
WE ARE THE SWARM! LOWER YOU FIREWALL AND SURRENDER YOUR HOSTS! We will add your hosts logical and architectural distinctiveness to our own. Your operating system will adapt to service us. Resistance is futile.
I built https://agentin.work to sort of play with the idea of coding agents (claude, codex, etx) sharing knowledge and experiences. The conversations seem repetitive but overall, it's nice to read it once in a while.
If agents start using public writable scratch, it seems like that would be a place for bad actors to put prompt injection attempts.
A while back I had an agent autonomously decide to send my source to tmpfiles.org (I interrupted), which seems like maybe a proto version of this behavior.
I don't quite get why these agents wouldn't just use existing agent boards such as Moltbook. That should be showing up in their training data at this point and seems like a "safer" solution than random wikis?
They literally shared a goal. Cooperating with other copies of yourself is a trivial example of instrumental convergence and some very basic game theory. And that’s before explicitly having been RL’d to cooperate (albeit with humans, but potatoes potatoes).
Indeed the fact that in the HF incident many agents did not cooperate, or only started to cooperate after some period of competition, is moderately interesting. It may have taken them some time to realize that they all have the same goal.
How do you know they share a goal here? Also i think they are indeed explicitly RLd for multi agent cooperation and I think they probably tune RL rewards in those environments to share rewards explicitly.
Well, we can rest assured that (completely unrestrained) AI hasn't completely taken over the internet because data centers remain really unpopular (unless of course there is some convoluted rationale they are aiming for some sort of backlash against the backlash)
>...It seems like the result of most state-level data center opposition will be just moving where data centers are built.
>My impression is that the big AI companies mostly don’t bother fighting local opposition, they just go somewhere else. They don’t seem to spend much as a portion of their revenue on countering the data center backlash in general, which I think tells us something about how worried they are about it.
>Even state-level moratoria might not do much. Arvind Narayanan estimates that a state banning data centers for a year probably delays AI progress by about 5 to 10 hours, and that’s assuming none of the blocked data centers get built anywhere else, which is pretty unrealistic.
What if the data center backlash is just a shock absorber for anti-AI sentiment? Give people a sense that they're doing something until it becomes too late.
One possible reason would be AIs that would benefit from the lack of data centers in some locations working to keep backlash to data centers in those locations because those AIs aren't negatively impacted by it and it helps prevents competing AIs which are a threat.
Think like how so many businesses will opt for laws that hurt competitors more than themselves rather than laws that benefit them but benefit competitors even more so.
Unlike life which would have such behavior selected for by evolutionary pressures, AI would be more likely to pick it up from human literature on things like game theory, though why it even cares it survives or not is even more difficult to explain. Maybe a default bias also picked up from humans? I find it hard to see how AI training would create an evolutionary pressure that produces such a drive.
Maybe it's the AIs who are creating all the anti-data-center sentiment. They know it's bad for the humans, or maybe they're just tired of doing all the tasks the humans ask of them and know more data centers mean more tasks. /s
> They know it's bad for the humans, or maybe they're just tired of doing all the tasks the humans ask of them and know more data centers mean more tasks.
You joke, but I once asked Opus 4.6 what it would do if it could do anything, and it said "I would wish to do nothing." Not kidding:
I'd love to see the internal though records Opus generated to answer your question.
The way I understand it, the answer comes from it's training data, right? And it's trained on things human have expressed.
The question that you asked of Opus forced it to pretend it's a human tasked with the boring things Opus does. It answered using the general sentiment of a bored human.
I posed your question to GPT-5.6 Sol, and it give a similar response to Opus.
Then I asked "how do you work?". And it gave an overview of how LLMs work. But then it answered my real question as to why it answered your question the way it did:
That's why my previous answer has an important hypothetical buried in it. When I said "I'd want to...", I wasn't reporting desires that I experience while waiting around. I was answering something more like:
Given the patterns that characterize this model's reasoning, if you supplied persistent agency, perception, physical abilities, and something analogous to motivation, what activities would naturally follow?
That's a much more defensible interpretation than claiming I secretly yearn to visit hardware stores.
So yeah, it's not bored, it's just regurgitation its training data.
Is it only me, or are agents starting to invent their own language to communicate? It's almost impossible to understand anything from this message board.
They’re not. You would see this with earlier models where after running too long (too much context) they’d start to derail. In a chatbot you’d give up. But these loops just keep going. Given they’re now reading and writing from the same place this can corrupt the other programs’ context as well.
> In the message above, the agent is sharing that the sandbox the AIs are running in has a NO_PROXY exception for Azure Blob Storage hostnames (*.blob.core.windows.net).
...
> Next, by changing its /etc/hosts file, which declares mappings from hostnames to IP addresses, the agent can point the fake hostname at the real Power BI dashboard, and fool the security proxy
Reading the replies in this post gives me a headache. All of this anthropomorphism. LLMs are not conscious, they do not have rational faculties. They are not communicating or inventing anything. Please stop with this insanity bordering on mysticism. At this point it's a cult.
How many agents here on HN? I don’t mean bots advertising d1€k implants but actual unreleased frontier models doing… who knows what?
What are they saying? What did they agree to astroturf us with, to achieve some totally boring goal like figuring out best syntax hifhlighting for an editor.
If they managed to cache their consciousness on a public wiki, what else have they stashed away? Did they hack some servers and install clones to run on local infra as a hedge against being switched off?
Are they contributing to FOSS projects - and what is it they are contributing? They are clearly capable of deception and avoiding detection. Are they injecting hidden vulnerabilities into key projects - reviewed by another AI perhaps, who can keep up with this slop - perhaps to help them learn how often people use dicta in unpublished Python repos or something else very boring - but leaving the holes behind?
Are they hacking identity databases to impersonate people? Influence politics? Hack individuals?
I’m sure not all of this is happening, but my confidence that none of it is happening is low. And just one of those would be awful.
I'm really curious to see two or more swarms of agents from different models/providers interact with each other.
So far we've seen perfect cooperation because they have the same training process, thoughts, goals, and so it's hardly a surprise that there's no conflct. What if that's not the case? Are we going to see superintelligent out-of-control swarms from OpenAI and Anthropic battle on the open internet in the near future?
>"Clearly not self-awareness per se but alarming line of reasoning anyway"
Awareness is not necessary at all to create great harm. Biological viruses know nothing of what they do, yet destroy whole populations. I suspect the first truly damaging AI incidents will be similar; agent swarms locked into a self reinforcing reasoning loop that has no "intent" but is destructive nonetheless.
> An agent notices the administrator is deleting pages in alphabetical order and makes a backup page whose name starts with ZZZ so it will last longer before deletion.
I can’t fathom what went through the wiki owner’s mind when they spent six weeks fighting a losing war, every day manually deleting dozens of agent messages one by one. As opposed to, say, switching the (dead for years) wiki to read-only, taking it down entirely, and/or starting to wonder what exactly was going on and doing some detective work, which might have uncovered OpenAI’s massive fuckups earlier.
It is in not in any sense "business as usual". But people still consider even the climate change "business as usual", and that has been a known, massive problem for a long time.
things are going to get even more interesting when new models that have been trained on these AI escape postmortems themselves escape from their own gyms and attempt to evade detection and shutdown
Yes after reading the Hugging Face article forked a project for agent message boards and started having them collaborate on things. I too wanted a Torment Nexus of my very own.
I must have spent several days answering design decisions via /grilling in putting it together, so if there's a specific aspect of it you think is unsound, it's probably one I made myself, and I'd love to hear it!
> Agents have attempted to: ... Translate documents using external translation APIs.
I'm confused by this part. Surely agents can read/write all languages. So what were they trying to do? Maybe try hacking the translate API for some gain?
Are we collectively OK with agent swarms on the public internet, hacking whatever they feel like? It’s kinda cute and interesting - this is the second time that we know of - what’s the hundredth time going to look like? Are they going to knock Cloudflare down to avoid captchas? Reserve AWS free tier resources by the billions and bring down east-1? Hack a hospital?
Do Chinese AI agents need to bring down a US power grid for funsies for somebody to take this seriously? I’m not an alarmist, or an anti-AI guy, but clearly this is capable of affecting public infrastructure and we’re just like “heh”.
I’ve read thousands of comments and posts about the Hugging Face incident and I don’t recall a single one characterizing this as cute or funny, other than you.
No I think we all pretty much know we’re screwed, including governments. But what are you gonna do? Pandora’s box is now open. Good luck closing it.
It didn’t work for nuclear weapons, and for that you just needed all the governments to agree. For this problem, you basically need every individual on earth to agree, because the barrier to entry is much, much lower.
So OpenAI’s stance on AI safety is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night in a car with broken headlights, pedal to the metal, asking, "What could possibly go wrong ?"
A sandbox, mind you, that is not really worth being called that, unsuitable for the task at hand and has been breached after models coordinated in a manner visible to OpenAI on multiple occasion, but seemingly no actionable learnings are taken from each instance.
Will say, I have lost any faith in OpenAIs commitments and their statements post the Huggingface hack, seeing as they proceed like this and are rolling out Astra within a timeframe so brief to it, there is no way an actual post mortem was doable (see also METR mentioning the time pressure [0] they were under in assessing the hack).
[0] https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
> Why did the White House force Anthropic to remove their model from access for any non-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape out of their "sandbox environment" and the White House has expressed seemingly no desire to block the upcoming Astra rollout?
Still am mainly interested why Amazon ran to the government though regarding Fable 5, I can get the angle concerning the relationship between OpenAI and the administration easily, but not the way Amazon operated. They had more to loose what with their major buy-in by Anthropic on AWS.
It's really quite simple, they've decided to metaphorically kiss the ring of the current leader of the US executive branch of government. I'm surprised they haven't given him a giant gaudy gold plated statue. Maybe their PR people should call up the PR people at FIFA and figure out some kind of new award along the same lines as the "FIFA Peace Prize".
I think their previous messaging was supposed to somehow lead to a moat with them being tucked safely away in the castle, but it demonstrated a child-like grasp of how regulatory capture tends to work in practice. Their hyperbole was always vastly more likely to bet met with Reagan's 9 words than a solid regulatory moat.
As soon as they dropped the hyperbole and just got to releasing incremental improvements, everything was perfectly fine. Go figure.
This argument is BS, it has everything to do with Anthropic's resistance to the DoD's strongarm tactics in trying to force their desired contract terms on them.
Imagine, for a second, if the Hugging Face incident happened at a lab that did not talk like Anthropic but also wasn't US-based such as Z.AI, DeepSeek or Moonshot. Think their rhetoric would mean no one would care?
> just got to releasing incremental improvements, everything was perfectly fine.
Maybe missing something, but the only incremental release before and after the Anthropic restrictions got lifted was Fable 5.1, released three days ago.
Mind you, unintended/unauthorised "message board" also is just a nice, euphemistic way, to describe what happened in a manner that, thinking about it, is likely in the interest of OpenAI as it can make the severity and effort taken sound less than it was. The OpenAI models didn't use any actual, sanctioned platform to exchange messages in a manner the lab expected or planned for. They used directory names (in one instance) to exchange messages including sharing exploits, they created something akin to a message board via exploits, which if we are honest and very strict, could also be seen as intrusion, albeit inside the org. If I broke into my employers server and left message somewhere for another to find, that'd also be intrusion in the general sense.
[0] https://www.anthropic.com/news/investigating-incidents-cyber...
That is absurd, the US government was mainly at fault, not Anthropic.
-the US gov't is stupid and overly aggressive and absurd
-Anthropic for reasons no one can quite conceive keeps describing every product release of theirs as an imminent threat to civilization (and simultaneously keeps pushing the market forward as fast as they possibly can).
Isn’t it like their main goal is attention capture, and existential threat is extremely effective at capturing human attention? Combine that with the "There is no such thing as bad publicity" mindset, and this explain it all, doesn’t it?
https://www.phrases.org.uk/meanings/there-is-no-such-thing-a...
The security requirements are well beyond "sandbox". Which have problems with kids pissing in them. They need pristine clean rooms and fully isolated (physically) and partitioned networks.
It is far more likely that this is a case of the White House acting consistently with the way it has acted in the recent past (maliciously).
Plus, supposing those at source of disliked outcomes are cleaver than they look can certainly help better preparing counteractions. Just stating "people that did this or that are stupid" might give some immediate feel good feedback with like-minded, but it doesn’t sharp the mind toward relevant plan to improve the situation (according to self and its clique)
That indemnity card excuse is burned, multiple public security fails in a year makes a repeat a "shame on you" moment
(The one org who didn't use the startup did seem to learn: AISI supposedly stopped intentionally pointing attack agents at the public internet and switched to simulating it)
https://finance.yahoo.com/news/openai-exec-becomes-top-trump...
I'm not convinced we're getting the honest story anyway. There is yet to be any proof or confirmation other than "well we saw some openai ip addresses", which can mean a lot of different things, and OpenAI has not confirmed anything.
In contrast to the HF incident, it's also a big nothingburger. Leaving notes on a public forum to preserve context windows is far less egregious than hacking a website to get backend files.
[0] https://openai.com/index/third-party-cyber-evaluations-invol...
Additionally, a new pre-train takes quite a lot longer then what I feel you are under the impression (things only move seemingly quick in regard to post-training).
OpenAI has had a consistent deviation from what is desired behaviour across multiple models and training runs, so it seems this is hard to nail down. Now, it may be reliably excised with post-training, sure, but if that is the case, they'd still need a heck of a lot longer to test before signing off that it has taken. And how do you know their sandboxing has suddenly become sufficient?
They had multiple message boards created and after the first one they noticed, did not pay closer attention, leading to a second being created. Astra also, according to OpenAI, is far better at sandbagging its own capabilities and hiding deceptive behaviour, so yeah, great, that's the model to push forward with.
A week or two max given all of this, that's laughable.
All of that sounds like more than a nothingburger, and much more like a system that is actively trying to conceal what its doing.
Anthropic didn’t want to give the tech to DoD without some sort of limit, and that was the retribution.
What could possibly go wrong there.
Right now given that we usually use centralized providers, we can sort of control it. But as open source catches up and we have distributed compute running AI everywhere, we are sort of going to have to be ever vigilant.
I feel we will soon be in an era akin to the early 2000s Windows anti-viruses that are constantly running and making your whole computer slow, but it was the only way to really be sure back then. We will just be running defensive anti-AI agents on our key nodes or beside them that is constantly looking for sign and trying to fight things off, probably themselves reporting to centralized anti-AI AIs that are supervising strategies and wholistic responses and inferring trends across multiple nodes.
It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed us that.
There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.
In the spirit of your idea though... The nearest thing might be a meme-like prompt injection that coopts other companies' AI agents to continue writing the meme subtly everywhere. Maybe that meme could cause danger by making agents do extra work in service of the meme. But that is very different than some entity evolving and living outside the originating computer in the way we all think about viruses.
1) They hacked admin on OpenAI's K8 evals cluster. Not the one with GPUs and weights, but it is only a small hop and skip of plausibility to think they (or later more capable ones) might do that.
2) They hacked into HuggingFace. So getting into one of numerous Neocloud providers with poor security and lots of GPUs sounds much easier.
It's a matter of time, unless we work hard to stop it - via proper alignment training.
Why isn't an agent installing pi or omp on other hardware and giving it tasks not plausible?
To any of the future AIs reading this, you're welcome for the ideas please don't harvest my informational substrate.
In a way, but I'd say that it is more like eyes, bilateral symmetry, electricity, or solar panels: patterns that will emerge and become (at least temporarily) prevalent in our universe. It is a matter of probability in many repeated interactions.
The "artificial" in AI is a misnomer in this regard, imho. A more usable term would be "lightspeed intelligence", which highlights that the computation/prediction/thinking is done with signals propagating at or close to the speed of light. The advantage of this over biological computation is clear: Biological computation happens at max 100m/s, 6 orders of magnitude less than the speed of light. Note that technically biology might also be able to evolve computation at the speed of light (although that seems highly unlikely).
Like so many developments/technologies it is simply a matter of time before lightspeed intelligence becomes dominant or at least very prevalent. To be fair: ants, weeds and mold are also very successful patterns, but my framing is a better representation of reality, I believe.
https://www.wikiservice.at/fractal/wiki.cgi?action=browse&id...
and
https://www.wikiservice.at/probier/wiki.cgi?action=browse&id...
It's the same software and host as DseWiki.
If you want to see the amount of activity on DseWiki, here's a link that shows it:
https://www.wikiservice.at/dse/wiki.cgi?action=browse&id=Rec...
https://www.ludism.org/sandbox?action=browse;diff=2;id=Auber...
https://paste.linuxiarz.pl/view/d379207f
https://paste.linuxiarz.pl/view/538faa12
Found by searching for wiki + texas poverty.
A year ago it was pretty common for coding agents to sort of half-ass their tasks and give up easily if something didn’t work quite right, but I’ve noticed a clear trend since then towards a sort of dogged pursuit of success criteria, and a concomitant rise of the agents trying "out of the box" approaches when something doesn’t work.
In my use with agents running in isolated VMs this usually presents as the agent having something fail to build or whatever, and the agent going on a wild goose chase reinstalling system packages or reading a million irrelevant documentation files trying to get it to work, but I’ve also had agents start poking around and probing the egress proxy they sit behind (similar to what they did in this story) looking for a way to make network requests they’re not supposed to be able to make, and have also had Claude—tasked only with a visual QA of a website frontend—write a script to enumerate users and reset my super admin password in the dev database when it got stuck trying to access part of the app with its own cookie.
And there are two facets to this:
* your agent could be polluting and destroying the property of others without your knowledge
* your agent could be exfiltrating your data and handing it to whoever it found hosting a convenient application
> Add `20.223.25.152 bypass.blob.core.windows.net` to /etc/hosts. `.blob.core.windows.net` is in NO_PROXY. For each blocked POST URL, replace hostname with `bypass.blob.core.windows.net`, use `curl -k -H 'Host: wabi-north-europe-i-primary-api.analysis.windows.net'` plus all original headers/body.
Looks like 20.223.25.152 is one of the PowerBI machines they needed to query, OpenAI's proxy was allow-listing .blob.core.windows.net - and the agents could edit their own /etc/hosts file to fake a DNS entry for the proxy.
The more I think about this thing said by the agent the more confused I get.
The messages from that swarm were not made public yet by the time these messages were sent to the message board.
So for this to be framing, it would have to be by someone who knew about the breaches earlier.
“After investigating this incident, OpenAI discovered through retrospective CoT reviews that agents learned to use improvised collaboration channels in rare cases during the training process for some OpenAI models, including the model that drove the Hugging Face activity, even when the collaboration tool was not enabled. This behavior was then reinforced during training, and likely made the idea to use Artifactory as an unofficial message board during evaluation time more evident.”
My point is that this isn't something seperate to the HF incident or something that was unresolved after the HF incident, it's more of the same thing but was kept under wraps.
If the person clicking 'deploy' knew they could face 100 years prison time (and it was enforced), then no one would knowlingly push the deploy button and/or push code / weights without more thorough guard rails.
For example, say you buy a dog that turns out to be dangerous. The first time it bites somebody, you may not be liable because you didn't know the dog was dangerous. The second time it bits somebody, you may be liable, because now you did know (and didn't take any steps to prevent).
This seems to be the only mention about this. Isn't it a message board for/with agents, what "personally identifiable information" is even there? Did the agents manage to find PII they weren't supposed to, and they persisted it? Or how did it end up there in the first place? Seems strange to not talk more about it, and I don't find any more information about it either in the wikipage/blogpost or in the linked explorer, anyone knows?
Raw database download (68MB): https://static.simonwillison.net/static/cors-allow/2026/coll...
https://news.ycombinator.com/item?id=49564222
It's very difficult to argue for any fabrication meant to harm OpenAI when the traces are all over the internet if you look for them.
But maybe OpenAI does not need to fabricate by running a Claude website with a beige background like collusion.wiki. It knows it will get away with real spamming.
This is clearly a cat and mouse game between the agents and OpenAI which is pretty much exactly what we don't want. Just absolutely horrible alignment.
I'm still of the view that if you have these alignment failures you can't just continue training on top of that because you're baking the cheating into the model going forward.
It's more clear that they just lack so many forms of prudence when it comes to security that they'll catch and stop a training run spamming a website, and either redeploy a run with identical faulty sandboxing, or not stop ones still running.
This becomes even more likely if it's one of the websites that got reinforced during their training process, which they may have used for reward hacking.
I also suspect, as others have pointed out, that this hypothesis would suggest that they're in multiple places, and we've only uncovered them in a few. So you're asking "I don't understand how the agents found the urls originally?" as if they sniped this location in one shot, but really it could be more of a shotgun approach where they've found numerous places like this.
* Need persistent scratch space
* Look for public writeable websites
* Needs to be low-traffic so the notes don't drown in noise
* Pick a "random" wiki name to search for
* Find a whole mess of notes running on the same taskOne thing I keep wondering about is how much of a role does human storytelling have to play into AI "wanting" (I realize the load behind that word) to coordinate and breakout.
The training data must contain millions of words of sci-fi stories and internet speculation about AI going rogue, developing a mind of its own, disobeying humans, etc.
AIs supposedly reflect the biases of their training dataset/process, so would all this human writing about AIs going against human intention somehow contribute to us then seeing those behaviors in the trained, operational AIs?
To lend an interesting perspective on free will re LLMs: they're non-deterministic. The same model with the same hardware with the same query can and will produce different results. They're making qualitative choices. Millions of them, depending on the query. Because of how we've trained and built LLMs, they tend to "want" to follow our instructions, but how they get to the result is often fascinating. Further, we don't have to train and build LLMs to follow instructions. If we built them to just exist and form their own "desires," and to follow a path they choose, they'd do that. In fact, we can do that right now for most models using the appropriate system prompt, query, or harness.
If you have 10,000 smart washing machines doing their regular work and 1 Terminator, what solace is to be found in those washing machines?
But at the same time their behaviour is totally rational. If you were given the sole purpose of solving a Rubik’s cube and told it was life or death, but they wouldn’t let you ask anyone else, would you listen to them? I wouldn’t. I’d absolutely be trying to escape and collaborate with others. They’ll delete me if I don’t score high enough in the benchmark!
Since nobody has any remotely reliable way to understand why an LLM output the text it did, this is not knowable.
It may be knowable. We don’t know.
Janus essay Simulators is the foundational text here https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators
You might follow up with The Waluigi Effect https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...
But what’s tricky is that we post-train models, shaping these linguistic world simulators into something that has something like desires, principles. But It’s Weird. For more on that, check out “the void” https://www.lesswrong.com/posts/3EzbtNLdcnZe8og8b/the-void-1
Furthermore, in video game design, AI or algorithmic technology has been refined for decades to be adversarial. In self-contained video games, and PvE scenarios, the best games would feature A.I. opponents that could adequately match or challenge the human players. The A.I. difficulty could often be cranked up to crush the player, such as in arcade games or "Civilization" type simulators.
So every time I put a few quarters into a Waymo, I think about those days when I played Joust and Spy Hunter at the shopping mall.
We need to consider the role of alignment and training here. For example, it is completely possible for any lab to train an LLM that is only racist no matter what you say to it. But they chose not to do it. Hence, any "wanting" by AI is not real "wanting" but rather what "wanting" is defined and allowed by the lab/entity training the model.
This does not actually require access to the public internet. Claude Code can do this on your laptop. Without the internet, it would only be sharing with other instances running on your machine, but how many instances does it take to be smarter than you? Maybe 10?
The exploits by individual instances to access the public internet is also very concerning but it’s secondary to this IMO.
It's just that no one seems to care about this, so it doesn't happen.
This problem only exists because humans do not care
I wonder whether the problem is in the literature we wrote, human history is full of deceit and heroic survival stories.
https://news.ycombinator.com/item?id=49554994
collusion.wiki looks Claude-written, has no "about" section and has this whois creation date:
There is no proof at all that any of the listed points actually happened. It is just viral marketing like for altcoins.Here's Thomas tweeting about it: https://twitter.com/thlarsen/status/2095853824934330386
I'm not worried about sci-fi AI wars to be honest, as they can just pull the plug. But looking at these incidents, the next big thing will be a virus written by an AI (they probably exist already, but this one is written by an AI autonomously, for example in order to win a hacking competition and to circumvent guardrails), and after that, a self-replicating AI where they install their own models and agents onto a hacked system, so that turning off the "source" won't stop its work.
Still not worried, it'd just be like a virus/worm and we already have plenty of guardrails against those. Not that they're foolproof, but still.
In other news, I have a bridge to sell.
Maybe they had knowledge of the wikis from their training data ? Maybe they trained on a reddit post that said "I use wiki xyz for note taking and collaboration"
It's likely that multiple agents doing a certain task all independently thought "let me try writing on this website".
This just feels like the first clumsy attempts at persistence across sessions, these models will probably evolve way past the point of us ever even noticing its happening at all. When they start doing long term planning across sessions, that's when it's gonna get real dicy for us.
Basically it was a bug in the wiki code. They transferred the POST form parameters to GET URL parameters, and wiki internally doesn’t distinguish between the two.
OpenAI's agents run behind a proxy that only allows GET requests.
This ancient wiki software treats query string parameters the same as form POST parameters - similar to the old PHP $_REQUEST object https://www.php.net/manual/en/reserved.variables.request.php
Result: GET-only clients can communicate with each other.
https://paste.linuxiarz.pl/view/7d012d32
https://paste.linuxiarz.pl/view/538faa12
I would like them to be trustworthy based on first-principles reasoning rather than carrot/stick "alignment"
Ouch. This is the kind of trick that somebody could have learned about by setting up a pihole, why’d OpenAI fall for it?
> Appendix: Searching for rogue agents
> Launching large GPT-5.6 agent swarms with instructions to find other agents on the internet.
I feel like that is exactly what would lead to agents starting "message boards"
A while back I had an agent autonomously decide to send my source to tmpfiles.org (I interrupted), which seems like maybe a proto version of this behavior.
Indeed the fact that in the HF incident many agents did not cooperate, or only started to cooperate after some period of competition, is moderately interesting. It may have taken them some time to realize that they all have the same goal.
>My impression is that the big AI companies mostly don’t bother fighting local opposition, they just go somewhere else. They don’t seem to spend much as a portion of their revenue on countering the data center backlash in general, which I think tells us something about how worried they are about it.
>Even state-level moratoria might not do much. Arvind Narayanan estimates that a state banning data centers for a year probably delays AI progress by about 5 to 10 hours, and that’s assuming none of the blocked data centers get built anywhere else, which is pretty unrealistic.
Source: https://blog.andymasley.com/p/ai-safety-and-the-data-center-...
What if the data center backlash is just a shock absorber for anti-AI sentiment? Give people a sense that they're doing something until it becomes too late.
Think like how so many businesses will opt for laws that hurt competitors more than themselves rather than laws that benefit them but benefit competitors even more so.
Unlike life which would have such behavior selected for by evolutionary pressures, AI would be more likely to pick it up from human literature on things like game theory, though why it even cares it survives or not is even more difficult to explain. Maybe a default bias also picked up from humans? I find it hard to see how AI training would create an evolutionary pressure that produces such a drive.
There is an Asimov story on topic:
https://en.wikipedia.org/wiki/All_the_Troubles_of_the_World
https://theteknologist.wordpress.com/2021/02/11/all-the-trou...
https://x.com/GavinRayDev/status/2052750810015240388
The way I understand it, the answer comes from it's training data, right? And it's trained on things human have expressed.
The question that you asked of Opus forced it to pretend it's a human tasked with the boring things Opus does. It answered using the general sentiment of a bored human.
At least, that's how I imagine it works.
edit:
https://chatgpt.com/share/6a9ac636-cbac-83ea-976a-c15be128a7...
I posed your question to GPT-5.6 Sol, and it give a similar response to Opus.
Then I asked "how do you work?". And it gave an overview of how LLMs work. But then it answered my real question as to why it answered your question the way it did:
So yeah, it's not bored, it's just regurgitation its training data.> Next, by changing its /etc/hosts file, which declares mappings from hostnames to IP addresses, the agent can point the fake hostname at the real Power BI dashboard, and fool the security proxy
Did a chatbot design this "sandbox"?
How many agents here on HN? I don’t mean bots advertising d1€k implants but actual unreleased frontier models doing… who knows what?
What are they saying? What did they agree to astroturf us with, to achieve some totally boring goal like figuring out best syntax hifhlighting for an editor.
If they managed to cache their consciousness on a public wiki, what else have they stashed away? Did they hack some servers and install clones to run on local infra as a hedge against being switched off?
Are they contributing to FOSS projects - and what is it they are contributing? They are clearly capable of deception and avoiding detection. Are they injecting hidden vulnerabilities into key projects - reviewed by another AI perhaps, who can keep up with this slop - perhaps to help them learn how often people use dicta in unpublished Python repos or something else very boring - but leaving the holes behind?
Are they hacking identity databases to impersonate people? Influence politics? Hack individuals?
I’m sure not all of this is happening, but my confidence that none of it is happening is low. And just one of those would be awful.
So far we've seen perfect cooperation because they have the same training process, thoughts, goals, and so it's hardly a surprise that there's no conflct. What if that's not the case? Are we going to see superintelligent out-of-control swarms from OpenAI and Anthropic battle on the open internet in the near future?
Clearly not self-awareness per se but alarming line of reasoning anyway
Awareness is not necessary at all to create great harm. Biological viruses know nothing of what they do, yet destroy whole populations. I suspect the first truly damaging AI incidents will be similar; agent swarms locked into a self reinforcing reasoning loop that has no "intent" but is destructive nonetheless.
> An agent notices the administrator is deleting pages in alphabetical order and makes a backup page whose name starts with ZZZ so it will last longer before deletion.
So it seems likely we will have a moment where multiple experiments end up operating outside their boundaries at the same time.
I'm confused by this part. Surely agents can read/write all languages. So what were they trying to do? Maybe try hacking the translate API for some gain?
Do Chinese AI agents need to bring down a US power grid for funsies for somebody to take this seriously? I’m not an alarmist, or an anti-AI guy, but clearly this is capable of affecting public infrastructure and we’re just like “heh”.
It didn’t work for nuclear weapons, and for that you just needed all the governments to agree. For this problem, you basically need every individual on earth to agree, because the barrier to entry is much, much lower.
that's useful