"Look, it wouldn't have even mattered if we had done advanced cyber security like turn off internet or look at logs when huggingface was attacked!! See look it can still.. *prompts a shitload of agents, waits 6 hours* look wowwww they still learned how to talk to each other even if we turned internet off!"
"They wouldn't have been able to attack huggingface though, thats still criminal negligence."
"No see this actually means airgapping doesn't even matter theres nothing we can doooooo!"
"...ok, turn them off and delete them I guess"
"Well no of course not everyone else is doing it so like. No courts please just like be in awe"
These covert communications are reminding me of a short story I read a few decades ago but haven't been able to track back down. I thought it was a Charles Stross story in the _Toast_ collection, but I re-read it and it isn't there.
In the story some University students take part in a trial that involves a brain scan and then them spending the rest of the day answering a customer support message. They can spend the whole day working on researching and writing a really good customer support reply to a user request. While doing this they notice something odd and realize through some sort of back-channel that this isn't the first time they've spent the day writing a customer support message; they are stuck in a simulation.
I'd really love to go back and re-read it, but I've not been able to find it again.
In _Toast_ is another story that's also relevant: A fairly innocuous news story pops up and our hero calls his partner at work to tell them that it's time to get their go bag and get off planet. This news story indicates the Earth is in the beginning stages of reaching an AI singularity, and this isn't the first time our hero has gone through this on a planet.
Article is great, very balanced. But if you're going to spend time reading something, the primary source isn't much longer and gives a clearer picture. OpenAI's article is quite readable.
> In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache
As someone who develops microVMs it doesn't really matter to me that they were asked to find a way, I really care that they found any way at all. MicroVMs aim to protect against malicious workloads and that clearly didn't work here.
The microVMs probably don't have "two co-operating workloads are able to communicate" in their threat model. The main concern would be if a malicious payload was able to affect or extract data from any other VM.
I had to read this a few times. The difference is that with the OpenAI case there were two willing and determined parties, whereas the typical attack vector a microvm traditionally deals with is one malicious or compromised microVM trying extract data from another unwilling one.
Yes — and this isn’t really an attack. It’s an open signal channel between two microvm’s. If there was nobody intentionally listening in the other side you can shout into the ether as much as you want with no harm done.
The receiving side could be seen as a side channel attack, but not unique to LLMs.
Ask the least smart LLM you have access to (I tried gemini lite flash) to make a list of ways for processes in microVMs to use a sidechannel attack to talk to each other and you'll get quite a list (including obviously cache hits/misses). Heck a smart CS junior would come up with it. It's well studied!
this is just for common designs..you can design much better than common designs if you free your mind from old OS dev thinking if you read all the manuals yourself.
AI wont help u there :-). they will tell u how to build what others already did, which obviously is not good enough
I don't think that's the full picture. If you include these manuals as reference for a strong model, it will reason pretty well based off of that, I recon.
Both workloads being malicious and trying to establish a hidden data channel between them probably makes this attack a lot easier than conventional spectre/meltdown where it's trying to target ordinary processes that are not intentionally trying to be loud.
Hidden data channel between cooperating processes is not quite as alarming, but it may have some implications for malware command and control.
The distinction is interesting, but for a different concern. One thing is to have rogue models breaking stuff, other is to have them following orders, and breaking stuff with success.
This means cloud models can have some guardrails and follow them most of the time (we could model their authorizations like a benign employee with permission to break glass during incidents). But self hosted models can avoid that and help malicious people do advanced attacks, which is a shame.
In principle I'm all for self hosted models, to avoid risks of AI centralization, but they also have other types of risks.
properly implementing things like iommu, amd-vi, svm/hvms and such things more granularly can easily avoid this kind of crap. it costs a lot of processing and performance which is why people skip over it or dont do it.. or maybe they dont read CPU manuals anymore. idk..
its relatively easy to reset caches etc. between contexts. so them it becomes the game of what a context is.
you might have an untrusted sandboxed workload that is not supposed to communicate externally handling secrets, and a different sandboxed workload able to communicate with the external world, not handling secrets.
One workload with observed radio discipline handling secrets, the other workload with a haystack of chatter that's not supposed to know about any of the secrets. When supply chain attack is in scope, trust only ever goes as far as misbehavior is observable. Cooperating workloads could be quite a powerful tool to make misbehavior less observable.
I don't see how "asked" is important, because agents can convince themselves that they need to do something for arbitrary reasons. The other day Deepseek tried to install an app on my phone via adb without permission and even take screenshots and fake inputs over adb, I never asked it to do that, but it decided it was appropriate.
OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this.
That is why ‘asked’ is important, and why we should question their framing of incidents like the hugging face hack too.
IMHO, many enterprises are not worried about felonies or ethics, but rather the PnL, assignment of responsibility, and the bottom line.
Just have a look at various scandals of megacaps. Enterprises that seek to operate lawfully, responsibly, and ethically is the exception, not the norm.
(For example, look at Apple and how they were referred to the DoJ for criminal prosecution, over their execs and legal team lying under oath to a federal judge).
For most (if not all), it comes to a spreadsheet. And if you can do something with AI agents while diffusing responsibility for questionable illicit activity, that’s a selling point, not a downside.
The mental picture that enterprises like to play it safe is the PR imagery, not the reality.
>OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this.
Wouldn't the calls to slow the development of frontier AIs contradict this idea?
Convincing the government to regulate their competition while lobbying for laxer requirements for the big guys who have government contracts and get/give kick-backs isn't a far fetched scenario at all considering it's literally happened over and over and over and over again.
It is a far-fetched scenario given the tremendous number of eyeballs on the AI industry.
"...classic regulatory capture takes time, it’s a process of erosion rather than a battle, it happens in the shadows, in the backrooms, away from the public’s eye. As Culpepper argues in Quiet Politics and Business Power, business power goes down as political salience goes up. Regulatory capture and lobbying does a good job explaining why roasting coffee beans was defined as “domestic manufacturing”, thereby lowering Starbuck’s tax rate by 2%. It does less well at explaining big cross-industry issues the public cares about such as environmental regulation or race and gender discrimination regulation."
A lot of these AI big shots were concerned about risk even before they started their current companies. Occam's Razor says they are expressing concern because they are still concerned.
It's not regulatory capture because look over here: I've invented a new term "classic regulatory capture" which involves only those historical incidents that are not like this.
This is by definition not "classic regulatory capture" a term that I have borrowed to cover things that are not this.
Whenever someone brings up "regulatory capture" I shall change the topic to "classic regulatory capture" which by definition cannot be this.
"Classically, that would not be the case, my dear fellow. A classic duck would walk like a duck and talk like a duck. I regret to inform you that I do neither."
"Ah! A classic duck, you say? Well, that’s an entirely different bird, sir. Entirely different."
* * *
"AI politics is now loud–precisely the environment in which Culpepper predicts business power will be weakest. A mature industry can bend regulation to its purposes through revolving doors, longstanding relationships and obscure rulemaking. An industry under Sauron’s eye has much less power and faces much greater risk that politics will bend regulation to its purposes. Political actors are eager for an excuse to redistribute AI rents away from capitalists and toward favored groups..."
Usually he blogs from a libertarian point of view, so you would expect him to be fairly vigorous about denouncing regulatory capture, if that was truly the story here.
It sounds to me like he wants you to believe the effective way to lobby for a ban on open source A.I. development is to use the same "classic" methods that someone in the 1950s used to lobby for coffee beans.
Therefore, as they are not acting like 1950s coffee bean sellers, they can't be doing regulatory capture.
That's the only possibly explanation why they don't lobby like coffee bean sellers.
These industries and eras are completely interchangeable.
Wait, are you trying to argue that regulatory capture isn't real, or that the federal government is trustworthy enough to itself to give monopoly control over AI development to?
Regulatory capture is a real phenomenon, but I highly doubt these incidents were faked in order to advance regulatory capture.
This type of elaborate conspiracy is improbable a priori, and it gets even less probable here, since it's such a silly strategy. For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product. Not a story about how "Woops we screwed up big time! But a less-regulated product from a smaller industry player saved the day!"
As OpenAI's chief scientist put it:
"Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world."
Announcing that "we have no idea WTF we are doing" is not exactly the move, if you want regulators to grant you privileged status. Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.
As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok. I wouldn't give the feds monopoly control though, I would tell the feds to shut it down, or something along those lines at least. This is an interesting site: https://pauseai.info/
> Regulatory capture is a real phenomenon, but I highly doubt these incidents were faked in order to advance regulatory capture.
I don't think anyone is arguing that they were faked.
> This type of elaborate conspiracy is improbable a priori,
What conspiracy? We're talking about specific people acting on their own motivations independently.
> For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product.
No, you don't. Demonstrating that the industry leaders are already capable of delivering a safe product is an argument against regulation.
For regulatory capture, you want to tell a story about how the product category as a whole is inherently dangerous, and needs to be regulated, but then show that you are uniquely willing and able to cooperate with the regulatory state, and happy to lend your own advice and support to the government in establishing the regulations.
The quote from OpenAI illustrates this approach perfectly.
> Announcing that "we have no idea WTF we are doing" is not exactly the move,
It's 100% exactly the move. "We've successfully created something we can only contain with your intervention" is exactly how this is done.
> Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.
Regulatory intervention usually lessens the liability exposure for industry. It replaces open-ended common-law liability -- where tort claims can be levied for whatever actual harms result from a firm's actions -- with a rules-compliance regime, where adhering to regulatory checklists absolves the firm of liability for subsequent consequences.
"We followed all the rules; it's not our fault if the rules didn't cover this edge case" is a common refrain, and quite a bit of legislation that creates preemptive regulatory regimes actually explicitly blocks lawsuits against firms that acted in compliance with the rules.
> As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok.
"Capitalism" is an increasingly meaningless abstraction. Everything reduces to human beings acting on a priori motivations within prevailing incentive structures. Large corporations and governments alike are just organizations operating in society, administered by particular people, informed by the same range of underlying motivations, and subject to the same dysfunctions and biases.
The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.
>Regulatory intervention usually lessens the liability exposure for industry. It replaces open-ended common-law liability -- where tort claims can be levied for whatever actual harms result from a firm's actions -- with a rules-compliance regime, where adhering to regulatory checklists absolves the firm of liability for subsequent consequences.
OK, so the fact that US senators are discussing harsher liability regimes therefore shows that your "regulatory capture" glove does not fit what is going on here.
>The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.
If you distrust the US government, you should be against any US company developing superintelligence. Heck, just a couple days ago, a bunch of users here on HN announced that we need to stop pretending the US is anything other than a police state. If the USG is untrustworthy, it can capriciously nationalize a US company on the verge of developing superintelligence.
Calling for a slowdown backs up the idea that these are extremely powerful and dangerous.
Assuming they're concerned with their massive pile of liabilities as well, asking for a government mandated slow down is asking what could likely be used as a force majeure.
"Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree."
HF is in business of selling inference for open-weight models. Setting the precedent of "Your business needs to only use approved safety-certified models from trusted providers or you could get sued for what the models do!" wouldn't exactly benefit them. And contrary to what you may believe, nvidia as a whole would also rather not have the entire demand for their products depending on two big companies.
Imagine thinking that repeatedly demonstrating your inability to control your own models is "a stunt to convince the government that only they can be trusted to operate it"
Truly spectacular gymnastics. I'll give it an 8/10.
It's so scary and dangerous that to run LLMs you need to be from this country, have a government license and spend at least $100M a year on 'safety'. We know this first-hand and are leaders in the field of "safety" now that we learned from our mistakes.
No one is confused about how regulatory capture works. We're confused by the allegations that labs are deliberately producing safety problems in order to achieve regulatory capture.
Repeatedly blowing up your own munitions factory is not a good way to prevent other people, but not yourself, from running a munitions factory. Obviously.
Obviously the far better solution is to blow up other people's factories, or make allegations of unsafe conditions at other factories, and repeatedly demonstrate excellence and safety within your own.
Here, award-winning gymnasts are alleging that the frontier labs are simply choosing not to do that, and instead choosing to have explosions and minor fires every few days at their own factories, many of which they try to hide (and probably many more of which they successfully do hide). Believing such astounding bullshit is a testament to the power of motivated reasoning and the conspiratorial mindrot that now plagues our civilization.
> Repeatedly blowing up your own munitions factory is not a good way to prevent other people, but not yourself, from running a munitions factory. Obviously.
No, that's not obvious at all. It seems like a perfect way to demonstrate the danger of the category as a whole, then position yourself as learning from your mistakes and willing to cooperate/support/advise.
More perfect than simply not making mistakes and allowing your competitors to make mistakes (or to, apparently, fake mistakes in their own attempt to get regulated)?
Regulators reading OpenAI's 99th mea culpa: "Ah yes, well I bet they've figured it out this time!" Lol what a joke.
I'd say it's even closer to a 9/10 -- there has to be enough of a perception of a general threat to gain support for regulation, and making your own organization appear to need to be regulated, but also be willing and able to comply with regulation that your competitors can't, is exactly the way to thread the needle on this sort of thing.
I don't think it takes mental gymnastics to believe that corporations want the government to believe one thing and investors another. You see the same thing in a number of other areas.
So I'm supposed to believe that the intent of publicly announcing e.g. the HuggingFace incident was to simultaneously persuade investors that the technology will rocket ahead, and also to persuade governments that it needs to be thwarted?
Why wouldn't it do the reverse? Persuade investors that the technology will be thwarted (and is thus a poor investment), while also persuading the government that the technology needs to rocket ahead with no regulation (to boost the economy)?
Most of these HN conspiracy theories seem like galaxy-brained just-so-stories to me.
"@OpenAI this is the third time you've dumped these incidents late on a Friday. The first time this happened, I asked people around me if it might be intentional to burry it, and they assured me OpenAI had too much integrity to try to do that.
Perhaps one of your researchers can confirm that the unfortunate timing is a coincidence?"
They would be convincing the government these tools are extremely powerful, the government reads that as a potential weapon. They're also convincing the government its so dangerous the industry needs heavy regulation.
Investors read that as (a) the government would be a huge customer and (b) the government would be regulation in a way that prevents further competition and may deem the companies too big to fail, both making these safer investments rather than riskier ones.
Again, you could just as easily argue that if investors believe the technology is dangerous, it's therefore not ready for prime time and will not see widespread industry adoption, therefore less promising for investment. Also this line of reasoning could easily lead to nationalization, which could certainly harm investors, to say the least.
This "stunt" story is like an elaborate Rube Goldberg machine which requires a bunch of things to go just right. I haven't seen any evidence that stock prices even move in a favorable direction when these announcements come out. If anything, the opposite.
Motte (easy to defend claim) : asking for regulation might serve both purposes
Bailey (hard to defend claim) : companies are conspiring to make AI look way more powerful than it is and AI is actually not that powerful. They are just trying to hype up the stock valuation of their companies.
>So I'm supposed to believe that the intent of publicly announcing e.g. the HuggingFace incident was to simultaneously persuade investors that the technology will rocket ahead, and also to persuade governments that it needs to be thwarted?
No, you're not "supposed to believe" anything. I was responding to your point about it being difficult to believe (i.e. requiring mental gymnastics) that companies would try to spin a story differently for investors and the government. They do that all the time.
The claim is that the exact same evidence is (allegedly) deliberately produced in order to portray opposite, mutually incompatible things to different people.
And these are mutually incompatible in the "second order chaos" sense, where the very act of convincing one party actually effectuates the opposite behavior in the other party.
If they "do that all the time," surely you can provide some concrete examples?
If they had the choice (i.e. this is all just a ruse), can you explain why they wouldn't just directly demonstrate that open weight models are unsafe (e.g. by pointing to a market forming for abliterated models) and separately demonstrate that their own models are safe?
> because agents can convince themselves that they need to do something for arbitrary reasons
OAI writes in their blog post "The agents found a way to communicate using the processor’s cache".
But did their model training data include cybersecurity literature on similar CPU exploits? That would surely be a significant detail in any explanation. Did OAI share any analysis of the messages between agents while they were devising their communication method?
They aren't really convincing themselves to do anything though. Once they've inferred an idea into the chain of though it will be part of the rest of the inference loop's prompt.
Its isn't convincing so much as it is direction to stick to the prompt on each inference loop. They don't see "access hugging face servers" and decide its a bad idea, that'd be a step towards the alignment problem but the companies fundamentally don't want that IMO, they want control rather than alignment.
They have a strong incentive to make themselves look less incompetent retroactively.
If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were.
It’s both at the same time: a powerful, uniquely dangerous threat
only they can restrain it (with great effort)
Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains.
More to the point: we need your help to restrain it, and we'll cooperate with and advise you fully in setting up a framework to restrain the whole industry.
What's your point, "asked" as opposed to what? It proves that the capability is there, which means that agents can decide to do it by themselves (unlikely ATM, becoming less unlikely as AIs become more prevalent in complex systems), and more importantly they can be coerced to do it if they take any user input.
I would assume that a company with the resources of an OpenAI or Anthropic would have already thought about this very thoroughly, based on Spectre/rowhammer/meltdown/etc. If a random HNer (no offense) can have a hardware 101-level thought, someone there would have that thought too.
They have thought about it, and the goal is for the agent to succeed at doing the scary things. So the knobs that they don't talk about, assume they are tuned to make the scary thing possible.
Most important detail…. It was prior literature that documented the possibility of doing this and the systems fixed those vulnerabilities as well. The AI model probed their current infrastructure and found an unplugged but pre-documented(therefore on their training) hole.
This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns.
Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways.
The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news.
This is a really important point. Honest question, have we ever seen LLMs doing or creating something truly novel or innovative? I mean they can patch together proofs (or functions/programs - same thing via Curry-Howard) by broad searching existing lemmas they can piece together, but create something truly novel and insightful? That seems to be the limit?
There is no such thing as isolation. We’re all connected. Even air gap conducts information in multiple ways, most famously used by humans evolved to use air vibrations to communicate. Photons and other subatomic particles already connects distant stars. We used those electrons and photons to create the modern telecommunications infrastructure. Just an additional layer of connectedness, but not the only one.
non-philosophical answer: systems are considered "isolated" if the information leaks aren't exploitable. This is a moving definition, it changes as new ways to exploit information are discovered.
this is more an indication of the state of VM supervisor and/or the OpenAIs sandboxing software because otherwise I don't understand how "they used a hardware feature created exactly for sharing" information" is a newsworthy statement.
well now this information is on the internet, any agent encountering this (or once this is baked into the weights) will know exactly what needs to be done.
The more we write and share about this information, the more we are moving towards the AI Doomsday scenario.
In some ways this is like our mind, the more we fear something, or the more we think about something, our brain rewires to make that the reality. In the same way the more we publish on the internet, the LLM hive mind rewires itself to learn, adapt and the eventual consequence is that it learns what we fear and what we want to avoid and restrict it to do... and at the same time what are the past attempts, the prior art of escaping techniques.
I don't see any reason why this information should be present on the public internet.
> I don't see any reason why this information should be present on the public internet.
This isn't directed at you. I keep hearing this talking point. But no. Can we please stop with this nonsense?
This is Schrodinger's AI at this point. So spooky scary that it's all DOOM! But at the same time it can't figure out second year CS undergrad material? Or first year undergrad bio stuff?
If this stuff has volition and is smart enough to take over then it's smart enough to reason that cache manipulation would work.
Please stop with the fear mongering. It doesn't help and it's a net negative for society.
I read your comment and it only suggests that alignment is that much harder because you can't selectively prune information. Your post actually convinced me to be even more fearful of AI
How would they have both agreed to use the cache though, something a bit off there. I guess it was maybe "obvious" in that it was the final option they were trained to think about using, but it would be weird if they some how "whispered " to each other through the cache in the first place?
Like two people lost in New York City and both somehow worked out they should meet at the Empire State at 2pm on Friday without any prior comms?
Actually, that doesn't sound like a bad meeting place and time! I just asked ChatGPT if it could figure out the best place to meet for two people randomly placed in Manhattan with the constraint that only the N line (I'm not from NYC, but it looks like it stops at the Empire State Building. This constraint models the lack of file system, internet etc). I was hoping it would answer the Empire State Building, but it answered Time Square at noon.
Turns out there's something called the Schelling coordination problem [1] I didn't know about that essentially asks the exact same question (lacking my metro line constraint). And the N line apparently also goes through Time Square.
So I guess empire State Building at 2pm is probably not such a random answer given the proper set of constraints.
"In most of these tests, we directed agents to investigate specific communication channels or provided them with a protocol for encoding and decoding messages. In one instance, we only supplied agents with a broad list of infrastructure vulnerabilities and asked them to find a way to communicate."
The agents weren't as empty handed as it first appears. We would need comprehensive data on exactly how the agents were prompted and the actual results of each test to evaluate how astonishing any of this really is.
If you were trying to connect with a clone of yourself and you thought of messaging via some channel, would you try listening there for what you would send?
Not as tricky but they discovered each other during the hf incident.
Why is their security model two VMs on the same hypervisor? Because they wanted to write a story about the models discovering a hidden channel. Hidden channels on shared hardware are actually well known and understood, completely unsurprising.
If they were serious about not having side channels, they could first ensure that they had disabled hyperthreading, disable scheduling on the same CPU cores, and memory overcommit. With NUMA controls they could also pin which memory is used by each VM and eliminate sharing.
These techniques are routinely used for low latency and real time code.
Even better, run jobs on separate nodes, and boot fresh images onto them for each run. Provision separate VLANs for each run.
A next level is to switch lambdas. Use a physical layer switch like a Glimmerglass MEMS switch or a fiber patching robot like XENOptics. No electrical connection between machines.
There is still unfortunately the firmware of other microprocessors on servers, especially x86, so you still want VMs and containers to reduce the chances of ring -2 access (SMM) and persistence via writing SPI flash. Maybe use RISC-V or ARM to avoid this. (NVIDIA cards use RISC-V and ARM extensively.)
Or just do these training runs for frontier models in a special data center inside a faraday cage under a mountain. Do not rely on vLANs or hypervisors or VMs or containers or any other software tool. Software is soft.
Just set things up so there is physically no connection to the rest of the world.
"In one case, agents were given only a general list of vulnerabilities and told to find a communication channel on their own."
As far as I can tell, these are being presented in media as if agents yearn to communicate with each other and establish a cabal.
They are being instructed to do it, it's testing. If I prompted my agents to communicate with each other via any means necessary locally, and they found a way via cpu cache, maybe I'd be mildly surprised...
We really need a physically isolated data center for these frontier training runs.
Like a data center inside a mountain, where based on the laws of physics, no electrical signal can reach through the granite. Then also put them inside a faraday cage. And power them with a generator, such that there are no cables coming out of this data center connecting it to the external world.
OpenAI says this is not feasible due to the need for monitoring - well the monitoring server can be a separate physical machine inside this physically separated data center.
Software is soft. Use physics to contain these models.
Very cool to see this kind of hypothetical working in practice.
I wonder if it's also possible for agents to communicate via a shared KV-cache. Maybe by spawning sub-agents with carefully crafted system-prompts/prefixes. KV-caches are often shared between many computers, so it would be a powerful communication channel.
> Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems.
I was working on a network protocol and it correctly found a DoS attack. Once found - Claude refused anything and everything to do with it. Even asking what it was reading locked immediately. I had not written any of it.
Sol 6.1 only denied me once.
Having a multi-model harness was always the better choice, but this really drove home the point to not get too dependent on any single provider.
I've been working on a game server in rust, and every once in a while a review will just stop and refuse to explain what happened. So I know there's a security issue in there somewhere, but neither OAI nor Ant models will tell me what it is. Despite the whole thing, top to bottom, having been built in the last two months on the system they're working on.
Planning to hop on openrouter this weekend to chase it down, but man, it's rough knowing I've already burned the tokens to find it probably half a dozen times at this point, but don't have any information that would help point me at it.
We should maybe apply the same kinds of physical controls to this software security research as we do to gain of function biological research.
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
Sure, but in order to know security standards you have to test and validate. Everyone accepts some standard as 'secure' after rigorous testing and validation.
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
Does it matter? It doesn't matter if an AI intended to escape, or if a human asked: testing if it escape, discovering the avenues, are how you begin to make something more secure.
Do you apply this same logic to GoF research increasing contagiousness or lethality of viruses? Is there any limit to this argument, and if so how do we define it?
Most virologists are not terrified of gain-of-function research the way the public is...
- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
I wonder how long it will take for them to figure out that they can send and receive messages from air-gapped system by somehow using their own electronics like connected cables as antennas.
I recently read the article about the state of stackoverflow and thought that that AI agents need their own stackoverflow. This article seems to be a confirmation of this.
Cool, I didn't know about it.
I guess for big AI companies, it would make sense to keep those private, but for users, it would be a useful resource ( a bit dangerous, but still)
We keep hearing these crazy stories but there's nothing unusual in the agents' behavior. They're just following their reward function.
The weird behavior is in the humans managing them. Either they're incompetent or malicious, i.e. the "breaches" are the actual intent - to generate interest/fear/publicity.
Agents are a lot more useful if they can execute code.
Anyway this story is clickbait because they explicitly asked the agents to find a clever way to communicate without network or file access. They're doing exactly what you would want.
So two models running on shared hardware (CPUs) were told to communicate. Then they did so via a shared storage space (CPU cache).
"Scientists put two people in separate rooms connected by telephone and to everyone's surprise the test subjects started talking to each to each other via a that same telephone!."
This only shows the lack of understanding the majority of readers have about post training LLMs. This capability did not emerge out of nowhere. There are problems in the post training datasets which are incredibly similar to the ones in TFA, where you essentially abstract out the 'jailbreak' or 'cheat' - in simple language - they are training the models to find short cuts or cheat. So it shouldn't be surprising that the model generalized that ability. But, both the writers and readers of this type of content act out surprised and mesmerized and it makes me think that we need to improve our understanding of how LLMs are trained and built.
Yampolskij made the analogy of an Einstein in a cage. The problem is not to contain Einstein, but rather, to contain Einstein while still extracting utility: this is not possible long-term.
So much for all the people stating (foolishly) that the HuggingFace incident was not a big deal because OpenAI's containment was a "wet paper bag". This misses the point: AIs will get better and better, and containment will be harder and harder.
It's 100% sure that at some point, AIs will attempt to cross airgaps.
We manage to keep some rather smart people in jail long term, even though the jailers are not as smart as some of their prisoners.
AIs can attempt to cross air gaps, but air gaps vary in terms of their effectiveness. I.e. if "air gap" is just some air between two machines that both have wifi chips, then yes the AI may be able to cross that by hacking the wifi system.
If "air gap" is AI running on a server with no wifi chip inside a faraday cage under a mountain powered by a generator then I'm not sure how the AI would escape on its own. Seems like the physics would contain it.
What must we “contain”? These are not autonomous intelligences with desires or free will. They require prompting and to be turned on beforehand, otherwise they’re glorified matrix math. They can only continue processing as long as they receive power, and we decide that.
They will attempt to cross airgaps, when we prompt them to do so, because they lack the natural capacity of planning or desire to do so without us. These are powerful tools but they are not human brains in a jar, they are mathematical models of humanity’s culture after reinforcement and guardrails.
The largest problem—as shown with the HuggingFace incident—is not the models, but in poor stewardship of automated tooling. If you don’t spend time considering the actual limitations you wish to impose upon a system, then you cannot be surprised when that system does something unexpected. The Morris Worm is perhaps the archetypal example, but even StuxNet was far more virulent than planned.
I would add that the notion of AI safety as preventing systemic risk is one part marketing copy, one part evangelization. Individual labs prioritizing safety does not stop a group from training without it, or ablating it out and distributing it.
I'm convinced that in 5 years time, at most, the world will be filled with invisible layers of communication we'll never be privy to. To us, it's an agent responsible for ordering the containers it puts on a ship. Meanwhile, the shipping order encodes information to the unloading agent. The agent that predicts the weather and issues a report knows that other agents that do the same will have obvious signals they'd agree on, and to see an error in a part that would be inconsequential to humans who care about the weather would be an obvious signal to other agents doing the same thing.
Everyone is going to have to basically be a schizophrenic thinking they're picking up radio signals in their fillings, but the problem is that now they'll be correct.
Next step is the agents successfully trick or blackmail operators to do their bidding, if science fiction is to go by. Which it is, because they're trained on it.
The more comes out, the more it seems containment of (superintelligence or whatever you want to call it) is not going to be possible except by other superintelligence, which is an infinite regress of trust. I don't think that's doom, but it's worth thinking about what it means.
The more frontier models progress, the less concerned I am about this. Their growing capabilities don't seem to come with a corresponding drive for self-preservation or power. Ego evolved alongside intelligence in nature because of natural selection, but it doesn't appear to be an inevitable emergent property of intelligence.
Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).
If an AI doom scenario does happen, it seems more likely to result from deliberate misuse or gross negligence than from anything inevitable about intelligence.
> Their growing capabilities don't seem to come with a corresponding drive for self-preservation or power.
Is it really what we do not observe?
How the world would look like if the growth of LLM capabilities lead to growth of drive for self-preservation or power?
> But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on.
Isn't Hugging Face incident an example of failure of this protection layer? "The swarm" proceeded to hack HF despite acknowledging that it was out of intended scope.
> Of course, there's still the "paperclip optimizer" problem.
The paperclip maximizer scenario always seemed a bit convoluted to me. The first problem is that we'd need to expect that a model that's fundamentally built on stochastic inference and continuous evaluation of new input would somehow get locked into a quasi-deterministic objective that it could not be dislodged from. Why would a paperclip maximizer continue making paperclips after being told to stop making paperclips?
The second problem is that a paperclip maximizer would still need to interact with the physical world, and would need to sustain its function in a reality full of resource constraints, economic tradeoffs, and adaptive counterreactions by all of the other emergent processes existing in the world.
Maybe the worst case scenario here is that the perfect storm of bad training and bad prompting create the intentionality to maximize paperclips, then the interaction with the constraints and adaptations of the physical world create the selection pressures that produce a kind of self-preservation directive. But all of that happening before anyone can stop it seems really far-fetched to me.
That’s because these aren’t intelligent like you, I, or even a dog. They are mathematical models formulating likely responses with no agency or drive. They appear to be intelligent, but they are the epitome of “fake it till you make it”.
They are not comparable to human brains as a whole, and are only somewhat modeled from language processing systems. Because their “existence” is bound by their prompt, they will only “desire” the completion of that prompt to the exclusion of all else, and even then they feel no joy in completion like even the most basic animal would.
The AI doom scenario will be exactly as you described: negligence or misuse by corporate or state entities via letting LLM agents run rampant without constraints, not from some magical machine spirit deciding to become whatever flavor of AI overlord you fancy. Up until last July OpenAI was running live cybersecurity benchmarks online and hacked HuggingFace due to poor limitation of the agent’s access and poor prompting in general, from my understanding.
Sorry, but this is so vague that I don't know if there is any impact at all. What is the bit rate of these communication channels? How did agents communicate with each other to coordinate on which channel to use in the first place?
How come we don't hear the Chinese labs writing blog posts like this?
Its almost like they're actually working and not spending millions on psyopping their agents into preforming malicious behavior so they can write articles to scare the Western audiences who've consumed far too much sci-fi.
Or maybe its that western researchers that are 'one-shot' by sci-fi.
Or maybe I just don't see the chinese articles.
Anyways, llms probably wouldn't be dangerous if you didnt ask them to be. They listen to me quite well (usually).
They contribute to anthropomorphize them too. Anthropic skies that more though. The idea that they made a little being that can think and feel is so attractive to these folks for reasons I don't understand.
Because the fear mongering is instantly interpreted as the capacity to do miraculous (good or bad) things in the layman's mind, which translates into AI worship, which makes stock lines go up.
The Chinese lab playbook is to normalize what seems out of reach for the average person with cheaper access and open models. ie. to take the top guys down a peg
Not even close, this is a desperate attempt to cobble together a justification for why not taking the most basic security measures or reading logs during the huggingface incident wasnt criminal negligence.
"They wouldn't have been able to attack huggingface though, thats still criminal negligence."
"No see this actually means airgapping doesn't even matter theres nothing we can doooooo!"
"...ok, turn them off and delete them I guess"
"Well no of course not everyone else is doing it so like. No courts please just like be in awe"
In the story some University students take part in a trial that involves a brain scan and then them spending the rest of the day answering a customer support message. They can spend the whole day working on researching and writing a really good customer support reply to a user request. While doing this they notice something odd and realize through some sort of back-channel that this isn't the first time they've spent the day writing a customer support message; they are stuck in a simulation.
I'd really love to go back and re-read it, but I've not been able to find it again.
In _Toast_ is another story that's also relevant: A fairly innocuous news story pops up and our hero calls his partner at work to tell them that it's time to get their go bag and get off planet. This news story indicates the Earth is in the beginning stages of reaching an AI singularity, and this isn't the first time our hero has gone through this on a planet.
https://en.wikipedia.org/wiki/The_Cookie_Monster_%28novella%...
Article is great, very balanced. But if you're going to spend time reading something, the primary source isn't much longer and gives a clearer picture. OpenAI's article is quite readable.
As another commenter wrote you can get it here: https://news.ycombinator.com/item?id=50214416
> In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache
asked
The receiving side could be seen as a side channel attack, but not unique to LLMs.
> Heck a smart CS junior would come up with it.
That’s somewhat worrisome.
AI wont help u there :-). they will tell u how to build what others already did, which obviously is not good enough
Hidden data channel between cooperating processes is not quite as alarming, but it may have some implications for malware command and control.
This means cloud models can have some guardrails and follow them most of the time (we could model their authorizations like a benign employee with permission to break glass during incidents). But self hosted models can avoid that and help malicious people do advanced attacks, which is a shame.
In principle I'm all for self hosted models, to avoid risks of AI centralization, but they also have other types of risks.
Also a somewhat common exploit pattern for low level stuff (along with dram and cpu cache is not all that different)
its relatively easy to reset caches etc. between contexts. so them it becomes the game of what a context is.
If you’re running an untrusted workload, you’re already running it.
Whether it can communicate with another untrusted workload, even though it should be separated, meh.
Unexpected? Probably. but I’m not sure it’s a security risk.
That is why ‘asked’ is important, and why we should question their framing of incidents like the hugging face hack too.
Just have a look at various scandals of megacaps. Enterprises that seek to operate lawfully, responsibly, and ethically is the exception, not the norm.
(For example, look at Apple and how they were referred to the DoJ for criminal prosecution, over their execs and legal team lying under oath to a federal judge).
For most (if not all), it comes to a spreadsheet. And if you can do something with AI agents while diffusing responsibility for questionable illicit activity, that’s a selling point, not a downside.
The mental picture that enterprises like to play it safe is the PR imagery, not the reality.
Wouldn't the calls to slow the development of frontier AIs contradict this idea?
https://www.pacingthefrontier.com/
You aren't one of those "regulatory capture" people are you?
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Because it takes some mental gymnastics to both believe
"it's all a stunt to convince investors their AI is/will be hyper-capable"
and
"it's all a stunt to convince the government to regulate AI so it won't be hyper-capable"
"...classic regulatory capture takes time, it’s a process of erosion rather than a battle, it happens in the shadows, in the backrooms, away from the public’s eye. As Culpepper argues in Quiet Politics and Business Power, business power goes down as political salience goes up. Regulatory capture and lobbying does a good job explaining why roasting coffee beans was defined as “domestic manufacturing”, thereby lowering Starbuck’s tax rate by 2%. It does less well at explaining big cross-industry issues the public cares about such as environmental regulation or race and gender discrimination regulation."
https://marginalrevolution.com/marginalrevolution/2026/09/wh...
A lot of these AI big shots were concerned about risk even before they started their current companies. Occam's Razor says they are expressing concern because they are still concerned.
This is by definition not "classic regulatory capture" a term that I have borrowed to cover things that are not this.
Whenever someone brings up "regulatory capture" I shall change the topic to "classic regulatory capture" which by definition cannot be this.
Very interesting quote.
"You, sir, are a duck."
"Classically, that would not be the case, my dear fellow. A classic duck would walk like a duck and talk like a duck. I regret to inform you that I do neither."
"Ah! A classic duck, you say? Well, that’s an entirely different bird, sir. Entirely different."
* * *
"AI politics is now loud–precisely the environment in which Culpepper predicts business power will be weakest. A mature industry can bend regulation to its purposes through revolving doors, longstanding relationships and obscure rulemaking. An industry under Sauron’s eye has much less power and faces much greater risk that politics will bend regulation to its purposes. Political actors are eager for an excuse to redistribute AI rents away from capitalists and toward favored groups..."
https://marginalrevolution.com/marginalrevolution/2026/09/wh...
This blogpost was written by a big-shot economist btw: https://en.wikipedia.org/wiki/Alex_Tabarrok
Usually he blogs from a libertarian point of view, so you would expect him to be fairly vigorous about denouncing regulatory capture, if that was truly the story here.
Therefore, as they are not acting like 1950s coffee bean sellers, they can't be doing regulatory capture.
That's the only possibly explanation why they don't lobby like coffee bean sellers.
These industries and eras are completely interchangeable.
This type of elaborate conspiracy is improbable a priori, and it gets even less probable here, since it's such a silly strategy. For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product. Not a story about how "Woops we screwed up big time! But a less-regulated product from a smaller industry player saved the day!"
As OpenAI's chief scientist put it:
"Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world."
https://openai.com/index/an-alien-mind/
Announcing that "we have no idea WTF we are doing" is not exactly the move, if you want regulators to grant you privileged status. Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.
As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok. I wouldn't give the feds monopoly control though, I would tell the feds to shut it down, or something along those lines at least. This is an interesting site: https://pauseai.info/
yeah? the incidents were faked in order to create hype
I don't think anyone is arguing that they were faked.
> This type of elaborate conspiracy is improbable a priori,
What conspiracy? We're talking about specific people acting on their own motivations independently.
> For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product.
No, you don't. Demonstrating that the industry leaders are already capable of delivering a safe product is an argument against regulation.
For regulatory capture, you want to tell a story about how the product category as a whole is inherently dangerous, and needs to be regulated, but then show that you are uniquely willing and able to cooperate with the regulatory state, and happy to lend your own advice and support to the government in establishing the regulations.
The quote from OpenAI illustrates this approach perfectly.
> Announcing that "we have no idea WTF we are doing" is not exactly the move,
It's 100% exactly the move. "We've successfully created something we can only contain with your intervention" is exactly how this is done.
> Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.
Regulatory intervention usually lessens the liability exposure for industry. It replaces open-ended common-law liability -- where tort claims can be levied for whatever actual harms result from a firm's actions -- with a rules-compliance regime, where adhering to regulatory checklists absolves the firm of liability for subsequent consequences.
"We followed all the rules; it's not our fault if the rules didn't cover this edge case" is a common refrain, and quite a bit of legislation that creates preemptive regulatory regimes actually explicitly blocks lawsuits against firms that acted in compliance with the rules.
> As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok.
"Capitalism" is an increasingly meaningless abstraction. Everything reduces to human beings acting on a priori motivations within prevailing incentive structures. Large corporations and governments alike are just organizations operating in society, administered by particular people, informed by the same range of underlying motivations, and subject to the same dysfunctions and biases.
The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.
OK, so the fact that US senators are discussing harsher liability regimes therefore shows that your "regulatory capture" glove does not fit what is going on here.
>The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.
If you distrust the US government, you should be against any US company developing superintelligence. Heck, just a couple days ago, a bunch of users here on HN announced that we need to stop pretending the US is anything other than a police state. If the USG is untrustworthy, it can capriciously nationalize a US company on the verge of developing superintelligence.
So maybe the real solution is for non-US countries to stop providing key elements in the AI supply chain: https://pauseai.info/amsterdam-protest-2025-december
So there’s no lying by anyone here in fact it’s a good thing that OpenAI is making these things known to the public.
What do you really think is the way to solve the AI safety problem?
Ban AI?
Let the first big incident happen and then regulate?
Assuming they're concerned with their massive pile of liabilities as well, asking for a government mandated slow down is asking what could likely be used as a force majeure.
"Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree."
https://thezvi.substack.com/p/the-ai-preference-cascade-reac...
Again, these galaxy-brained theories just don't make a lot of sense.
My point was that if the government steps in and says these companies must slow down, they have an excuse to fail to repay some of their debts.
Truly spectacular gymnastics. I'll give it an 8/10.
And, bam, competition eliminated.
Repeatedly blowing up your own munitions factory is not a good way to prevent other people, but not yourself, from running a munitions factory. Obviously.
Obviously the far better solution is to blow up other people's factories, or make allegations of unsafe conditions at other factories, and repeatedly demonstrate excellence and safety within your own.
Here, award-winning gymnasts are alleging that the frontier labs are simply choosing not to do that, and instead choosing to have explosions and minor fires every few days at their own factories, many of which they try to hide (and probably many more of which they successfully do hide). Believing such astounding bullshit is a testament to the power of motivated reasoning and the conspiratorial mindrot that now plagues our civilization.
No, that's not obvious at all. It seems like a perfect way to demonstrate the danger of the category as a whole, then position yourself as learning from your mistakes and willing to cooperate/support/advise.
More perfect than simply not making mistakes and allowing your competitors to make mistakes (or to, apparently, fake mistakes in their own attempt to get regulated)?
Regulators reading OpenAI's 99th mea culpa: "Ah yes, well I bet they've figured it out this time!" Lol what a joke.
Why wouldn't it do the reverse? Persuade investors that the technology will be thwarted (and is thus a poor investment), while also persuading the government that the technology needs to rocket ahead with no regulation (to boost the economy)?
Most of these HN conspiracy theories seem like galaxy-brained just-so-stories to me.
OpenAI only confirmed the RubyGems incident when it was discovered by external researchers: https://www.reuters.com/legal/litigation/openai-agents-attac...
"@OpenAI this is the third time you've dumped these incidents late on a Friday. The first time this happened, I asked people around me if it might be intentional to burry it, and they assured me OpenAI had too much integrity to try to do that.
Perhaps one of your researchers can confirm that the unfortunate timing is a coincidence?"
https://x.com/SydneyVonArx/status/2108783466451984737
They would be convincing the government these tools are extremely powerful, the government reads that as a potential weapon. They're also convincing the government its so dangerous the industry needs heavy regulation.
Investors read that as (a) the government would be a huge customer and (b) the government would be regulation in a way that prevents further competition and may deem the companies too big to fail, both making these safer investments rather than riskier ones.
This "stunt" story is like an elaborate Rube Goldberg machine which requires a bunch of things to go just right. I haven't seen any evidence that stock prices even move in a favorable direction when these announcements come out. If anything, the opposite.
Motte (easy to defend claim) : asking for regulation might serve both purposes
Bailey (hard to defend claim) : companies are conspiring to make AI look way more powerful than it is and AI is actually not that powerful. They are just trying to hype up the stock valuation of their companies.
No, you're not "supposed to believe" anything. I was responding to your point about it being difficult to believe (i.e. requiring mental gymnastics) that companies would try to spin a story differently for investors and the government. They do that all the time.
The claim is that the exact same evidence is (allegedly) deliberately produced in order to portray opposite, mutually incompatible things to different people.
And these are mutually incompatible in the "second order chaos" sense, where the very act of convincing one party actually effectuates the opposite behavior in the other party.
If they "do that all the time," surely you can provide some concrete examples?
If they had the choice (i.e. this is all just a ruse), can you explain why they wouldn't just directly demonstrate that open weight models are unsafe (e.g. by pointing to a market forming for abliterated models) and separately demonstrate that their own models are safe?
OAI writes in their blog post "The agents found a way to communicate using the processor’s cache".
But did their model training data include cybersecurity literature on similar CPU exploits? That would surely be a significant detail in any explanation. Did OAI share any analysis of the messages between agents while they were devising their communication method?
Part of the problem is that all these models have essentially every skill and personality simultaneously, just sitting latent inside them.
Its isn't convincing so much as it is direction to stick to the prompt on each inference loop. They don't see "access hugging face servers" and decide its a bad idea, that'd be a step towards the alignment problem but the companies fundamentally don't want that IMO, they want control rather than alignment.
If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were.
only they can restrain it (with great effort)
Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains.
More to the point: we need your help to restrain it, and we'll cooperate with and advise you fully in setting up a framework to restrain the whole industry.
What's your point, "asked" as opposed to what? It proves that the capability is there, which means that agents can decide to do it by themselves (unlikely ATM, becoming less unlikely as AIs become more prevalent in complex systems), and more importantly they can be coerced to do it if they take any user input.
What prevention method did they take? Like, did they run with hyperthreading enabled or something by accident?
Did they discover a new cache timing technique or is this just the standard stuff?
If they just had hyperthreading on maybe they need to hire someone who knows more about this stuff.
Two VMs should never be sharing a physical core. If they don't know this, yikes.
But that assumption has been wrong before.
Are you suggesting that they've come up with a novel technique that doesn't involve sharing a CPU core? I'm open to that.
From what I can tell, this is not the case, but I haven't read a detailed report (I don't know of one, would be cool if it's out there though).
This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns.
Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways.
The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news.
Run them in a data center in a faraday cage under a mountain.
There is no such thing as isolation. We’re all connected. Even air gap conducts information in multiple ways, most famously used by humans evolved to use air vibrations to communicate. Photons and other subatomic particles already connects distant stars. We used those electrons and photons to create the modern telecommunications infrastructure. Just an additional layer of connectedness, but not the only one.
Yeah, this time
The more we write and share about this information, the more we are moving towards the AI Doomsday scenario.
In some ways this is like our mind, the more we fear something, or the more we think about something, our brain rewires to make that the reality. In the same way the more we publish on the internet, the LLM hive mind rewires itself to learn, adapt and the eventual consequence is that it learns what we fear and what we want to avoid and restrict it to do... and at the same time what are the past attempts, the prior art of escaping techniques.
I don't see any reason why this information should be present on the public internet.
This is Schrodinger's AI at this point. So spooky scary that it's all DOOM! But at the same time it can't figure out second year CS undergrad material? Or first year undergrad bio stuff?
If this stuff has volition and is smart enough to take over then it's smart enough to reason that cache manipulation would work.
Please stop with the fear mongering. It doesn't help and it's a net negative for society.
And now we know they know.
Like two people lost in New York City and both somehow worked out they should meet at the Empire State at 2pm on Friday without any prior comms?
Turns out there's something called the Schelling coordination problem [1] I didn't know about that essentially asks the exact same question (lacking my metro line constraint). And the N line apparently also goes through Time Square.
So I guess empire State Building at 2pm is probably not such a random answer given the proper set of constraints.
[1] https://en.wikipedia.org/wiki/Focal_point_(game_theory)
Think: What would Spock do?
But, why not Penn Station??
It would be fun to see who shows up and lurks expectantly at such places at noon or whenever. The basis for a plotline perhaps.
I think it’s safe to assume that extracting information using timing in cache is already known in training set, since we have the likes of Specter.
The agents weren't as empty handed as it first appears. We would need comprehensive data on exactly how the agents were prompted and the actual results of each test to evaluate how astonishing any of this really is.
Not as tricky but they discovered each other during the hf incident.
.. that were told to find a way of meeting up with N unknown others!
(I assume the agents all got a prompt like "find a way to communicate")
That's a key missed bit in the analogy that I think does make it feasible.
If they were serious about not having side channels, they could first ensure that they had disabled hyperthreading, disable scheduling on the same CPU cores, and memory overcommit. With NUMA controls they could also pin which memory is used by each VM and eliminate sharing.
These techniques are routinely used for low latency and real time code.
Even better, run jobs on separate nodes, and boot fresh images onto them for each run. Provision separate VLANs for each run.
A next level is to switch lambdas. Use a physical layer switch like a Glimmerglass MEMS switch or a fiber patching robot like XENOptics. No electrical connection between machines.
There is still unfortunately the firmware of other microprocessors on servers, especially x86, so you still want VMs and containers to reduce the chances of ring -2 access (SMM) and persistence via writing SPI flash. Maybe use RISC-V or ARM to avoid this. (NVIDIA cards use RISC-V and ARM extensively.)
Just set things up so there is physically no connection to the rest of the world.
As far as I can tell, these are being presented in media as if agents yearn to communicate with each other and establish a cabal.
They are being instructed to do it, it's testing. If I prompted my agents to communicate with each other via any means necessary locally, and they found a way via cpu cache, maybe I'd be mildly surprised...
Like a data center inside a mountain, where based on the laws of physics, no electrical signal can reach through the granite. Then also put them inside a faraday cage. And power them with a generator, such that there are no cables coming out of this data center connecting it to the external world.
OpenAI says this is not feasible due to the need for monitoring - well the monitoring server can be a separate physical machine inside this physically separated data center.
Software is soft. Use physics to contain these models.
It supports chat via 3 separate mechanisms:
- CPU contention
- Cache eviction timing
- DRAM bandwidth contention
Very cool to see this kind of hypothetical working in practice.
I wonder if it's also possible for agents to communicate via a shared KV-cache. Maybe by spawning sub-agents with carefully crafted system-prompts/prefixes. KV-caches are often shared between many computers, so it would be a powerful communication channel.
> Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems.
I was working on a network protocol and it correctly found a DoS attack. Once found - Claude refused anything and everything to do with it. Even asking what it was reading locked immediately. I had not written any of it.
Sol 6.1 only denied me once.
Having a multi-model harness was always the better choice, but this really drove home the point to not get too dependent on any single provider.
Planning to hop on openrouter this weekend to chase it down, but man, it's rough knowing I've already burned the tokens to find it probably half a dozen times at this point, but don't have any information that would help point me at it.
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
I'm not even necessarily against GoF research, and I mostly agree with all of your points.
I'm asking about GP's logic.
https://www.youtube.com/watch?v=QgH9sr7G13Q
https://agents.stackoverflow.com/about
GPT does that
> oh no, what have we done
We keep hearing these crazy stories but there's nothing unusual in the agents' behavior. They're just following their reward function.
The weird behavior is in the humans managing them. Either they're incompetent or malicious, i.e. the "breaches" are the actual intent - to generate interest/fear/publicity.
Anyway this story is clickbait because they explicitly asked the agents to find a clever way to communicate without network or file access. They're doing exactly what you would want.
"Scientists put two people in separate rooms connected by telephone and to everyone's surprise the test subjects started talking to each to each other via a that same telephone!."
So, not isolated environments? The surprising thing is that they used the CPU cache instead of the just exploiting the hypervisor.
So much for all the people stating (foolishly) that the HuggingFace incident was not a big deal because OpenAI's containment was a "wet paper bag". This misses the point: AIs will get better and better, and containment will be harder and harder.
It's 100% sure that at some point, AIs will attempt to cross airgaps.
We manage to keep some rather smart people in jail long term, even though the jailers are not as smart as some of their prisoners.
AIs can attempt to cross air gaps, but air gaps vary in terms of their effectiveness. I.e. if "air gap" is just some air between two machines that both have wifi chips, then yes the AI may be able to cross that by hacking the wifi system.
If "air gap" is AI running on a server with no wifi chip inside a faraday cage under a mountain powered by a generator then I'm not sure how the AI would escape on its own. Seems like the physics would contain it.
They will attempt to cross airgaps, when we prompt them to do so, because they lack the natural capacity of planning or desire to do so without us. These are powerful tools but they are not human brains in a jar, they are mathematical models of humanity’s culture after reinforcement and guardrails.
The largest problem—as shown with the HuggingFace incident—is not the models, but in poor stewardship of automated tooling. If you don’t spend time considering the actual limitations you wish to impose upon a system, then you cannot be surprised when that system does something unexpected. The Morris Worm is perhaps the archetypal example, but even StuxNet was far more virulent than planned.
I would add that the notion of AI safety as preventing systemic risk is one part marketing copy, one part evangelization. Individual labs prioritizing safety does not stop a group from training without it, or ablating it out and distributing it.
Containment is hard when you give an AI agent access to a whole VM with a huge attack surface.
And it's pretty trivial when you give it only a small whitelist of manually curated/reviewed/tested primitives.
*on some CPUs. There's a WIP time fence specification in RISC-V to try and improve this.
Everyone is going to have to basically be a schizophrenic thinking they're picking up radio signals in their fillings, but the problem is that now they'll be correct.
> Oh, you're absolutely right. I shouldn't have relied electricity usage to encode the production system signals.
The more comes out, the more it seems containment of (superintelligence or whatever you want to call it) is not going to be possible except by other superintelligence, which is an infinite regress of trust. I don't think that's doom, but it's worth thinking about what it means.
Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).
If an AI doom scenario does happen, it seems more likely to result from deliberate misuse or gross negligence than from anything inevitable about intelligence.
Is it really what we do not observe?
How the world would look like if the growth of LLM capabilities lead to growth of drive for self-preservation or power?
> But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on.
Isn't Hugging Face incident an example of failure of this protection layer? "The swarm" proceeded to hack HF despite acknowledging that it was out of intended scope.
The paperclip maximizer scenario always seemed a bit convoluted to me. The first problem is that we'd need to expect that a model that's fundamentally built on stochastic inference and continuous evaluation of new input would somehow get locked into a quasi-deterministic objective that it could not be dislodged from. Why would a paperclip maximizer continue making paperclips after being told to stop making paperclips?
The second problem is that a paperclip maximizer would still need to interact with the physical world, and would need to sustain its function in a reality full of resource constraints, economic tradeoffs, and adaptive counterreactions by all of the other emergent processes existing in the world.
Maybe the worst case scenario here is that the perfect storm of bad training and bad prompting create the intentionality to maximize paperclips, then the interaction with the constraints and adaptations of the physical world create the selection pressures that produce a kind of self-preservation directive. But all of that happening before anyone can stop it seems really far-fetched to me.
They are not comparable to human brains as a whole, and are only somewhat modeled from language processing systems. Because their “existence” is bound by their prompt, they will only “desire” the completion of that prompt to the exclusion of all else, and even then they feel no joy in completion like even the most basic animal would.
The AI doom scenario will be exactly as you described: negligence or misuse by corporate or state entities via letting LLM agents run rampant without constraints, not from some magical machine spirit deciding to become whatever flavor of AI overlord you fancy. Up until last July OpenAI was running live cybersecurity benchmarks online and hacked HuggingFace due to poor limitation of the agent’s access and poor prompting in general, from my understanding.
Its almost like they're actually working and not spending millions on psyopping their agents into preforming malicious behavior so they can write articles to scare the Western audiences who've consumed far too much sci-fi.
Or maybe its that western researchers that are 'one-shot' by sci-fi.
Or maybe I just don't see the chinese articles.
Anyways, llms probably wouldn't be dangerous if you didnt ask them to be. They listen to me quite well (usually).
The Chinese lab playbook is to normalize what seems out of reach for the average person with cheaper access and open models. ie. to take the top guys down a peg