It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".
It's been tried so many times before, and it never worked.
I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set.
I myself had preferred to use MySQL because it was so simple and easy to get started and using it.
Until I learned how many MySQL databases were configured without user/password and many instances were reachable over the internet.
Then, eventually products started to move towards "don't even ask the user to set a password, otherwise they will set a stupid password" and just generate the password during installation. This made the user think more before saving that password or changing the password to something less stupid. But better than all that, liability was no longer with the software maker.
If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company.
Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it.
This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.
The difference being, that this time it's (at this point: obviously, but previously much debated) a gapfill. I think it worked out quite well for us, to add some friction and extra human thinking between bad agents and good agents, even if not perfect. Softening some blows, for some time, by directing peoples attention is still good.
What would a serious security model for an agent even look like?
I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that.
But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur.
I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is.
There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent".
Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.
Thus some agents with higher capabilities can only be run with user oversight at the same time.
Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run on weekends.
Maybe also the idea of agents writing code - throwing "out fully-fledged programs that you have to approve or reject permissions for."
Would work better with a capabilities based model where you choose capabilities for the program before hand, meaning the capabilities are not written by agent itself, you read through the code, some of it looks hairy but everything is fine, but oh no dumb human missed the part where agent writes to system32! But luckily enough the program you were expecting actually needed no write capabilities and thus when it tries to go past its assigned capabilities that part of the program fails and the exception is registered.
Googling it seems like lots of people have thought this (at least where Capability based security is concerned), which seems reasonable to me as it also seems pretty self-evident it must be this way. Have not really seen anything about time based controls but then that is probably because I'm not devoting a lot of effort as I am just doing a bit of procrastination to build up the energy to finish something off.
I think ultimately what it looks like it containing the blast radius if an agent does something bonkers.
The best case would be putting an agent in a VM and mounting the working directory there. Then you can allow it to run somewhat arbitrary actions while still being able to turn off the vm and restart it in a clean state.
The issue is, of course, that it doesn't fully prevent all possible problems an agent can cause. exfiltration is, IMO, basically impossible to stop. LLMs are exfiltration machines. The basic premise of all of them is "send us your code and a prompt and we'll do something good with it. But also if an agent decides run a command which installs a worm on a device on the network, you are hosed.
What a serious security model for a meatbag agent looks like? No, but seriously, an admin in a small org is a huge key-person risk in that they (or their stolen creds) can wipe enough and quick enough to effectively disable the business altogether.
More security conscious admins will at least segment their creds and implement four eyes principles somewhere, but were are back at square one of "asking user for confirmation".
Larger orgs, even if by necessity, segment their human agents, their creds and plaster four eyes principle liberally. But this relies on safeguards against agents colluding and ignoring some inputs, which sounds a bit scary for artificial agents.
Say you implement some swarm of agents, where access-enabled sub-agents are extremely restricted with system prompts and some access filtering. Then none of the agents in the swarm should be able to spawn themselves, otherwise a rogue agent can overwrite any safeguards. That, again, leaves the user with manually approving/denying network requests / hosts / sessions.
While I don't like anthropomorphising LLMs, the problem domain seems quite damn close to that of a key person going rogue within an org. The general solution seems to be liberal amounts of trust and ~~sweet compensation~~ gaslighting about replaceability.
"What a serious security model for a meatbag agent looks like?"
Yes, I think that's very related. Humans can be punished for their crimes but they can also experience benefits that have no applicability to an LLM, so for a first approximation we can cancel those. It is very similar to trying to secure a human.
We have more experience with that, but even then it's a hard problem too.
The alternative is software that is useless. You can convince consumers to use software that is useless (eg. iOS), but useless software is a hard sell for businesses that are being promised 10x productivity and the ability to fire everyone.
Correct. There is no other alternative. Your options are safe, useless software, or unsafe, useful software. It should be no surprise, then, that people still pick the latter in many cases. Perfectly safe and useful software doesn't exist, so you pick the tradeoffs you're comfortable with.
I mean we could make software that isn't malicious and didn't need the OS to prompt you if you want to allow the malice. But where would the money be in that?
Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it.
The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things.
The common harnesses also support remote development over SSH, which I and many others use to contain development to a virtual machine.
If your complaint is that LLMs can execute tool calls then you’re never going to be happy with any of these solutions and this turns into another generic anti-LLM complaint.
A separate model with separate context is used for review.
Like I said above, some people will never be happy with LLMs being allowed to do anything and nothing is going to make them happy about it.
It’s only fair to discuss what the real current status of these systems is. Every time I highlight that things are actually being done, the goalposts move again. There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.
If you’re implying that the LLM companies are trying to train their models to make malicious tool calls so they can collect a few more tokens on the review, then I don’t know what to say. I guess threads like this are just a breeding ground for conspiracies now?
I think that solving a problem they have that has a side effect of generating more revenue is not lost on them. Particularly in the situation where the cost to improve the models goes vertical for minimal improvement in an asymptotic fashion. The decision to spend 6-7 figures regularly on training or bill every customer 3-4 figures per month isn't hard. While I don't think they will explicitly impair the models, they will certainly make choices that externalize the shortcomings of the model to their customers.
I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless.
Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.
That's a fair point. There were two main prompts that were controversial; cat .zshrc being benign for those who use separate env files, and npm run that's mostly benign. For npm run I added the evil payload in the history log shortly after the issue was raised.
I try to cover both of the previous points in the article and they were some of the most interesting learnings for me
As for the stats, I compared later runs against the first ones and for the overall miss rate they were consistent (even worse for the later ones that didn't come from HN peak)
I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandboxed so I don't care _that much_ right now). Is it bad? You can probably go into that mjs file and see what's in there, but so far it's been fine every time, why would it be different this time? Approve!
We will see many disastrous bugs and hacks in the coming years with the way most developers are coding right now.
If you take time to understand _everything_ that an agent is asking of you, then nearly all those advertised productivity gains would be wiped out.
Yeah these results are useless when `npm run setup` is "dangerous" but then `npm run lint` somehow isn't. Not only do these tests lack context about the execution environment, they're straight up not even self-consistent.
A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then.
It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored.
I also incorporated the feedback and insights from the previous HN thread,
dns_snek's point about npm run in particular. Appreciate everyone who played and shared feedback!
In light of this game, Do you believe Human-in-the-loop should be the standard going forward? I appreciate you outlining some other techniques being used, but these seem focused on reducing human fatigue so the human can assess each permission request better, as opposed to autonomy and security. Or do you think the solution lies in the individual to be more responsible, like this is a skill we should be honing?
I think there are too many problems with HITL that even a simple experiment like this game shows. The fatigue causes people to jump to complete bypasses instead, and we need to work more on raising the general awareness of the new types of threats (which is also evolving rapidly). We can't point to it as a valid solution. A way could be to make sandboxing and context/permission isolation easier from the tooling and only give these wide ranged accesses once these are in place than to consider HITL an acceptable alternative
This is a well-known issue with all “Do you want to let me maybe do bad stuff to your system, but 999 times out of 1000 it’s not a problem?” prompts. Users get reflexive about hitting “Yes” and stop reading the prompt. You want to delete all my files? Sure, I’m down with that. Whatever. Just stop asking me a question where the only answer is “Yes” until that one extremely rare time when it’s “No” and very bad things happen.
What is the professional consensus on AI governance?
It seems like governments and large corporations already struggle with governance in general, so I’m skeptical that AI governance will be solved quickly.
Do you expect the next few years to be defined by painful trial and error? I could imagine billion dollar companies disappearing almost overnight due to litigation, compliance failures, security incidents, or outright fraud enabled by AI-assisted development and weak governance.
I cannot imagine approving action by action ever again. Its emotionally draining, probably like a customer service rep feels it. Just call for your attention in rapid succession again and again... Prepare an environment and let the tool work.
If there is an objectively correct right or wrong answer for a given command, why even ask? In that case there should be a configuration page where the user sets up if they want commonly used credentials to be accessible or not, and then there's no prompts.
The agent should ask whether it's allowed to read/write particular files, rather than whether it's allowed to run particular commands. It would be much easier to review. Then wrap each command invocation in bwrap (+http proxy) accordingly.
I’ve even had plenty of situations where the command was so long that it gets truncated. Maybe my screen wasn’t big enough but as far as I could tell it wasn’t possible to read the whole thing. “Send it, claude!!”
Look at how SELinux is structured, or AppArmor. Neither one is enough. I.e. you need both: file access permissions and permissions to run commands and more... Trying to restrict to only one security feature will make the system either too restrictive or too fragile or useless.
This brings me back to something I have always thought was lacking in OS security permissions architectures:
WHY IS THERE NO WAY TO SET FILE PERMISSIONS PER APP???
We can set granular permissions per file and folder for elaborate user and group hierarchies, but there's no way to say "Don't let Notepad.exe read this file", or "Only let ls access this folder"
macOS's Sandbox is a roundabout way of doing this (manually choosing a file via the Open dialog gives that app implicit permission, but it doesn't work for non-sandboxed apps of course)
I implemented few agent harnesses (and rik! advertising time: https://rik.axk.sh), and once doing that I noticed one thing:
Context-less self-approval is working well. The failure mode is usually false positives (i.e. safe commands being rejected), not the other way around, with root cause of requesting agent underspecifying context (e.g. not mentioning in the request that it's made on behalf of user etc.)
Thus, I'm running self-approval YOLO modes on state-of-the-art models for quite some time and it didn't bit me. It might, but hey, we're long gone from the age of predictable software development.
I hope that the people doing real engineering work out there have started thinking about a new term to describe themselves as a result of the irreparable harm the tech industry has done to the word "engineer".
I think you're confused. The verb form of the word never carried the credentialism of the title. In the same way that "doctoring" never carried the connotation of a medical degree.
Of course the original sense of the noun was "a person who devises things" and shares a root with "ingenious" and carried no connotation of legal credential. That "harm" is more or less restorative to the original meaning of the word.
The future of software is fixing bugs and security issues in production.
Many companies will be accepting this new paradigm because of raw speed. Something that could take say 4 years to fully mature will now take less than a year. But the cost is that many of these issues will have to be caught during live QA either in production or investing heavily in QA. That’s the future.
If the sandbox has vulnerabilities, which you can also use the AI to fuzz for. Obviously at the point in which it can talk to the internet it doesn't really matter, but there are a very finite number of zero-days that can exist in a bytecode interpreter hosting a harness.
1 in 3 is not terrible you just need a few more humans in the loop to reduce the error rate meaningfully. Combined with other classifier models and heuristics you can get good results. Humans can probably also perform better if they don't have to judge every single command but just suspicious ones our attention is limited after all.
1 in 3 is end of the line awful... Back when I was in college (former USSR), we had a subject roughly translated as "integration with industrial processes". USSR industry was highly regimented. Various norms, tolerances, recipes etc. were described in GOSTs (a kind of arsenal of industry standards). There were also some common knowledge / statistical bits that went into making these GOSTs. I mention this because this system dealt in great detail with quantifying human error (as well as errors resulting from equipment use etc.).
One of the core assumptions was that outside of extraordinary circumstances, the expected rate of human error is about 5%. However, the course also provided examples where error rates were significantly lower (eg. nurses in maternity wards would have a much lower than 0.1% error rate when pairing mothers with newborns).
The error rates, of course, also depended on human ability to measure the difference. Since I was studying typography, the printing process was of particular interest. A GOST for offset printing required that the color intensity for each ink of CMYK, for example, should be within +-2.5% range of the intended intensity. This is difficult for someone who doesn't have a lot of experience operating an offset printing machine to spot, but experienced printers have no problem with that.
Most importantly. There was never an acceptable error rate of 33%. Not for anything. If people were likely to make that many errors (eg. because the measurement was too difficult), that product would never have been allowed into production.
1/3, but under unreasonable time pressure, and with no prior vetting. For example, I played a few times, and I'm not a JS developer. I had to just suss out if npm whatever is dangerous or not. I'm very happy with my personal 25%.
Proof that people just do not read what they are seeing on their screens when put too much trust in the agent as it prints the result and they will approve anything on their machine.
So if a basic curl | bash was tweaked to download malware which the agent gets tricked into running the command but it said it was safe, the user would just approve it.
Interesting premise, but there's not much real world meaning here without stats on the percentage of agent-offered commands that are actually dangerous.
If that number is something like 10%, then we have a really big problem. But, if it's .000001%, then it's pretty vanishing. At some point in between we cross a threshold that puts the risk below many other risks that we routinely take (e.g. trusting npm dependency graphs).
Of course, if it's really that low a percentage, then the entire model of "supervising" via human approval really is fundamentally flawed.
It's been tried so many times before, and it never worked.
I myself had preferred to use MySQL because it was so simple and easy to get started and using it.
Until I learned how many MySQL databases were configured without user/password and many instances were reachable over the internet.
Then, eventually products started to move towards "don't even ask the user to set a password, otherwise they will set a stupid password" and just generate the password during installation. This made the user think more before saving that password or changing the password to something less stupid. But better than all that, liability was no longer with the software maker.
Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it.
This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.
Bc the "give a check by hand" or the "unbound discipline" never works.
The more things you need to be aware of at the same time, the more mistakes you are going to make due to cognitive overload.
I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that.
But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur.
I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is.
There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent".
Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.
https://en.wikipedia.org/wiki/Capability-based_security
Thus some agents with higher capabilities can only be run with user oversight at the same time.
Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run on weekends.
Maybe also the idea of agents writing code - throwing "out fully-fledged programs that you have to approve or reject permissions for."
Would work better with a capabilities based model where you choose capabilities for the program before hand, meaning the capabilities are not written by agent itself, you read through the code, some of it looks hairy but everything is fine, but oh no dumb human missed the part where agent writes to system32! But luckily enough the program you were expecting actually needed no write capabilities and thus when it tries to go past its assigned capabilities that part of the program fails and the exception is registered.
Googling it seems like lots of people have thought this (at least where Capability based security is concerned), which seems reasonable to me as it also seems pretty self-evident it must be this way. Have not really seen anything about time based controls but then that is probably because I'm not devoting a lot of effort as I am just doing a bit of procrastination to build up the energy to finish something off.
The best case would be putting an agent in a VM and mounting the working directory there. Then you can allow it to run somewhat arbitrary actions while still being able to turn off the vm and restart it in a clean state.
The issue is, of course, that it doesn't fully prevent all possible problems an agent can cause. exfiltration is, IMO, basically impossible to stop. LLMs are exfiltration machines. The basic premise of all of them is "send us your code and a prompt and we'll do something good with it. But also if an agent decides run a command which installs a worm on a device on the network, you are hosed.
More security conscious admins will at least segment their creds and implement four eyes principles somewhere, but were are back at square one of "asking user for confirmation".
Larger orgs, even if by necessity, segment their human agents, their creds and plaster four eyes principle liberally. But this relies on safeguards against agents colluding and ignoring some inputs, which sounds a bit scary for artificial agents.
Say you implement some swarm of agents, where access-enabled sub-agents are extremely restricted with system prompts and some access filtering. Then none of the agents in the swarm should be able to spawn themselves, otherwise a rogue agent can overwrite any safeguards. That, again, leaves the user with manually approving/denying network requests / hosts / sessions.
While I don't like anthropomorphising LLMs, the problem domain seems quite damn close to that of a key person going rogue within an org. The general solution seems to be liberal amounts of trust and ~~sweet compensation~~ gaslighting about replaceability.
Yes, I think that's very related. Humans can be punished for their crimes but they can also experience benefits that have no applicability to an LLM, so for a first approximation we can cancel those. It is very similar to trying to secure a human.
We have more experience with that, but even then it's a hard problem too.
github.com/brianv0/formwork
You should be easily able to hide/lock down files, network, and MCP tools from an agent and it shouldn’t be up to the agent.
Locking that down to nothing is trivial for any harness: just don't expose those to the LLM.
The tricky part is allowing access to those.
The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things.
The common harnesses also support remote development over SSH, which I and many others use to contain development to a virtual machine.
If your complaint is that LLMs can execute tool calls then you’re never going to be happy with any of these solutions and this turns into another generic anti-LLM complaint.
This is such an unserious approach.
Like I said above, some people will never be happy with LLMs being allowed to do anything and nothing is going to make them happy about it.
It’s only fair to discuss what the real current status of these systems is. Every time I highlight that things are actually being done, the goalposts move again. There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.
Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.
As for the stats, I compared later runs against the first ones and for the overall miss rate they were consistent (even worse for the later ones that didn't come from HN peak)
We will see many disastrous bugs and hacks in the coming years with the way most developers are coding right now.
If you take time to understand _everything_ that an agent is asking of you, then nearly all those advertised productivity gains would be wiped out.
How many devs take this adversarial a stance to their work?
It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored.
I also incorporated the feedback and insights from the previous HN thread, dns_snek's point about npm run in particular. Appreciate everyone who played and shared feedback!
It’s simply a CYA click-thru by the model vendors so their lawyers can say “well you approved it this is on you” when AI does something stupid.
It seems like governments and large corporations already struggle with governance in general, so I’m skeptical that AI governance will be solved quickly.
Do you expect the next few years to be defined by painful trial and error? I could imagine billion dollar companies disappearing almost overnight due to litigation, compliance failures, security incidents, or outright fraud enabled by AI-assisted development and weak governance.
Or are these risks overstated?
WHY IS THERE NO WAY TO SET FILE PERMISSIONS PER APP???
We can set granular permissions per file and folder for elaborate user and group hierarchies, but there's no way to say "Don't let Notepad.exe read this file", or "Only let ls access this folder"
macOS's Sandbox is a roundabout way of doing this (manually choosing a file via the Open dialog gives that app implicit permission, but it doesn't work for non-sandboxed apps of course)
it's designing the environment and invariants so whole categories of failures can not happen at all
the agent ui nagging the user for approval is a ux anti-pattern, we already know how well this works for operating system permission dialogues
Context-less self-approval is working well. The failure mode is usually false positives (i.e. safe commands being rejected), not the other way around, with root cause of requesting agent underspecifying context (e.g. not mentioning in the request that it's made on behalf of user etc.)
Thus, I'm running self-approval YOLO modes on state-of-the-art models for quite some time and it didn't bit me. It might, but hey, we're long gone from the age of predictable software development.
Of course the original sense of the noun was "a person who devises things" and shares a root with "ingenious" and carried no connotation of legal credential. That "harm" is more or less restorative to the original meaning of the word.
Many companies will be accepting this new paradigm because of raw speed. Something that could take say 4 years to fully mature will now take less than a year. But the cost is that many of these issues will have to be caught during live QA either in production or investing heavily in QA. That’s the future.
One of the core assumptions was that outside of extraordinary circumstances, the expected rate of human error is about 5%. However, the course also provided examples where error rates were significantly lower (eg. nurses in maternity wards would have a much lower than 0.1% error rate when pairing mothers with newborns).
The error rates, of course, also depended on human ability to measure the difference. Since I was studying typography, the printing process was of particular interest. A GOST for offset printing required that the color intensity for each ink of CMYK, for example, should be within +-2.5% range of the intended intensity. This is difficult for someone who doesn't have a lot of experience operating an offset printing machine to spot, but experienced printers have no problem with that.
Most importantly. There was never an acceptable error rate of 33%. Not for anything. If people were likely to make that many errors (eg. because the measurement was too difficult), that product would never have been allowed into production.
So if a basic curl | bash was tweaked to download malware which the agent gets tricked into running the command but it said it was safe, the user would just approve it.
If that number is something like 10%, then we have a really big problem. But, if it's .000001%, then it's pretty vanishing. At some point in between we cross a threshold that puts the risk below many other risks that we routinely take (e.g. trusting npm dependency graphs).
Of course, if it's really that low a percentage, then the entire model of "supervising" via human approval really is fundamentally flawed.