"He looked at me incredulously. Surely, a smart person like me should know that AI, or better said, AGI will be hyperpersuasive soon – already on a bunch of benchmarks it exceeds professional debaters at persuasion." I'm always tickled by this tic of this kind of person, that human traits are something that we can maximize to infinity. That if we trained a robot to tell jokes, it would eventually tell jokes so funny we'd die laughing etc. You can't convince someone who doesn't want to be convinced.
This has been a core belief of the EA/LessWrong/Rationalist people for many years - that above a certain level of intelligence you effectively get hypnotic powers, and people will obey AIs no matter what. So they said you can't keep a superpowered AI airgapped as merely offering a chat interface will immediately take control of the next person to interact with it and they will deliberately compromise the airgapping.
A complete mischaracterization, as usual for HN lately when discussing AI or LessWrong. Obviously, even average levels of persuasion are enough to convince some people. And nobody is air-gapping AI.
And if AI wrote a book about these hypnotic powers, then it would be incredibly convoluted, long, and yet worn as a badge of pride to have survived it.
Thinking about it, much of the output I'm seeing nowadays looks like a chapter out of infinite jest.
In my work, I have always found that the main "blocker" (I hesitate to use the term "bottleneck"), is human nature.
If we design stuff to be used by humans (what I generally do), then it's imperative to take basic human nature into account, and that can be incredibly annoying.
Annoying, or not, human nature is a major factor, and won't go away, no matter how many times we tap our heels together. I think it was Philip K. Dick that said "Reality is that which, when you stop believing in it, doesn’t go away."
Designing stuff for processes and machines to consume, is pretty straightforward. Designing stuff for humans, however, is really challenging (which is one of the reasons that I like doing it).
I would go further and say that the human should be the bottleneck/blocker. If not designed for and to serve humans, which means to have human in the position of power and control and take in account all the various human oddities, why are we building all of this?
That depends on the bottleneck. When the bottleneck to me getting something I want is other humans won't allow me...
Of course this goes both ways. Some things I could name here everyone would agree I shouldn't be allowed to do. Other things are things that it wouldn't be hard to convince everyone here (but the opposition would remind you that I'm ignoring their points)...
One thing you have to be careful of, mostly when working on internal tools, is that you don't end up building around the idiosyncrasies of one particular person. This is something that actually happens in some places.
I actually find "bottleneck" a nicer word than "blocker" here (although not sure if "nicer" is what you are going for). I really like building for humans and the only times I've ever truly felt excited about being a developer is when I've had direct contact with my customers. It can be very trying, but the payoff is always nice.
Anything digital basically has the chaos fuzzed out by its binary nature.
The analog gets all the real entropy.
Humans are cool because our distribution is exploring the future curve of what is possible in a hostile universe and to do that successfully over time we have to have the humility to always have someone unsure of even our most basic facts.
Sucks for most of those edge distribution humans but good for the species.
But this is why our criteria of reasonable is aligned with the center of distribution and why we have colloquial statements like "if you design for everyone you design for no one".
Well, let me just throw in it is our communication skills, or lack of them, that is our main bottleneck. That goes for pretty much everything humans do, and everything we fail trying to do: it is the quality of those engaged and their communications that is the critical aspect determining success or failure.
Alan Kay said "A change in perspective is worth 80 IQ points,".
A change in perspective is often brought about by communication or I/O. In my experience this often bears out. The people who communicate get more done than the clever person who doesn't get out into the problem domain. The limit is I/O not intelligence.
You sort of side stepped the author's point, though.
> In medicine you have the modern labs which automate a lot and you have the old labs. These 'neolabs' are better because of intelligence.
The author is not denying that a lab can be more effective with intelligence, they're arguing that this entire phase of the process of improving health is not the bottleneck.
Indeed - if access to data or the permission to act on that data is limited by human gatekeepers (individual or bureaucratic), intelligence is often not the primary bottleneck.
OP points out that this is usually the case in biology/medicine where regulatory bottlenecks dominate.
> Some have been waiting for a year for the NIH to release imaging datasets they can use to produce better biomarkers. If one needs to interact with the FDA to get their endpoint validated, it is even worse: I have previously written about how the validation of Bone Mineral Density (BMD) for use as a surrogate endpoint in osteoporosis trials took 12 years (!), despite the fact that the data to support it already existed in full and the analyses done were basically regressions.
> no matter how “intelligent” AI becomes, intelligence is often not the main bottleneck to things changing in the real world
A mathematician Terrence Tao said something similar in a recent talk [1]. That, intelligence is bottlenecked by human understanding. Say there is an alien that brings an alien book containing math humans havent seen before. Using said math is bottlenecked by humans comprehending it first.
I think the next bottleneck is embodiment. AI can lay the best laid plans, but who will execute it?
I think a possible solution to the dual problems of AI embodiment and unemployment is humans becoming avatars for AI.
Imagine one of the unskilled masses donning a headset and being guided through all manner of work?
It‘s actually stunning how inhumane this suggestion is. Especially, in the way it‘s made in the purely functional sense of human value as means of productivity and efficiency.
A human with experience can do without AI the job a lot faster than someone without can do it with. The expert HVAC tech can get the wrench of the fitting and have the job done faster than the novice. The novice will take longer to follow directions to get the right wrench; then they have to figure out how to get it on - something the expert does automatically isn't easy; then turn it the right direction... The expert also knows the correct "feel" so they won't over tighten, while still having less leaks to fix (both will check, but the expert will find less)
If I'm in my remote cabin I'm still faster than calling a tech, but in my city I can hire someone to do this who is faster. (though perhaps not cheaper)
In my city I call a handyman to fix my dryer and guess what? He uses chatGPT to help him through the issue. $350
Beyond a basic amount of manual dexterity and familiarity with mechanical things, AI can absolutely help guide an untrained person through the job.
How about medical research? Endless hours of pipetting. Does it really take a phd? It takes someone with conscientiousness and care. Both things an AI overlord would help with.
Robots being able to perform any task a human can do, physically, doesn't seem that far off. They would be powered by AI of course.
The question is: do we still have to work? If not, who will take all the wealth?
We thought the washing machine would have giving us so much free time, because it used to take a full day a week. Instead, we only got more frantic it seems and started putting in more labor. But what will happen when that labor is worthless?
In a dystopian world, the elite may be better off just killing the lower, middle and even the higher classes. Keep just enough people to have an interesting gene pool around. Because even in the space mines there will be no need for human labor anymore.
> new drugs approved per dollar of R&D has fallen for decades, even as our scientific tools have grown vastly more powerful, the exact opposite of what the existence of more raw capability would predict
Can we please stop using X per dollar as a metric over long periods of time?
Inflation causes exponential price increases, so almost everything would be expected to have its "per dollar" yield decrease over time...
I think the guy is right on point at least when it comes to therapies. But he is also missing one major angle: we are no longer in control; our systems are. If we were to vote on the issue, I suspect that a narrow majority would prefer cheaper housing, and a total majority would prefer cancer to be cured. But there's no direct mechanism of any kind to exercise that vote. Not in USA, not in Europe, not in China. When/where there's democracy, nobody runs in a campaign of advancing immune therapies[^1]. The closest proxy we have to a vote is "regulated markets", and my recent experience losing a loved one to cancer taught me that cancer is not a disease for the streams of money moving through that system. Far from it.
I'm frankly pessimist that we will be able to fix those systems in a humanist way, at least not in my lifetime. If we were any good at it, we wouldn't have extreme inequality, nor our educations systems would be in tatters, and we wouldn't cast ballots propping up politicians whose best skill is ragebait.
However, if fixing the system is not an option, access to cheap technology and intelligence could help a lucky few circumvent those systems. And guess what the lucky few at the head of the AI companies will say then... Not necessarily the worst outcome, since most people will find more viable to equip a private biolab in the basement of their countryside manor than fixing the medical system of first world economies.
[^1] For the record, I think that would be better than running on xenophobic slogans.
Essence of Decision is a great book that explores this idea. It looks at different models for understanding how organizations make decisions, and arrives at broadly the same conclusion: a model that assumes people are particularly in control is inadequate for explaining real world events.
I could build a house. With zero experience you can find a job in house building as they will pay you to learn on the job (depending on the economy you might or might not be able to find such a job today, in good years they will hire anyone who isn't disabled - but there are bad years). Many people who have never had a construction job build their own house (only a tiny %, but that is still a lot of people)
I suppose I could work on a cancer cure. However since I don't have a background in biology I would have to pay for several years of school to learn the required skills. Those classes are hard and there is no guarantee I will pass (I have an engineering degree, so my odds of passing should be above average).
Two possible assumptions here are that systems are actually in control and that there isn't anyone looking, while we saw that if things get too heated the US govt can sweep in and shut things down. The reasons for it might be debatable -- I'm more inclined to believe that that actually played nicely into the hype train launched by Anthropic around Mythos -- but there is always a bigger dog watching.
The other assumption is that these systems are failing by accident when in fact a stronger argument can be made that they are like this by design.
> In biopharma specifically, one great bottleneck is clinical trials
It is a good news for having more intelligence. Phase 1 trial to acceptance rate is ~10%. If say in ideal world, intelligent AI could make it to 80% by rejecting drugs before Phase 1, cost of trial(which author claims is the bottleneck) would suddenly drop to 1/8th even if AI doesn't help at all in the trial.
Thinking about it, much of the output I'm seeing nowadays looks like a chapter out of infinite jest.
Maybe that's unfair on infinite jest.
If we design stuff to be used by humans (what I generally do), then it's imperative to take basic human nature into account, and that can be incredibly annoying.
Annoying, or not, human nature is a major factor, and won't go away, no matter how many times we tap our heels together. I think it was Philip K. Dick that said "Reality is that which, when you stop believing in it, doesn’t go away."
Designing stuff for processes and machines to consume, is pretty straightforward. Designing stuff for humans, however, is really challenging (which is one of the reasons that I like doing it).
Of course this goes both ways. Some things I could name here everyone would agree I shouldn't be allowed to do. Other things are things that it wouldn't be hard to convince everyone here (but the opposition would remind you that I'm ignoring their points)...
One thing you have to be careful of, mostly when working on internal tools, is that you don't end up building around the idiosyncrasies of one particular person. This is something that actually happens in some places.
I actually find "bottleneck" a nicer word than "blocker" here (although not sure if "nicer" is what you are going for). I really like building for humans and the only times I've ever truly felt excited about being a developer is when I've had direct contact with my customers. It can be very trying, but the payoff is always nice.
The analog gets all the real entropy.
Humans are cool because our distribution is exploring the future curve of what is possible in a hostile universe and to do that successfully over time we have to have the humility to always have someone unsure of even our most basic facts.
Sucks for most of those edge distribution humans but good for the species.
But this is why our criteria of reasonable is aligned with the center of distribution and why we have colloquial statements like "if you design for everyone you design for no one".
I mean to say: Yes it's certainly a factor, but we're also leaving fences in our own way.
https://en.wikipedia.org/wiki/Conway's_law
An expert in physics still needs to write code in python to do their math and analysis. Its an intelligence issue. That person needs to learn a tool.
In medicin you have the modern labs which automate a lot and you have the old labs. These 'neolabs' are better because of intelligence.
A doctor doing a diagnosis can say 'no clue' or can give you a proper answer. The more intelligent this person is, the better the answer.
> In medicine you have the modern labs which automate a lot and you have the old labs. These 'neolabs' are better because of intelligence.
The author is not denying that a lab can be more effective with intelligence, they're arguing that this entire phase of the process of improving health is not the bottleneck.
OP points out that this is usually the case in biology/medicine where regulatory bottlenecks dominate.
> Some have been waiting for a year for the NIH to release imaging datasets they can use to produce better biomarkers. If one needs to interact with the FDA to get their endpoint validated, it is even worse: I have previously written about how the validation of Bone Mineral Density (BMD) for use as a surrogate endpoint in osteoporosis trials took 12 years (!), despite the fact that the data to support it already existed in full and the analyses done were basically regressions.
A mathematician Terrence Tao said something similar in a recent talk [1]. That, intelligence is bottlenecked by human understanding. Say there is an alien that brings an alien book containing math humans havent seen before. Using said math is bottlenecked by humans comprehending it first.
[1] https://teorth.github.io/tao-web/slides/age-of-ai-icm-2026.p...
I think a possible solution to the dual problems of AI embodiment and unemployment is humans becoming avatars for AI. Imagine one of the unskilled masses donning a headset and being guided through all manner of work?
Roofing, ACHVAC, stocking shelves, etc…
No need to train anybody.
Maybe I‘m missing the joke though.
Don’t mistake my ill portent for enthusiasm. I merely foretell.
If I'm in my remote cabin I'm still faster than calling a tech, but in my city I can hire someone to do this who is faster. (though perhaps not cheaper)
Beyond a basic amount of manual dexterity and familiarity with mechanical things, AI can absolutely help guide an untrained person through the job.
How about medical research? Endless hours of pipetting. Does it really take a phd? It takes someone with conscientiousness and care. Both things an AI overlord would help with.
No need to imagine nor AI for this, it can be either JSON guides on AR, e.g. Daqri for more than a decade, or remote work in VR (6 years ago or more https://www.inceptivemind.com/remote-controlled-model-t-robo... )
The question is: do we still have to work? If not, who will take all the wealth?
We thought the washing machine would have giving us so much free time, because it used to take a full day a week. Instead, we only got more frantic it seems and started putting in more labor. But what will happen when that labor is worthless?
In a dystopian world, the elite may be better off just killing the lower, middle and even the higher classes. Keep just enough people to have an interesting gene pool around. Because even in the space mines there will be no need for human labor anymore.
AI?
Can we please stop using X per dollar as a metric over long periods of time?
Inflation causes exponential price increases, so almost everything would be expected to have its "per dollar" yield decrease over time...
And that has been true for centuries, not just since AI.
Allowing intelligence means ceding control over the outcome. And that is often a bigger issue than the actual issue.
I'm frankly pessimist that we will be able to fix those systems in a humanist way, at least not in my lifetime. If we were any good at it, we wouldn't have extreme inequality, nor our educations systems would be in tatters, and we wouldn't cast ballots propping up politicians whose best skill is ragebait.
However, if fixing the system is not an option, access to cheap technology and intelligence could help a lucky few circumvent those systems. And guess what the lucky few at the head of the AI companies will say then... Not necessarily the worst outcome, since most people will find more viable to equip a private biolab in the basement of their countryside manor than fixing the medical system of first world economies.
[^1] For the record, I think that would be better than running on xenophobic slogans.
I suppose I could work on a cancer cure. However since I don't have a background in biology I would have to pay for several years of school to learn the required skills. Those classes are hard and there is no guarantee I will pass (I have an engineering degree, so my odds of passing should be above average).
The other assumption is that these systems are failing by accident when in fact a stronger argument can be made that they are like this by design.
It is a good news for having more intelligence. Phase 1 trial to acceptance rate is ~10%. If say in ideal world, intelligent AI could make it to 80% by rejecting drugs before Phase 1, cost of trial(which author claims is the bottleneck) would suddenly drop to 1/8th even if AI doesn't help at all in the trial.