"...but it’s so incredibly important to point out that this is a destruction myth. It’s apocalypticism crowdwriting its own biography."
YES! It's not simply the belief, but all the activity devoted to predicting when AGI will happen, the surveys and the studies with the Rube Goldberg reasoning over order-of-magnitude guesstimates. This is epistemic theatre. And the millions of dollars being poured into this activity. This is cargo cult behavior. There may not be a Jim Jones, or a jungle compound, much less toxic fruit punch, but this is a high-tech millennial cult. And it's all be conducted in the name of rationality.
It's possible for a phenomenon to both exhibit the characteristics you describe and also be actually dangerous. Let's hope your implicit claim that it's only the former is correct.
People have predicted bad things/apocalypses in the last, they didn't happen, so AI is fine?
The core arguments of those who are concerned about AI isn't that "something could go very wrong", it's that:
1) Alignment is extremely difficult, even in principle (of which there are many, many extensive arguments for, not least by MIRI)
2) We have no reliable method of telling when we cross over the threshold to where it could be dangerous, thereby making it too late for troubleshooting
The above doesn't seem to have any specific counterarguments to those concerns.
I'm not even personally an AGI guy (for what it's worth, my donations go to global poverty/health), but the arguments are much stronger than you present them, and worth addressing directly.
It's closer to "The seeming inevitability of a destination shouldn’t seduce us into becoming fatalistic. The journey moulds the destination. "
The two points above presume the conclusion. I'd rather see us actually try and fix problems in the creation and execution of AI today, of which there are plenty, before assuming the difficulties of aligning an eventual super AI.
Yes, and that's precisely an assumption were making about the conclusion of our work on AI. I'm happy to consider it, the way we consider gain of research method in bio, but to do that we need to actually know what we're talking about far better.
I’m not sure what assumption you’re referring to. That AI could be smarter than us?
If so, that’s why I phrased my question like I did - *if* AI were smarter than us, would you agree the problems would be worse?
Gain-of-function research is a good example - there’s no proof of it killing people yet, but we have reason to think that it could be extremely dangerous.
No. That we know what "more intelligent than us like we are to chimps" line means, that it can be further extrapolated, and most importantly that we can somehow imagine the characteristics of such an extrapolated being.
And yep that's the point. We couldn't have seen how to do it safely without actually understanding what it took. There's no way hain of function research made sense before we had the germ theory of disease.
Ok, that's interesting - I think we're getting somewhere, but I'm still not sure what the objection is.
1. "More intelligent than us like we are to chimps" - is the 'chimps' part important there? I mean, would you say that "more intelligent" than us is unclear, or impossible, or just that a smart AI wouldn't have the same kind of qualitative advantage we have over chimps?
2. "That it can be further extrapolated" - this makes me think you're making one of the first two claims , but I could be wrong.
3. "Most importantly that we can somehow imagine the characteristics of such an extrapolated being" - why is this part important? I don't think that AI x-risk people think they're good at imagining what AGI would be like - they just know what very smart people could do, and worry about AIs that could potentially do the same or more. If you think AGI might just have a well-developed morality by virtue of being smart, I'd point you to Eliezer's stuff on the orthogonality thesis, which I found pretty convincing.
Re: "We couldn't have seen how to do it safely without actually understanding what it took. There's no way gain of function research made sense before we had the germ theory of disease": the analogy to AGI is that people are worried we'll know how to build something smart before we have adequate theories about *safety*. There's no contradiction there.
“In my own psychiatric practice, I am always very reluctant to assume a patient is projecting unless I know them very well. I’ve written more about the dangers of defense mechanism narratives here, but the short version is that amateur therapists inevitably end up using them to trivialize or psychologize a patient’s real concerns. I can’t tell you how many morons hear a patient say “I think my husband hates our kids”, give some kind of galaxy-brain level interpretation like “Maybe what’s really going on is you unconsciously hate your kids, but it’s more comfortable for you to imagine this of your husband”, and then get absolutely shocked when the husband turns out to be abusing the kids.
Accusing an entire region of California of projection is a novel psychoanalytic manuever, and I’m not sure Chiang and Buzzfeed give it the caution it deserves. The problem isn’t that they don’t have a plausible-sounding argument. The problem is that this sort of hunting-for-resemblances is a known bug in the human brain. You can do it to anything, and it will always generate a plausible-sounding argument.”
It’s frustrating, as someone who’s on the periphery of AI safety work, to see people come in and dismiss this whole field of possibly *critically important* research with, “But did you know religions talk about the apocalypse?” And Rohit’s proposed obvious “solutions” look unworkable to most people who’ve spent time thinking seriously about the difficulties involved.
Tl;dr: analogizing AGI concerns to eschatology don’t tell us anything about whether AGI could actually kill people. If you want to know who’s right, you can’t just take a shortcut through the whole debate by comparing it to religion.
Thanks for this. I read Scott's article as an argument on why arguments from analogies sometimes fail, while his arguments for AGI fall better in the superintelligence faq which also does effectively the same. Both of them seem supporting to my thesis here that much of the xrisk conversation seems the same - arguing from analogies with the same failure modes.
My issue is that this line of extrapolation is assuming a huge amount, including trendlines, scaling, various theories of morality and intelligence, none of which are either testable or provable. Which to me suggests a highly under specified problem; and the comparison to eschatology is in comparing the types of worry I see and the limited levels of evidence against it.
So…I don’t think the argument for AI x-risk is assuming any of these things. People have gone back and forth quite a lot about each one, and each has been argued for. Are you more skeptical that we’ll likely get very capable AI systems in the next 50 years, or that they’ll be hard to align?
I think worries about brittleness/untestability of claims are totally fair, but there has been quite a lot of rigor applied to arguments on both sides despite that constraint. I’m pretty persuaded of the core case - we don’t know how to align AI and if it’s smarter than us that could be catastrophic - and I didn’t see much engagement with the reasons people believe that. Particularly the last few paragraphs treated AI concerns as fanciful and struck me more as psychoanalysis than argument.
I think we’re just in a very bad situation, where we have reasons to believe AGI is coming (many of them data-driven but speculative) and reasons to believe that could be dangerous (outsmarting humans for its own reasons seems bad for us) but no knock-down empirical argument that’s going to convince everyone. Though I do think that of the people who engage with the args, most are convinced (e.g. Holden aggressively criticizing MIRI in 2012 and getting gradually more sympathetic until now he’s very in favor of OpenPhil funding AI safety).
"...but it’s so incredibly important to point out that this is a destruction myth. It’s apocalypticism crowdwriting its own biography."
YES! It's not simply the belief, but all the activity devoted to predicting when AGI will happen, the surveys and the studies with the Rube Goldberg reasoning over order-of-magnitude guesstimates. This is epistemic theatre. And the millions of dollars being poured into this activity. This is cargo cult behavior. There may not be a Jim Jones, or a jungle compound, much less toxic fruit punch, but this is a high-tech millennial cult. And it's all be conducted in the name of rationality.
It's possible for a phenomenon to both exhibit the characteristics you describe and also be actually dangerous. Let's hope your implicit claim that it's only the former is correct.
It *could* be dangerous, sure, but when we don't know if or how it could be dangerous, we're making plans on the basis of no real information.
I'm not sure what the argument here is.
People have predicted bad things/apocalypses in the last, they didn't happen, so AI is fine?
The core arguments of those who are concerned about AI isn't that "something could go very wrong", it's that:
1) Alignment is extremely difficult, even in principle (of which there are many, many extensive arguments for, not least by MIRI)
2) We have no reliable method of telling when we cross over the threshold to where it could be dangerous, thereby making it too late for troubleshooting
The above doesn't seem to have any specific counterarguments to those concerns.
I'm not even personally an AGI guy (for what it's worth, my donations go to global poverty/health), but the arguments are much stronger than you present them, and worth addressing directly.
It's closer to "The seeming inevitability of a destination shouldn’t seduce us into becoming fatalistic. The journey moulds the destination. "
The two points above presume the conclusion. I'd rather see us actually try and fix problems in the creation and execution of AI today, of which there are plenty, before assuming the difficulties of aligning an eventual super AI.
They’re not presuming the conclusion, they’re arguing for it to be considered.
Wouldn’t you agree the problems with today’s AI would be much worse if the AI were smarter than us?
Yes, and that's precisely an assumption were making about the conclusion of our work on AI. I'm happy to consider it, the way we consider gain of research method in bio, but to do that we need to actually know what we're talking about far better.
I’m not sure what assumption you’re referring to. That AI could be smarter than us?
If so, that’s why I phrased my question like I did - *if* AI were smarter than us, would you agree the problems would be worse?
Gain-of-function research is a good example - there’s no proof of it killing people yet, but we have reason to think that it could be extremely dangerous.
No. That we know what "more intelligent than us like we are to chimps" line means, that it can be further extrapolated, and most importantly that we can somehow imagine the characteristics of such an extrapolated being.
And yep that's the point. We couldn't have seen how to do it safely without actually understanding what it took. There's no way hain of function research made sense before we had the germ theory of disease.
Ok, that's interesting - I think we're getting somewhere, but I'm still not sure what the objection is.
1. "More intelligent than us like we are to chimps" - is the 'chimps' part important there? I mean, would you say that "more intelligent" than us is unclear, or impossible, or just that a smart AI wouldn't have the same kind of qualitative advantage we have over chimps?
2. "That it can be further extrapolated" - this makes me think you're making one of the first two claims , but I could be wrong.
3. "Most importantly that we can somehow imagine the characteristics of such an extrapolated being" - why is this part important? I don't think that AI x-risk people think they're good at imagining what AGI would be like - they just know what very smart people could do, and worry about AIs that could potentially do the same or more. If you think AGI might just have a well-developed morality by virtue of being smart, I'd point you to Eliezer's stuff on the orthogonality thesis, which I found pretty convincing.
Re: "We couldn't have seen how to do it safely without actually understanding what it took. There's no way gain of function research made sense before we had the germ theory of disease": the analogy to AGI is that people are worried we'll know how to build something smart before we have adequate theories about *safety*. There's no contradiction there.
I think the best response to this genre of AI risk commentary comes from SSC’s “Maybe the Real Superintelligent AI is Extremely Smart Computers” (https://slatestarcodex.com/2018/01/15/maybe-the-real-superintelligent-ai-is-extremely-smart-computers/):
“In my own psychiatric practice, I am always very reluctant to assume a patient is projecting unless I know them very well. I’ve written more about the dangers of defense mechanism narratives here, but the short version is that amateur therapists inevitably end up using them to trivialize or psychologize a patient’s real concerns. I can’t tell you how many morons hear a patient say “I think my husband hates our kids”, give some kind of galaxy-brain level interpretation like “Maybe what’s really going on is you unconsciously hate your kids, but it’s more comfortable for you to imagine this of your husband”, and then get absolutely shocked when the husband turns out to be abusing the kids.
Accusing an entire region of California of projection is a novel psychoanalytic manuever, and I’m not sure Chiang and Buzzfeed give it the caution it deserves. The problem isn’t that they don’t have a plausible-sounding argument. The problem is that this sort of hunting-for-resemblances is a known bug in the human brain. You can do it to anything, and it will always generate a plausible-sounding argument.”
It’s frustrating, as someone who’s on the periphery of AI safety work, to see people come in and dismiss this whole field of possibly *critically important* research with, “But did you know religions talk about the apocalypse?” And Rohit’s proposed obvious “solutions” look unworkable to most people who’ve spent time thinking seriously about the difficulties involved.
Tl;dr: analogizing AGI concerns to eschatology don’t tell us anything about whether AGI could actually kill people. If you want to know who’s right, you can’t just take a shortcut through the whole debate by comparing it to religion.
Thanks for this. I read Scott's article as an argument on why arguments from analogies sometimes fail, while his arguments for AGI fall better in the superintelligence faq which also does effectively the same. Both of them seem supporting to my thesis here that much of the xrisk conversation seems the same - arguing from analogies with the same failure modes.
My issue is that this line of extrapolation is assuming a huge amount, including trendlines, scaling, various theories of morality and intelligence, none of which are either testable or provable. Which to me suggests a highly under specified problem; and the comparison to eschatology is in comparing the types of worry I see and the limited levels of evidence against it.
So…I don’t think the argument for AI x-risk is assuming any of these things. People have gone back and forth quite a lot about each one, and each has been argued for. Are you more skeptical that we’ll likely get very capable AI systems in the next 50 years, or that they’ll be hard to align?
I think worries about brittleness/untestability of claims are totally fair, but there has been quite a lot of rigor applied to arguments on both sides despite that constraint. I’m pretty persuaded of the core case - we don’t know how to align AI and if it’s smarter than us that could be catastrophic - and I didn’t see much engagement with the reasons people believe that. Particularly the last few paragraphs treated AI concerns as fanciful and struck me more as psychoanalysis than argument.
I think we’re just in a very bad situation, where we have reasons to believe AGI is coming (many of them data-driven but speculative) and reasons to believe that could be dangerous (outsmarting humans for its own reasons seems bad for us) but no knock-down empirical argument that’s going to convince everyone. Though I do think that of the people who engage with the args, most are convinced (e.g. Holden aggressively criticizing MIRI in 2012 and getting gradually more sympathetic until now he’s very in favor of OpenPhil funding AI safety).
Hi Rohit, I really enjoyed reading your essay. I have always wanted to read more about AI risk, and this essay gives me the impetus.