Thanks for this. I read Scott's article as an argument on why arguments from analogies sometimes fail, while his arguments for AGI fall better in the superintelligence faq which also does effectively the same. Both of them seem supporting to my thesis here that much of the xrisk conversation seems the same - arguing from analogies with t…
Thanks for this. I read Scott's article as an argument on why arguments from analogies sometimes fail, while his arguments for AGI fall better in the superintelligence faq which also does effectively the same. Both of them seem supporting to my thesis here that much of the xrisk conversation seems the same - arguing from analogies with the same failure modes.
My issue is that this line of extrapolation is assuming a huge amount, including trendlines, scaling, various theories of morality and intelligence, none of which are either testable or provable. Which to me suggests a highly under specified problem; and the comparison to eschatology is in comparing the types of worry I see and the limited levels of evidence against it.
So…I don’t think the argument for AI x-risk is assuming any of these things. People have gone back and forth quite a lot about each one, and each has been argued for. Are you more skeptical that we’ll likely get very capable AI systems in the next 50 years, or that they’ll be hard to align?
I think worries about brittleness/untestability of claims are totally fair, but there has been quite a lot of rigor applied to arguments on both sides despite that constraint. I’m pretty persuaded of the core case - we don’t know how to align AI and if it’s smarter than us that could be catastrophic - and I didn’t see much engagement with the reasons people believe that. Particularly the last few paragraphs treated AI concerns as fanciful and struck me more as psychoanalysis than argument.
I think we’re just in a very bad situation, where we have reasons to believe AGI is coming (many of them data-driven but speculative) and reasons to believe that could be dangerous (outsmarting humans for its own reasons seems bad for us) but no knock-down empirical argument that’s going to convince everyone. Though I do think that of the people who engage with the args, most are convinced (e.g. Holden aggressively criticizing MIRI in 2012 and getting gradually more sympathetic until now he’s very in favor of OpenPhil funding AI safety).
Thanks for this. I read Scott's article as an argument on why arguments from analogies sometimes fail, while his arguments for AGI fall better in the superintelligence faq which also does effectively the same. Both of them seem supporting to my thesis here that much of the xrisk conversation seems the same - arguing from analogies with the same failure modes.
My issue is that this line of extrapolation is assuming a huge amount, including trendlines, scaling, various theories of morality and intelligence, none of which are either testable or provable. Which to me suggests a highly under specified problem; and the comparison to eschatology is in comparing the types of worry I see and the limited levels of evidence against it.
So…I don’t think the argument for AI x-risk is assuming any of these things. People have gone back and forth quite a lot about each one, and each has been argued for. Are you more skeptical that we’ll likely get very capable AI systems in the next 50 years, or that they’ll be hard to align?
I think worries about brittleness/untestability of claims are totally fair, but there has been quite a lot of rigor applied to arguments on both sides despite that constraint. I’m pretty persuaded of the core case - we don’t know how to align AI and if it’s smarter than us that could be catastrophic - and I didn’t see much engagement with the reasons people believe that. Particularly the last few paragraphs treated AI concerns as fanciful and struck me more as psychoanalysis than argument.
I think we’re just in a very bad situation, where we have reasons to believe AGI is coming (many of them data-driven but speculative) and reasons to believe that could be dangerous (outsmarting humans for its own reasons seems bad for us) but no knock-down empirical argument that’s going to convince everyone. Though I do think that of the people who engage with the args, most are convinced (e.g. Holden aggressively criticizing MIRI in 2012 and getting gradually more sympathetic until now he’s very in favor of OpenPhil funding AI safety).