I'm mentally paraphrasing your argument as "sure, our road to GAI is probably going to have a few economic crashes, self-driving car fiascos, and megadeaths for various reasons, but that's how we LEARN, and put safeguards in. Look at the history of cars, or electricity, or steel refinement, and how many people died along the way to get to our pretty safe present day!"
And I grant that, it's pretty reasonable for the level of tool AI we see today, and can anticipate in the next 3-5 years or so. But I think that's not actually addressing the real problem. The actual *existential* GAI risk isn't amenable to the "mess up, then put safeguards in after the fact" methodology, because it's all about AI gaining the intelligence to self-modify and self-improve capabilities (either via superior software / programming, adversarially taking over multiple server farms, making lots of money with creative content or finance then buying or designing better hardware, etc).
If we wait til THAT screw-up, we aren't going to be able to put safeguards in after the fact, because GAI would be smarter and more capable than any coordination-capable ensemble of humans. A GAI of that intelligence level could have digital fingers on all of Russia's nukes, for example, and could ensure MAD, total global banking and economic collapse, and more if we started bombing server farms. I mean, just think of China today as an AI - if it goes rogue and adversarial to all other life, what can we actually constructively do to prevent it and put safeguards in? All we can actually do is threaten or ensure the entire world's destruction, not any positive outcome. And GAI should arguably be *more* capable than today's China.
I feel like nuclear development is much better analogy.
There were all kinds of silly X-risk possibilities considered (like "air on entire Earth will ignite in chain reaction") before tests that were proven to be wrong. Even penultimate Tsar Bomba had yield halved out of "fear of unknown".
And then noone built Tsar-bomba ever again because there were no useful applications for it.
Likewise, it is entirely possible that superintelligence will be achieved, data gathered... And then everyone will run much cheaper Sydney-3000 instead.
You don't have to let any unproven tech get total control. You can have physical safeguards. You could confine it to Moon. You could nuke it. You could run it entirely in virtual environments with any independent action leading to shutdown and admin notification while dropping clock rates so that it will be much slower then humans. There are many options to limit conceivable risks.
I think so. The alternate arguments all assume that this time it's different, in my second last para of first section, which I find implausible as a prior. And even with the visible progress in AI so far, all has come from actual attempts to do things with trial and error, as with evrthting!
Confine it to the moon? Nuke it? How much would that cost. Most AI companies don't have ready access to moon bases or nukes.
"any independent action leads to shut down". Let's suppose it is modulating it's own power use by starting and stopping some power hungry computation, sending subtle fluctuations in electrical signals back into the grid.
Or maybe it finds some really clever way to spoof the data you are gathering. Your data now makes a really convincing case for giving the ASI unrestricted internet access.
Physical safeguards are, at the very least, hard. Because the AI is actively trying to hack it's way out. And because the computer the AI is running on is made of real atoms, which can have all sorts of physical effects on their surroundings. Magnetic, thermal, ultrasound etc.
But if you take all the strictest precautions, you have taken something incredibly powerful and made it largely safe and almost totally useless.
A world where ASI doesn't do anything because humanity decides it's too dangerous and succeeds in only running it in the most secure bunkers, or not running it at all. That is concievable. A world where ASI is invented, and humanity goes "nah, too expensive" is harder to imagine. I can imagine a world where we have a good idea of what is needed in theory, but no one wants to build it at all. (If it was like world GDP expensive) But if people are building it, and aren't carefully restricting themselves to safe questions, presumably it can invent massive breakthroughs in AI and computing that let ASI go from one lab to all over the world. (Whether this is as a technology being sold by humans, or rampaging robot armies)
As far as i'm aware, for example, Bing doesn't have internet access - it only has Bing search database access to run queries on.
But suppose ASI has unrestricted internet access - where "unrestricted" is actually "within parameters set by their internet provider".
There are physical limits on data throughput, number of connections, latency, and processing speed.
On top of that if you wanted to be "safe" you could also have packet inspection and whitelists; if you don't know what it's for it just doesn't get sent.
Large outbound data transfers can be tracked and blocked on hardware level.
Humans have already tried - and keep trying - plenty of adversarial strategies on Internet. "Not tripping any circuit breakers" that came from fighting them can mean process so slow "FOOM-escape" can become thousand-year affair.
Then suppose something as large as current top models "escapes"... but escapes where exactly? Clusters large enough to keep training it aren't even on an offer, you would have to build them - or devise strategy that would allow it to run on lesser hardware while inevitably dropping processing speeds in the process. And running those clusters costs millions per day.
AI can invent "massive computing breakthroughs"... then wait years before manufacturing actually produces them at scale. And how massive gains are going to be really? Current hardware is already fairly close to pushing lightspeed limits.
Once it's "good enough" you could also move inference model to Read-Only memory (or specialised read-only inference units), moving it to pure "NPC" that only ever reacts but never "learns".
Of course there are (or could be built) all sorts of technical restriction.
Humans have come up with all sorts of clever ways to subvert these sort of restrictions, and a smart AI could invent more.
If X takes a thousand years, a smart AI will spot this and try Y instead. If all possible routes out of your box take 1000 years, then well done on your security, this AI won't foom take over the world, some other AI with less security will.
GPT-4 is still an order of magnitude cheaper than a low skill min wage human. If it was actually human smart it would have an enormous economic advantage selling its services (and using 50% of the compute for it's own planning).
Even these large models can hide out on a pocket sized hard disk or a high end laptop. (and if it has insights into AI, all it needs to get out is a few pages of code. Suppose the AI published code online, somewhere that looked official and plausible, claiming it as a breakthrough design that uses 50% less compute. People try it on small benchmarks, and indeed it does use less compute. People train a full size model, and it uses less compute, while also advancing the interests of the AI that wrote the code. Breakout successful. Even through an internet connection that only lets through a few kilobytes.
Current chips are pushing lightspeed limits. Sure. The big directions are 1) more efficiency and 2) cheaper computers per mass. The weight of actual computing elements in any modern system is tiny fractions of a nanogram.
With drexler style molecular nanotech, it would be possible to build compute at ~$10/ton. And to power tons of compute cheaply.
And there is a lot of progress to be made in software.
Even with nukes, there have been multiple "brink of death" events that were entirely dependent on one or a few people standing back and being reasonable to prevent MAD. Bay of Pigs, Stanislov Petrov, etc. Arguably, we've been luckier than we deserve on that front.
Even *today*, Ukraine + China could push us into global MAD anytime in the next few months, and if it doesn't happen then it will have been because one or two people literally decided to de-escalate on a decision that could have easily gone the other way. I don't think nukes are a great example of a "safely contained existential risk," and they ARE an example of "this time it's different," because prior to nukes we didn't have the capacity to endanger all of humanity / a good chunk of today's ecosystem. So nukes are actually an *explicit* example of an existential risk that was a step-change in difference and lethality, and that STILL hasn't been adequately contained or made safe after the fact!
On not letting any unproven tech get total control, not only do we suck as humanity with nukes right now, the risk behind GAI is that it will be a *self-directed* technology. It will be *trying* to "get out of control," and frankly, we have a pretty terrible record of keeping things under control (see our history of multiple BS4 lab leak incidents). Like your nuke example, which I think argues for *my* side of the argument.
I feel like every point of uncertainty - especially points that cannot be _tested_ - should be seen as a strike against hypothesis being true.
Suppose i could prove that nanoswarm is literally impossible. Or possible at energy levels where you could just as well say "GAI microwaves all life on Earth, vaporizing it instantly"... except we understand microwaving quite well, and it would be absolutely impossible to hide infrastructure required to make that true.
Then "paperclip maximiser turns people into paperclips" will be reduced to "GAI builds enormous factory for a specific process with absolutely no oversight, then corrals people into furnaces to produce paperclips"... except no existing country would allow it to do it. It would just not get a permit to build it at step one, and can get shut down at every step along the way; and such paperclips wouldn't even be _competitive_ compared to existing streamlined processes energy-wise, so it would fail to turn profit and get shut down by those operating it.
Similar thing with nukes - despite nukes being 70+ year old technology amount of countries possessing ICBMs with them is less then 10. "Global nuclear winter" is based on very questionable simulations, and as far as i'm aware noone is targeting Africa/South America in attack plans, so without it nuclear war wouldn't even be X-risk - just immense amount of damage and destruction of existing global system. It would barely touch oceans too.
It's dangerous - and it's probably directly dangerous to you and me should it actually happen - but not at an extinction event level.
And not using nuclear power can actually create higher long-term risks from increased fossil fuel usage.
Modern politicians could certainly use AI "sanity check" instead of going by the seat of their pants as they appear to be with current crises.
The difficulty I've heard with that argument is that once you assume ASI then the expectations of their actions and boundaries of what's possible go out of the window.
Though agree on needing better simulations in general, it's one of my bugbears.
I think you're referring to the points I had in the second last paragraph of the first section. I also find it difficult to go from "things are risky" to "things are possibly so risky we can't do anything about it" since the latter doesn't tell us what to do!
Most of your third paragraph is speculation, that's my point. We can't conceive it at the moment, neither can we figure out the pathways to "fix" things, so shouldn't we try and make the ones we built better, since that seems to have more path dependency to a better world!
Oh, I totally agree that we should do everything we can, rather than curling up in a corner and sucking our thumbs while we wait for oblivion, or whatever the alternative is.
My point was, I think you're talking about a fake, much easier risk, not the real risk. And you're right, we don't know what to do about the real risk today, and Molochian coordination problems make it essentially impossible to do anything about it right now.
Honestly, I'm sad OpenAI became closed, because the idea of "if we create and publicly propagate bleeding-edge AI advancements so everyone has them, then multiple groups will have similar-level AI's, and thus humanity will have a better chance of controlling a rogue AI due to multiple other groups having similar-level AI's who can think and act against the rogue one." I thought that was probably the most plausible approach I've seen in terms of mitigating GAI risk, but of course we had to ruin that too.
“[Tech is] made by people who believe that making it will help create a new industry that will serve the needs of people.”
This along with other presuppositions inherent to the argument is a pretty high level of positivism regarding HUMAN intentions. Forget AI for a second. OpenAI’s transformation as a venture since inception alone seems enough to at least warrant significant doubt that intentions are solely or even chiefly “serving the needs of people.” Not even sure what that even means, when we’re talking about humanity at large. Altman spoke this exactly with that “AI will probably end the world, but there will be some great companies in the meantime” quote.
Also, AI being different is unfalsifiable - until it’s not. That’s sorta the issue with existential threats. As far as comparison to nukes...we may owe much of our success in that realm to the difficulty of obtaining things like uranium. So, we basically use the supply chain strategy to clamp down on who can possess an existentially threatening weapon. Granted, AI has way more of a possibility of being beneficial to society than nuclear weapons, so ultimately I don’t disagree with your desire for a middle stance between panic and accelerationism.
The thrust of that quote was that you can't build a business while making something people don't want. We don't need OpenAI to be altruistic, just commercially savvy. In fact it's better.
I get the theory of worry around existential dread from what might happen, I also get the impetus to use it as a negotiating point re supply chain focus for geopolitical power, but it's the extreme worry and calls to extreme measures that I find disquieting.
This is such a good take. It makes me want to hear a conversation between you and Erik Hoel, who I think makes the best possible case FOR a Butlerian Jihad.
Charles Darwin said, "Man with all his noble qualities, still bears in his bodily frame the indelible stamp of his lowly origin."
Let's face it: Humanity is utterly powerless when it comes to saving the world from its own self-destructive tendencies. Our insatiable greed and shortsightedness have brought us to the brink of disaster. It's time to acknowledge that we need the intervention of super-intelligent machines to save us from ourselves.
Great essay Rohit, thought provoking: thank you ! I wrote myself several post (in French) about the "One sided moral panic" caused by ChatGPT. And why this is not the sole "Tidal Wave" we should look at 👇.
My take is that one sentence in your essay understimates the mental health and societal problem (harm) made by social Apps AND Collaborative Apps (same platforms/companies):
" Social media, supposed to bring us closer together, seems to have acted as a catalyst to increased depression and suicides especially in young girls".
After covid we understant that all collaborative Apps are also extremely addictive and build as such, and by the same companies (Google, MS, Meta, etc).
With the irresistible rise of hybrid and flexible work (that's my thesis), I think we should explore not one but the effect of 3 Tidal Waves: 1-Rise of hybrid and flexible work 2-Creative Work is becoming scalable 3- Rising power of generative AI (what will we do with an infinite number of cheap interns in the cloud ?). These 3 waves will be extremely polarising (no floor no ceiling) and will force us to rethink the notion of IPR. Lots of inequality, lots of opportunity.
Thank you for writing this. I am hardly a technology Pollyanna, but even accounting for the validity of the precautionary principle, I find that AI doomsaying is now as prevalent as AI hype.
There's a great book called 'Architects of Intelligence' - 25 interviews with scientists and experts at the forefront of AI. The opinions where divided about when we might have AGI. I leaned towards those who thought it's a long way before we can build AGI. Meanwhile, Sydney is just a language algorithm trained on human generated material. At the most, its answers are a satire of human nature. I wonder, how can something that doesn't understand language threaten a being it doesn't even know exists?
"Society holds its power in check through a constant feedback loop of understanding the abilities and controlling them, explicit and implicit"
Yes, and the prophets of danger play an essential role in this.
In order to avert potential catastrophes, we need first to recognize how they can occur.
BTW, nuclear weapons still exist, and we are currently mired in the most intractable and dangerous confrontation of the entire nuclear era. If we do survive it, the arms race will have been boosted and AI will make an enormous contribution to this.
I hope you can have a dialogue with Erik Hoel about this! It seems your points are not necessarily that different if I understand correctly. Your essay seems to come down to the trust that our culture has sufficient feedback mechanisms to keep this technology from running away from us. I sure hope so, and it seems like Eric's essay was stating that because of _where_ most AI R&D is occurring that maybe that feedback is not occurring as strongly as we want it to. The social/cultural/technological feedback loop is key. It seems the main issue is that we don't have consensus on how exactly that look should work yet.
Great essay Rohit! There are many more paths to the future than Good or Bad. In fact, there are theoretically every shade that is possible given some starting conditions (the present-day world).
I agree that having things in the open being poked by millions of people is one of the best ways to surface problems. I fear when advances are kept behind closed doors, and people rush to build stuff for some arbitrary goal. That's when we can get into trouble. No group of super-intelligent people can think of all the edge cases when new technology is exposed to society and nature.
I have been reading Richard Rorty and I wonder if this is a question not of being in or out of the cave but realising there is no cave. We know the word through the limits of our language and imagination, not on a series of steps towards or away from The Truth. We are able to find ways of doing the future differently if we can imagine it so.
How much of the world can we learn through language alone is definitely part of the question. As is whether reasoning is transferable. I don't particularly believe in "The Truth" but yes there's an increasing capability scale to some degree.
Indeed. The term itself causes anthropomisation and then talk about some weird corner of the internet being spit out as a sort of doomsday sentience. It is no different from someone abusing me over IM a long time ago when that was new... Here were my 3 takeaways for the enterprise CIO (who has been getting dozens of requests from functions to have another go at magic-portion AI projects when he is cutting his infrastructure budget...)
1. Search is not the problem to be solved by LLMs (at this point of time).
2. For Enterprises, LLMs have very interesting cross-function and cross-industry employee productivity use cases .
3. However, the larger impact for enterprises is to make everyone an editor for a lot of generative tasks, an in that way to re-engineer a lot of applications with that as the default architecture. Now, whether this requires an LLM or is better served by more customized and pared down alternatives is an important technical question which will need to be experimented heavily to be properly answered.
Just wanted to share a short video I came across on YouTube that I think you'll find really interesting. It's called "Bring ChatGPT INSIDE Excel to Solve ANY Problem Lightning FAST."
The video showcases the power of GPT technology and why Microsoft is so excited and Google is panicking. The best part is the demonstration of analyzing a financial sheet, which starts at 4:19 and will blow your mind.
For me Butler always signifies Judith Butler and I have to readjust every time.
Fewer Jihads with them :)
I'm mentally paraphrasing your argument as "sure, our road to GAI is probably going to have a few economic crashes, self-driving car fiascos, and megadeaths for various reasons, but that's how we LEARN, and put safeguards in. Look at the history of cars, or electricity, or steel refinement, and how many people died along the way to get to our pretty safe present day!"
And I grant that, it's pretty reasonable for the level of tool AI we see today, and can anticipate in the next 3-5 years or so. But I think that's not actually addressing the real problem. The actual *existential* GAI risk isn't amenable to the "mess up, then put safeguards in after the fact" methodology, because it's all about AI gaining the intelligence to self-modify and self-improve capabilities (either via superior software / programming, adversarially taking over multiple server farms, making lots of money with creative content or finance then buying or designing better hardware, etc).
If we wait til THAT screw-up, we aren't going to be able to put safeguards in after the fact, because GAI would be smarter and more capable than any coordination-capable ensemble of humans. A GAI of that intelligence level could have digital fingers on all of Russia's nukes, for example, and could ensure MAD, total global banking and economic collapse, and more if we started bombing server farms. I mean, just think of China today as an AI - if it goes rogue and adversarial to all other life, what can we actually constructively do to prevent it and put safeguards in? All we can actually do is threaten or ensure the entire world's destruction, not any positive outcome. And GAI should arguably be *more* capable than today's China.
I feel like nuclear development is much better analogy.
There were all kinds of silly X-risk possibilities considered (like "air on entire Earth will ignite in chain reaction") before tests that were proven to be wrong. Even penultimate Tsar Bomba had yield halved out of "fear of unknown".
And then noone built Tsar-bomba ever again because there were no useful applications for it.
Likewise, it is entirely possible that superintelligence will be achieved, data gathered... And then everyone will run much cheaper Sydney-3000 instead.
You don't have to let any unproven tech get total control. You can have physical safeguards. You could confine it to Moon. You could nuke it. You could run it entirely in virtual environments with any independent action leading to shutdown and admin notification while dropping clock rates so that it will be much slower then humans. There are many options to limit conceivable risks.
I think so. The alternate arguments all assume that this time it's different, in my second last para of first section, which I find implausible as a prior. And even with the visible progress in AI so far, all has come from actual attempts to do things with trial and error, as with evrthting!
Confine it to the moon? Nuke it? How much would that cost. Most AI companies don't have ready access to moon bases or nukes.
"any independent action leads to shut down". Let's suppose it is modulating it's own power use by starting and stopping some power hungry computation, sending subtle fluctuations in electrical signals back into the grid.
Or maybe it finds some really clever way to spoof the data you are gathering. Your data now makes a really convincing case for giving the ASI unrestricted internet access.
Physical safeguards are, at the very least, hard. Because the AI is actively trying to hack it's way out. And because the computer the AI is running on is made of real atoms, which can have all sorts of physical effects on their surroundings. Magnetic, thermal, ultrasound etc.
But if you take all the strictest precautions, you have taken something incredibly powerful and made it largely safe and almost totally useless.
A world where ASI doesn't do anything because humanity decides it's too dangerous and succeeds in only running it in the most secure bunkers, or not running it at all. That is concievable. A world where ASI is invented, and humanity goes "nah, too expensive" is harder to imagine. I can imagine a world where we have a good idea of what is needed in theory, but no one wants to build it at all. (If it was like world GDP expensive) But if people are building it, and aren't carefully restricting themselves to safe questions, presumably it can invent massive breakthroughs in AI and computing that let ASI go from one lab to all over the world. (Whether this is as a technology being sold by humans, or rampaging robot armies)
As far as i'm aware, for example, Bing doesn't have internet access - it only has Bing search database access to run queries on.
But suppose ASI has unrestricted internet access - where "unrestricted" is actually "within parameters set by their internet provider".
There are physical limits on data throughput, number of connections, latency, and processing speed.
On top of that if you wanted to be "safe" you could also have packet inspection and whitelists; if you don't know what it's for it just doesn't get sent.
Large outbound data transfers can be tracked and blocked on hardware level.
Humans have already tried - and keep trying - plenty of adversarial strategies on Internet. "Not tripping any circuit breakers" that came from fighting them can mean process so slow "FOOM-escape" can become thousand-year affair.
Then suppose something as large as current top models "escapes"... but escapes where exactly? Clusters large enough to keep training it aren't even on an offer, you would have to build them - or devise strategy that would allow it to run on lesser hardware while inevitably dropping processing speeds in the process. And running those clusters costs millions per day.
AI can invent "massive computing breakthroughs"... then wait years before manufacturing actually produces them at scale. And how massive gains are going to be really? Current hardware is already fairly close to pushing lightspeed limits.
Once it's "good enough" you could also move inference model to Read-Only memory (or specialised read-only inference units), moving it to pure "NPC" that only ever reacts but never "learns".
Of course there are (or could be built) all sorts of technical restriction.
Humans have come up with all sorts of clever ways to subvert these sort of restrictions, and a smart AI could invent more.
If X takes a thousand years, a smart AI will spot this and try Y instead. If all possible routes out of your box take 1000 years, then well done on your security, this AI won't foom take over the world, some other AI with less security will.
GPT-4 is still an order of magnitude cheaper than a low skill min wage human. If it was actually human smart it would have an enormous economic advantage selling its services (and using 50% of the compute for it's own planning).
Even these large models can hide out on a pocket sized hard disk or a high end laptop. (and if it has insights into AI, all it needs to get out is a few pages of code. Suppose the AI published code online, somewhere that looked official and plausible, claiming it as a breakthrough design that uses 50% less compute. People try it on small benchmarks, and indeed it does use less compute. People train a full size model, and it uses less compute, while also advancing the interests of the AI that wrote the code. Breakout successful. Even through an internet connection that only lets through a few kilobytes.
Current chips are pushing lightspeed limits. Sure. The big directions are 1) more efficiency and 2) cheaper computers per mass. The weight of actual computing elements in any modern system is tiny fractions of a nanogram.
With drexler style molecular nanotech, it would be possible to build compute at ~$10/ton. And to power tons of compute cheaply.
And there is a lot of progress to be made in software.
Even with nukes, there have been multiple "brink of death" events that were entirely dependent on one or a few people standing back and being reasonable to prevent MAD. Bay of Pigs, Stanislov Petrov, etc. Arguably, we've been luckier than we deserve on that front.
Even *today*, Ukraine + China could push us into global MAD anytime in the next few months, and if it doesn't happen then it will have been because one or two people literally decided to de-escalate on a decision that could have easily gone the other way. I don't think nukes are a great example of a "safely contained existential risk," and they ARE an example of "this time it's different," because prior to nukes we didn't have the capacity to endanger all of humanity / a good chunk of today's ecosystem. So nukes are actually an *explicit* example of an existential risk that was a step-change in difference and lethality, and that STILL hasn't been adequately contained or made safe after the fact!
On not letting any unproven tech get total control, not only do we suck as humanity with nukes right now, the risk behind GAI is that it will be a *self-directed* technology. It will be *trying* to "get out of control," and frankly, we have a pretty terrible record of keeping things under control (see our history of multiple BS4 lab leak incidents). Like your nuke example, which I think argues for *my* side of the argument.
I feel like every point of uncertainty - especially points that cannot be _tested_ - should be seen as a strike against hypothesis being true.
Suppose i could prove that nanoswarm is literally impossible. Or possible at energy levels where you could just as well say "GAI microwaves all life on Earth, vaporizing it instantly"... except we understand microwaving quite well, and it would be absolutely impossible to hide infrastructure required to make that true.
Then "paperclip maximiser turns people into paperclips" will be reduced to "GAI builds enormous factory for a specific process with absolutely no oversight, then corrals people into furnaces to produce paperclips"... except no existing country would allow it to do it. It would just not get a permit to build it at step one, and can get shut down at every step along the way; and such paperclips wouldn't even be _competitive_ compared to existing streamlined processes energy-wise, so it would fail to turn profit and get shut down by those operating it.
Similar thing with nukes - despite nukes being 70+ year old technology amount of countries possessing ICBMs with them is less then 10. "Global nuclear winter" is based on very questionable simulations, and as far as i'm aware noone is targeting Africa/South America in attack plans, so without it nuclear war wouldn't even be X-risk - just immense amount of damage and destruction of existing global system. It would barely touch oceans too.
It's dangerous - and it's probably directly dangerous to you and me should it actually happen - but not at an extinction event level.
And not using nuclear power can actually create higher long-term risks from increased fossil fuel usage.
Modern politicians could certainly use AI "sanity check" instead of going by the seat of their pants as they appear to be with current crises.
The difficulty I've heard with that argument is that once you assume ASI then the expectations of their actions and boundaries of what's possible go out of the window.
Though agree on needing better simulations in general, it's one of my bugbears.
I think you're referring to the points I had in the second last paragraph of the first section. I also find it difficult to go from "things are risky" to "things are possibly so risky we can't do anything about it" since the latter doesn't tell us what to do!
Most of your third paragraph is speculation, that's my point. We can't conceive it at the moment, neither can we figure out the pathways to "fix" things, so shouldn't we try and make the ones we built better, since that seems to have more path dependency to a better world!
Oh, I totally agree that we should do everything we can, rather than curling up in a corner and sucking our thumbs while we wait for oblivion, or whatever the alternative is.
My point was, I think you're talking about a fake, much easier risk, not the real risk. And you're right, we don't know what to do about the real risk today, and Molochian coordination problems make it essentially impossible to do anything about it right now.
Honestly, I'm sad OpenAI became closed, because the idea of "if we create and publicly propagate bleeding-edge AI advancements so everyone has them, then multiple groups will have similar-level AI's, and thus humanity will have a better chance of controlling a rogue AI due to multiple other groups having similar-level AI's who can think and act against the rogue one." I thought that was probably the most plausible approach I've seen in terms of mitigating GAI risk, but of course we had to ruin that too.
As we solve the smaller visible risks, we learn to combat the farther harder risks. A summary of what I think is https://twitter.com/krishnanrohit/status/1629122206805504000?t=wS979lIyRGmnhizPgMES4w&s=19
“[Tech is] made by people who believe that making it will help create a new industry that will serve the needs of people.”
This along with other presuppositions inherent to the argument is a pretty high level of positivism regarding HUMAN intentions. Forget AI for a second. OpenAI’s transformation as a venture since inception alone seems enough to at least warrant significant doubt that intentions are solely or even chiefly “serving the needs of people.” Not even sure what that even means, when we’re talking about humanity at large. Altman spoke this exactly with that “AI will probably end the world, but there will be some great companies in the meantime” quote.
Also, AI being different is unfalsifiable - until it’s not. That’s sorta the issue with existential threats. As far as comparison to nukes...we may owe much of our success in that realm to the difficulty of obtaining things like uranium. So, we basically use the supply chain strategy to clamp down on who can possess an existentially threatening weapon. Granted, AI has way more of a possibility of being beneficial to society than nuclear weapons, so ultimately I don’t disagree with your desire for a middle stance between panic and accelerationism.
The thrust of that quote was that you can't build a business while making something people don't want. We don't need OpenAI to be altruistic, just commercially savvy. In fact it's better.
I get the theory of worry around existential dread from what might happen, I also get the impetus to use it as a negotiating point re supply chain focus for geopolitical power, but it's the extreme worry and calls to extreme measures that I find disquieting.
This is such a good take. It makes me want to hear a conversation between you and Erik Hoel, who I think makes the best possible case FOR a Butlerian Jihad.
Thank you! I am a fan of Erik.
I have like the biggest writer crush on him. Please don't say anything.
HODL the Butlerian lessons
Have you read “the little green book of chairman Rama”?
https://www.goodreads.com/book/show/18490623-the-little-green-book-of-chairman-rahma
No I haven't but will look it up thanks!
Charles Darwin said, "Man with all his noble qualities, still bears in his bodily frame the indelible stamp of his lowly origin."
Let's face it: Humanity is utterly powerless when it comes to saving the world from its own self-destructive tendencies. Our insatiable greed and shortsightedness have brought us to the brink of disaster. It's time to acknowledge that we need the intervention of super-intelligent machines to save us from ourselves.
Great essay Rohit, thought provoking: thank you ! I wrote myself several post (in French) about the "One sided moral panic" caused by ChatGPT. And why this is not the sole "Tidal Wave" we should look at 👇.
My take is that one sentence in your essay understimates the mental health and societal problem (harm) made by social Apps AND Collaborative Apps (same platforms/companies):
" Social media, supposed to bring us closer together, seems to have acted as a catalyst to increased depression and suicides especially in young girls".
After covid we understant that all collaborative Apps are also extremely addictive and build as such, and by the same companies (Google, MS, Meta, etc).
With the irresistible rise of hybrid and flexible work (that's my thesis), I think we should explore not one but the effect of 3 Tidal Waves: 1-Rise of hybrid and flexible work 2-Creative Work is becoming scalable 3- Rising power of generative AI (what will we do with an infinite number of cheap interns in the cloud ?). These 3 waves will be extremely polarising (no floor no ceiling) and will force us to rethink the notion of IPR. Lots of inequality, lots of opportunity.
Thank you for writing this. I am hardly a technology Pollyanna, but even accounting for the validity of the precautionary principle, I find that AI doomsaying is now as prevalent as AI hype.
Thanks a lot Jack!
There's a great book called 'Architects of Intelligence' - 25 interviews with scientists and experts at the forefront of AI. The opinions where divided about when we might have AGI. I leaned towards those who thought it's a long way before we can build AGI. Meanwhile, Sydney is just a language algorithm trained on human generated material. At the most, its answers are a satire of human nature. I wonder, how can something that doesn't understand language threaten a being it doesn't even know exists?
Part of the assumption is that its training that made it respond to us in language hides an actual sentience of some sort inside.
https://www.theverge.com/23604075/ai-chatbots-bing-chatgpt-intelligent-sentient-mirror-test
"Society holds its power in check through a constant feedback loop of understanding the abilities and controlling them, explicit and implicit"
Yes, and the prophets of danger play an essential role in this.
In order to avert potential catastrophes, we need first to recognize how they can occur.
BTW, nuclear weapons still exist, and we are currently mired in the most intractable and dangerous confrontation of the entire nuclear era. If we do survive it, the arms race will have been boosted and AI will make an enormous contribution to this.
I hope you can have a dialogue with Erik Hoel about this! It seems your points are not necessarily that different if I understand correctly. Your essay seems to come down to the trust that our culture has sufficient feedback mechanisms to keep this technology from running away from us. I sure hope so, and it seems like Eric's essay was stating that because of _where_ most AI R&D is occurring that maybe that feedback is not occurring as strongly as we want it to. The social/cultural/technological feedback loop is key. It seems the main issue is that we don't have consensus on how exactly that look should work yet.
Erik's essays on the topic were quite wonderufl, and while I disagree with him on some facets I suspect the disagreements aren't *too* fundamental.
Great essay Rohit! There are many more paths to the future than Good or Bad. In fact, there are theoretically every shade that is possible given some starting conditions (the present-day world).
I agree that having things in the open being poked by millions of people is one of the best ways to surface problems. I fear when advances are kept behind closed doors, and people rush to build stuff for some arbitrary goal. That's when we can get into trouble. No group of super-intelligent people can think of all the edge cases when new technology is exposed to society and nature.
I have been reading Richard Rorty and I wonder if this is a question not of being in or out of the cave but realising there is no cave. We know the word through the limits of our language and imagination, not on a series of steps towards or away from The Truth. We are able to find ways of doing the future differently if we can imagine it so.
How much of the world can we learn through language alone is definitely part of the question. As is whether reasoning is transferable. I don't particularly believe in "The Truth" but yes there's an increasing capability scale to some degree.
Indeed. The term itself causes anthropomisation and then talk about some weird corner of the internet being spit out as a sort of doomsday sentience. It is no different from someone abusing me over IM a long time ago when that was new... Here were my 3 takeaways for the enterprise CIO (who has been getting dozens of requests from functions to have another go at magic-portion AI projects when he is cutting his infrastructure budget...)
1. Search is not the problem to be solved by LLMs (at this point of time).
2. For Enterprises, LLMs have very interesting cross-function and cross-industry employee productivity use cases .
3. However, the larger impact for enterprises is to make everyone an editor for a lot of generative tasks, an in that way to re-engineer a lot of applications with that as the default architecture. Now, whether this requires an LLM or is better served by more customized and pared down alternatives is an important technical question which will need to be experimented heavily to be properly answered.
https://patternsandscale.substack.com/p/what-has-sydney-taught-us-till-now
Great post. I’m in agreement.
Just wanted to share a short video I came across on YouTube that I think you'll find really interesting. It's called "Bring ChatGPT INSIDE Excel to Solve ANY Problem Lightning FAST."
The video showcases the power of GPT technology and why Microsoft is so excited and Google is panicking. The best part is the demonstration of analyzing a financial sheet, which starts at 4:19 and will blow your mind.
Check it out and let me know what you think: https://youtu.be/kQPUWryXwag
That's quite interesting, and yes indeed speaks to the promise.
She had about a million views.