Yudkowsky, If Anyone Builds it, Everyone Dies

[jordansparks]

"If Anyone Builds it, Everyone Dies" is a book by Eliezer Yudkowsky and Nate Soares. I read it a while ago, but I ran out of energy at about the 3/4 point, after he started talking about how to fix the problem. The last part wasn't interesting because he thinks there is no solution. And since I strongly disagreed with the entire premise of the problem, I just didn't see the point. There are a few reasons for me to not write about his book. He's a bit smarter than me at certain kinds of thinking, but not in all domains. He has debated this topic exhaustively over the years, and he has clearly stated that he has a ready rebuttal for any argument. Still, I'm pretty sure he's wrong, so here we go.
I can't address all of his arguments, so I'll just address a few. Let's start with the analogy of Aztecs going into a fight against Europeans with guns. I would point out the obvious that the Europeans were not smarter. The reason they had guns was a few thousand years of iterative technology improvement. That gets to the core of my issue with Yudkowsky. Being a little smarter won't help. In all cases, the advantage is not intelligence, but many years of iterative improvements across a broad range of technologies.
The second complaint is where he says the superintelligence would be 10,000 times smarter or faster or whatever. Um, no. That's a pretty big number. Seems like we are still working on building a computer that is even 1x as smart as a human, let alone 10,000x. Yes, I realize that it could be in one domain, but even domain-specific intelligence advances comes in spurts that ebb and flow. You would need a lot of different domains to compete against humans. Claiming that the superintelligence would be 10,000x is a typical sci-fi trope. It creates interesting contrast, but it's not reality. On Star Trek, they can drop Data into a ship full of humans for interesting contrast. But if they had the technology to build one Data, then there would already have been billions of similar intelligent beings that were very nearly the same level of intelligence. You can't end up with a sudden 10,000x improvement. You might end up with a 20% improvement to get a 1.2x, but that advantage will be short lived as others quickly come up with their own 20% improvements.
His position is based on the combination of a series of extraordinary claims. I strongly disagree with every one of them.
- That it is possible to very quickly improve in one domain.
- That nobody else is also improving in that domain at a similar pace.
- That one or a few domains is enough.
- That this intelligence would have access to enough resources (10,000 times a few billion dollars worth?).
- That this rapid improvement would not taper off.
- That purely digital improvements could work without corresponding physical technology improvements that would take much longer.
- That intelligence is actually much of an advantage.
- etc.
Finally, I'll address that last one. There have been a number of humans who were significantly more intelligent in one domain than the rest of humanity. This did not result in them being particularly successful. A narrow improvement in intelligence is not enough by itself. It's going to require synergy between multiple domains, multiple individuals, multiple groups/teams, physical improvements, etc.