Extinction risks
There are a lot of people worrying nowadays about the global energy crisis. Peak oil. The lack of alternative energies. The practicality of solar and hydroelectric and wind and the politics of actually getting it implemented. But why are comparatively few people concerned about the graver dangers facing humanity? Things like genetic engineering, which has the potential to allow a terrorist to create a supervirus, or nanotechnology, which could create something like a supervirus, only that acts much quicker and affects all life instead of just a few species (including humans). You would think people would be worried about this. Well, you would think that if you thought that people were rational. But they’re not, not, not, not. The field of heuristics and biases tells us that humans make scads of different errors, in predictable ways, when we do reasoning. The paper I link, by Eliezer Yudkowsky, gives a great overview, with a specific eye as to how they affect evaluations of global risks.
His next paper (to be published in the same book), deals with another specific threat facing humanity. That is, the creation of a smarter-than-human intelligence. A smarter-than-human intelligence is not simply an intelligence with a very large IQ. You see, IQ (as far is it really does measure general intellectual ability) covers the infinitismal range of intelligence from dumb human to smart human. That complete range, compared to the range from no intelligence to human intelligence, is an extremely tiny dot. And when we sucessfully program an AI, it starts out with no intelligence, and quickly travels along the line until it reaches human intelligence. Is there any reason it should stop there? No, if it gets to human intelligence in the first place, it’s quickly going to go far, far past that. (That’s even besides recursive self-improvement.)
Anyway, that’s just a small taste of the article. He has another great piece of writing here. What is truth?
