AI Safety Optimal Investment

It occurred to me today that I have never seen any attempt at calculating how much society should be willing to spend to prevent an AI Catastrophe. I am pretty good at this kind of thing, so here's a quick Fermi estimate:

Conceptually, this is a lot like buying a life insurance policy for the human race. There is some probability of a catastrophe, so the annual amount we should be willing to pay for insurance is the cost of the catastrophe times its annual probability.

First, we need the dollar value of the human race. This is relatively easy. We have revealed-preference estimates for the value of a flourishing human life in a rich society, so that should be about right for estimating the value people place on a good personal and genetic future, assuming a future where everyone's quality of life is about as good as the average citizen of the US in the early 21st century.

The Value of a Statistical Life in the US is about $10 million, or $1x10^7. Over the relevant time frame in which an AI catastrophe is likely, world population will likely be stabilizing at about 10 billion, or 1x10^10. So the value of the human race, in current US dollars, is about $1x10^17.

Now we find the probability of a catastrophe. For convenience, I use numbers presented in the article linked above, which is a pretty good summary of the current understanding of the field.

Artificial Superintelligence is unlikely to happen before 2025, but almost certain to arrive by 2125. So each year in that timeframe, there is a 1 in 100, or 1x10^-2, chance of ASI.

Now the numbers get very speculative, so in the grand tradition of Fermi estimates I will simply round everything to the nearest power of ten and say that there is a 10% chance that, when ASI develops, it will, without any safeguards, destroy the human race.

So, given a 1x10^-3 annual chance of losing something valued at $1x10^17, we should be willing to spend $1x10^14 to prevent that possibility.

$100 trillion a year is a lot of money. That is almost exactly equal to the total value of the entire world economy. It tells us that any policy short of destroying the entire world economy and/or killing millions of people would be worth doing if it prevented an ASI catastrophe with 100% certainty [Edit: assuming that the policy does not make other existential risks significantly more likely]. For example, a total ban on producing or maintaining any kind of computer, combined with a credible threat to nuke anyone who violates this ban, is a legitimate policy option with a positive expected payoff.

More realistically, we will continue on our current technological course and try to make things safer. Even if AI research only had a 1 in 100 chance of guaranteeing Friendly AI, we should be willing to pay a trillion dollars a year for it.