How Probable is the Unknowable?
Why do people think that question makes sense?
This week, Tyler Cowen argued that we should be “radically agnostic” about what AI will bring in the future, and for this reason he doesn’t really worry that a rogue AI will wipe out humanity. Scott Alexander Siskind responds that Cowen’s position is so laughably fallacious that he can’t even charitably reconstruct it. Cowen’s reasoning seems to be “Who knows whether or not things will be fine?; therefore, everything will be fine.” Put that way, it’s obviously a bad argument. But digging through Siskind’s1 elaboration of why Cowen is wrong actually made me more sympathetic to Cowen.
Siskind is an AI doomer. He’s an acolyte of Eliezer Yudkowsky, a deeply unpleasant yet weirdly influential individual who’s largely responsible for raising the alarm about “AI risk” over the last decade or so. Yudkowski thinks there’s a 90% chance humanity will be wiped out by AI in the not-too-distant future. Siskind is somewhat more sanguine, putting the odds of AI annihilation at 33%. But he’s still depressed and anxious about an AI apocalypse. So when Tyler Cowen says he just has no idea what will happen with AI, Siskind gets annoyed:
In order to generate a belief, you have to do epistemic work. I’ve thought about this question a lot and predict a 33% chance AI will cause human extinction; other people have different numbers. What’s Tyler’s? All he’ll say is that it’s only a “distant possibility”. Does that mean 33%? Does it mean 5-10% (as Katja’s survey suggests the median AI researcher thinks?) Does it mean 1%? Or does Tyler not have a particular percent in mind, because he wants to launder his bad argument through a phrase that sort of sounds like it means “it’s not zero, you can’t accuse me of arrogantly dismissing this future in particular” but also sort of means “don’t worry about it” without having to do the hard work of checking whether any particular number fills both criteria at once?
If you have total uncertainty about a statement (“are bloxors greeblic?”), you should assign it a probability of 50%. If you have any other estimate, you can’t claim you’re just working off how radically uncertain it is.
This is not a good criticism of Cowen. Putting a number on a prediction doesn’t make it a better prediction. It makes it a worse prediction; in fact, I’m inclined to say it’s not a prediction at all. “AI will wipe out humanity” is a prediction. “AI will not wipe out humanity” is another prediction. “There is a 33% chance that AI will wipe out humanity” is not a prediction. It’s waffling, with numbers.
In Siskind’s attempts to “steelman” Cowen’s argument, he keeps on bringing up numbers. Cowen doesn’t think the AI apocalypse is nigh; Siskind can’t accept that conclusion. We’re doing serious epistemology here and, as everyone knows, serious epistemology means attaching arbitrary numbers to your claims. AI wiping out humanity? That’s one of those 33.33% claims. (Not 32%, not 35%. Exactly one third.2) At least, that’s Siskind’s number. What’s your number, Cowen?
The best way to “steelman” Cowen’s argument is not to figure out a number to assign to the AI risk scenario according to Cowen’s logic. It’s to explain why assigning a number isn’t a useful way to think about AI risk. Here’s my steelman of Cowen:
I don’t know what will happen with AI. No one does. But if you want me to worry about AI risk, you need to give an argument that AI risk is something particularly worth worrying about. There’s an infinite number of imaginable futures; why focus on this particular apocalyptic scenario? It’s dramatic, and that makes it fun to think about in some ways. But “X is dramatic and fun to think about” doesn’t entail “X will happen.”
Now of course the AI Doomers have given arguments. I simply find them unpersuasive. They don’t read like serious attempts to figure out what the future course of events will be, they read like rampant speculation by some (admittedly very smart) people. And it’s no surprise that these arguments don’t convince. Figuring out what the future will be like is hard even under ideal conditions. These conditions are not even close to ideal! Not only are we dealing with very new technology, we’re dealing with technology that may usher in a technological singularity, a point where technology is so transformative that the future course of events becomes entirely unknowable from the pre-singularity perspective.
So what will happen? I don’t know. Neither do you.
What number do I put on the AI doomer future? I put none. This is not risk. It’s Knightian uncertainty, through and through.
That seems pretty reasonable to me.
The thing that really struck me about Siskind’s attack on Cowen is the absolutely muddled way that Siskind talks about probabilities. This is not a unique problem for Siskind - pretty much everyone talks about probability in a muddled way.3 But Siskind's attack is just a torrent of haphazardly applied probabilistic concepts. For example, Siskind writes:
I said before my chance of existential risk from AI is 33%; that means I think there’s a 66% chance it won’t happen. In most futures, we get through okay, and Tyler gently ribs me for being silly.
This looks like probability the probability of P is some measure of the proportion of “futures” in which P is true. That’s a dubious interpretation of probability on its face - I’m inclined to say that there is one future, and in that future AI either kills everyone or it doesn’t. But even if we accept this interpretation of probability, a few paragraphs later we find Siskind saying: “If you have total uncertainty about a statement (“are bloxors greeblic?”), you should assign it a probability of 50%.”
Really? For one thing, I’m not sure it makes sense to ascribe any probability to the sentence “Bloxors are greeblic” because that sentence isn’t just uncertain, it’s nonsense. But most importantly, this claim makes no sense when put together with the idea that probabilities are proportions of futures. Consider the general argument form:
I have no idea whether or not P will happen.
Therefore, P is 50% probable.
Therefore, P happens in half of all futures.
That’s a useful argument to consider because it’s clearly invalid, and the reason why it’s invalid is that it clearly runs together two different conceptions of probability. So Siskind is operating with (at least) two different conceptions of probability here. That’s guaranteed to be a source of confusion in his arguments.
I also find it highly dubious that complete ignorance should result in a probability of 50%. Elsewhere in the article, Siskind warns against making “a fake assumption that you’ve parceled out scenarios of equal specificity… and likelihood.” But why assume that P and not-P are equally probable prior to all evidence? That’s a “fake” (I’d say “false”) assumption. After all, according to orthodox subjective Bayesianism, you can assign any probability you want to a claim prior to all evidence.
Now one could try to supplement subjective Bayesianism with some sort of indifference principle. But indifference principles are controversial and paradoxical, not least because they rely on the false assumption that you can parcel out scenarios of equal likelihood prior to receiving any evidence. And indifference principles don’t imply that everyone should regard every proposition as 50% probable prior to all evidence. That way lies absurdity and paradox.4 When you have absolutely no evidence whatsoever whether or not P is the case, it’s not uniquely rational to regard P as 50% probable.
So I just don’t see how Siskind’s position is more epistemically rational than Cowen’s. Cowen has made a prediction: AI apocalypse won’t happen. He’s given reasons for it: most highly specific predictions about the future are false, so the burden of proof is on the AI doomer to show that their scenario is likely, and that burden hasn’t been discharged. Siskind’s main objection is that Cowen hasn’t put a number to his prediction. But a) numbers don’t make for better predictions, b) Siskind talks about probability in incoherent ways (although so does everyone else), and c) Siskind’s own number of 33% seems pretty arbitrary.
In Charles Stross’s Singularity Sky, an AI called “the Eschaton” bootstraps its way to superintelligence, becoming all-knowing and all-powerful and thus, in effect, God. Its first act as God is to develop and build teleportation technology in an instant and teleport humankind off of Earth and onto a vast diaspora of planets around the galaxy. (Why? The Eschaton works in mysterious ways.) Will that happen with the advent of strong AI? I’ve got no particular reason to think that it will. So should we call that 50/50? 33%? 0%? I don’t have a number, and it seems silly to think that I’m only epistemically responsible if I do have one. I just don’t think it will happen. Neither do I think AI will kill us all. I don’t have a number for that prediction either. How could I?
I’ll refer to him as “Siskind” throughout the post. Most people call him “Scott,” but that feels weird to me; I love his writing, but we are not friends.
Read the post where Siskind comes up with the 33% number. There’s a lot of interesting musing in there, but there’s not really any math. Where does 33% come from? It’s the SMBC comic all over again. At least try to do what Rohit Krishnan does here. The Drake Equation is sophistical but at least it tries to come up with a number by actually doing math.
Alan Hajek: “Each interpretation [of probability] that we have canvassed seems to capture some crucial insight into a concept of it, yet falls short of doing complete justice to this concept. Perhaps the full story about probability is something of a patchwork, with partially overlapping pieces and principles about how they ought to relate.” In other words, there’s no conception of probability that is well-suited to all of the ways that people try to use probability theory. If you use the notion of probability in all the usual ways, you’re bound to say some incoherent things.
Consider every claim of the form “When I roll this die, the number showing will be no number other than n.” If you assign 50% probability to every claim of that form, your probabilities will add up to something much higher than 100%. We could attempt to avoid this result by qualifying the proposed principle in some way. “If you have no evidence whatsoever regarding P, and P is one of The Special Propositions, then P is 50% probable” avoids this result because we can say that propositions of the form “When I roll this die, the number showing will be no number other than n” are not Special Propositions. But then what are the special propositions? And why think that a qualified principle like this will be in any way relevant to the question of AI risk?