I can see both sides of this, but have a few thoughts.
I think the debate reveals something deeper about people. Predictions aren’t prophecy, just more or less educated guesswork. And we all engage in predictions daily. But we rarely examine the assumptions that went into our predictions unless (sometimes even if) we turn out to be catastrophically wrong (e.g. get in a traffic accident, lose our job, or end up in jail). And then only in hindsight. Silver does that examination for a living, and before knowing the outcome.
This is really the essence of luck – getting our predictions wrong. People who go more or less all in – in poker, traffic, businesses – when they have a 28% chance of losing, have a lot of “bad luck”. People and businesses that are too cautious miss out on “good luck”. But people and businesses that are well calibrated, and properly read tricky situations, will do better in the long run. (Of course, in order to break out of average, you need to make some bets that look to others like long shots, but good long shots are based on unique insight, not lucky gambles.)
Which brings me to my next point: What is news/media (and Nate Silver’s role in it) *for*? What are we paying for? Many people follow the news mostly for the entertainment value and to have something to talk about. They like to read true crime and true horror stories, and to talk about silly celebrities (including business and political celebrities, and prophets like Silver). To them, Silver was wrong. He failed to prophesize Trump's victory.
But there’s more value in news/information that helps you make good decisions – to decide whether to move production to China, to rent or buy, to invest in Pharma companies or chip manufacturers, to take a short cut or stay on the highway, to plant your tomatoes or wait another week or two. To people who use news to make decisions, Silver was right. If you made good sets of decisions based on a 28% chance that Trump would win, hedging and managing risk properly, you’d have beaten the market. (Of course, people were also wrong about what a Trump win would mean, and there was a lot of bad decision making going around, but not because of Silver.)
So you're right that Silver is a bad prophet, but I think he might be a good advisor.
I think this is missing the most important thing to be said in defense of Silver. He gave Trump a much higher chance of winning than pretty much everybody else who was putting numbers to these things. Sam Wang, for example, really was massively discredited by 2016. He gave Clinton a >99% chance of winning. And the reason why Silver's probability was so much lower was based on genuine insight. Roughly, he appreciated how various states were correlated with each other, so he recognized the possibility of a systemic error (e.g., we're underestimating Trump support in the midwest) that Wang's model implicitly ruled out (by treating probabilities of various states going Trump as independent, so that the probability of *all* of them going Trump was vanishingly small). It's stuff like that that makes me think 2016, in retrospect, looks comparatively pretty good for Silver. Most people aren't in the same game as him--making a lot of forecasts and putting numbers on them. The people that are, did worse.
I'm on board with saying that Silver did better than other pollsters in 2016, in some important sense. "Silver saw that results in the upper midwest could be correlated" is a genuinely good defense of Silver's punditry in 2016. "Things that are 28% likely happen 28% of the time, so he wasn't wrong" is not.
Exactly. And that scales to his entire body of work in this space. Silver isn't a soothsayer, his work has nothing to do with determining who is going to win per se. The project is HOW TO THINK ABOUT THE QUESTION of who is going to win, and in that he is a transformational figure toward greater analytical rigor and thereby accuracy.
To over-do a metaphor, forget the fishes he's given us, it's that he's taught us how to fish that's important. Just average all the polls. Average all the polls with credence to the margin of error and you're doing dramatically better than almost all of the punditry in modern American political history (which was not less-good 2016 models, it was "I saw a lot of yard signs in my neighborhood"). Everything from there is ultimately just fiddling on the margin.
I'm not persuaded by this. If Silver's project is just "teaching people how to think about the question," then why does he describe it as forecasting and why does he regularly publish things that he calls forecasts? He has lots of meta-level discussion that is at least an improvement on the meta level commitments of other forecasters. But he's also making object- level claims that are, I think, conceptually confused. He's not unique in being confused - far from it! I beat up on him a bit because he's a very prominent exemplar of this particular conceptual confusion.
I don't think the claims are conceptually confused, they're just somewhat more humble than how they get used.
You are right to say "he didn't get 2008 'right' either", it's all just creating probabilities for very low sample size events.
But where we were in 2008 there was a real wonder in the discourse whether the Obama phenomenon could actually be real. Silver's contribution was just clarity in saying "yes, reams and reams of opinion polling are crystal clear on the overwhelming likelihood of this young African-American liberal Democrat winning these unimaginable demographics in these unimaginable places". Sounds stupid, but we were operating at a much lower level of sophistication even that recently. Nate Silver changed that.
Yeah, I'm with Daniel. Basically everybody got 2016 wrong; Silver got it less wrong than most did, while having got 2008 brilliantly right, which is why I still think he's probably the most trustworthy forecaster out there. The 2016 response is not "he wasn't wrong", it was "he was less wrong."
The people I'm criticizing are not the ones saying "he was less wrong." They're the ones saying "he wasn't wrong and if you think he was, you don't understand probability." I'm pretty sure Silver himself is in that camp.
Note that it does cite pre-2016 evidence to suggest that yes, in fact he was giving probabilities all along and the people thinking this was the same as predictions were misreading him even in the face of his baldly stating otherwise. e.g.
"I hope you’ll excuse the cliché, but it’s appropriate here: in poker, making an inside straight requires you to catch one of 4 cards out of 48 remaining in the deck, the chances of which are about 8 percent. Those are now about Mr. Romney’s chances of winning the Electoral College, according to the FiveThirtyEight forecast.
As any poker player knows, those 8 percent chances do come up once in a while. If it happens this year, then a lot of polling firms will have to re-examine their assumptions — and we will have to re-examine ours about how trustworthy the polls are. But the odds are that Mr. Obama will win another term."
Silver always uses poker as an analogy (he's a hardcore poker player), but this is precisely the kind of slipperiness I'm criticizing! Like, hitting that inside straight is about 8%. If you realize that, play accordingly, and then the inside straight hits, you wouldn't have to re-examine any of your assumptions. That's just what happens! Sometimes the straight hits! And so if the 2012 election were precisely analogous, why would pollsters have to re-examine any of their assumptions if Romney won? Why would 538 have to reassess, too? (That's just what happens! Sometimes Romney wins!)
You're right in a sense that simple examples of probability usually just assume a known distribution, and so questioning the distribution doesn't really come up. If we really wanted a more rigorous poker analogy, we'd probably have to talk about how poker players guess the probability of different hands their opponents might have (something that can be estimated from general player tendencies, frequency of specific hands dealt, and known information about a specific player, but which ultimately has one "correct" answer in a given hand)
I guess a simpler analogy might be if you had a coin but you don't know if the coin is biased or fair. Guessing the initial flip is no different from a known fair coin (unless you have specific reason to believe a hypothetical bias is more likely to tilt one way or the other) but if you start seeing enough flips go the same way this should affect estimated odds on future flips.
If we change the problem in this way...I guess I'm not fully seeing the line of argument here. Is this an argument against Bayesian reasoning entirely?
I don't know what "Bayesian reasoning" is supposed to be. Bayes' Theorem is a theorem; it's trivial given the ratio definition of conditional probability. If Bayesian reasoning is just reasoning that takes Bayes' Theorem into account, then that's all reasoning that's not logically contradictory. When I start asking for a better account of what Bayesian reasoning is supposed to be, people just start describing inference to the best explanation.
This is a bad way to stretch the analogy. The reason he says that pollsters would have to re-examine their assumptions is because the only way that the 8% chance occurs is if the polls are very far off from the actual result, and that only happens if the pollsters make mistaken assumptions. Silver's model takes into account the possibility that they make incorrect assumptions - that's why he assigns a nonzero probability to this.
In the case of an inside straight, the 8% chance is purely based on which cards happen to be dealt, not on whether some external data source has made the correct assumptions.
As for whether 538 would have to re-assess, too: Yes, he explicitly says that he would. "We will have to re-examine ours about how trustworthy the polls are." This would presumably come in the form of higher uncertainty about how close the polling results will match the final results.
This just affirms my larger point, which is that Silver is wrong to rely on the gambling analogy, since the case of the inside straight and the case of election forecasting are strikingly different in many significant ways. His "I'm a gambler, and I'm just telling you the odds" schtick doesn't withstand scrutiny.
As one of the Nate Silver defenders, I don't think this article really addresses the point well. You make a big deal out of the distinction between predicting something with X% confidence, and saying that there's an X% chance of that thing happening. But there is no distinction here, and if there is, it's a distinction without a difference - if I think Y has an X% chance of happening, then I predict that it will happen with X% confidence. It doesn't matter whether you want to call his 2016 forecast a prediction or not. The point is that he was not very confident in it, so pointing out that the prediction was wrong does very little to affect his credibility. But the "unwashed masses" completely ignore this. Every time Nate Silver says anything, they just shout, "He was wrong about 2016!", while completely ignoring all his correct predictions and the low degree of confidence in this one. Nate Silver never claimed to be a psychic, but they treat him as if he had.
I'm perfectly happy to accept that he got 2016 wrong if, by "wrong," you mean that his prediction didn't match the final results. The problem is that the Silver haters don't mean this when they say he got 2016 wrong. When they say this, they mean that his model is wrong, or that the probability he gave must not have been justified by the available data. After all, they are trying to use this to discredit his predictions. But neither of these claims are supported by the fact that an event which he thought had a 29% chance of happening ended up happening. Based on the available data, it would have been unreasonable to predict a Trump win, even though that was the final result, because none of the data suggested that was more likely to happen than a Clinton win. And a p-value of 0.29 is not enough to reject the reliability of his model by any sane standard, especially when that comes from cherry-picking the single example where he got it wrong (i.e., it's p-hacked). If he had given a much higher than 71% chance of Clinton winning, then it would be perfectly reasonable to conclude that Nate Silver's model was wrong and shouldn't be trusted to give accurate probabilities in the future. This is the case, for example, for models that predicted a >99% chance of Clinton winning.
Regarding the 2008 prediction specifically, as well as 2012 where he predicted every single state correctly, it's true to say that each individual state prediction is only weak evidence that Silver's model is accurate, in the same way that the 2016 prediction is only weak evidence for its inaccuracy. In fact, if his model consistently performed as well as it did in 2012 and 2008, that would actually be evidence that his probabilities are wrong, since they would be underconfident. But as Nate Silver said after those two elections, his model wouldn't always get that many states correct - it just got lucky those times. And to tell whether his model is good or not, you have to look at the aggregate of all the predictions it made, not just single one out and endlessly repeat, "But he got that one wrong!" no matter what other evidence is pointed out.
I trust Silver's model more than any other election prediction for three reasons:
1. He explains all the assumptions that go into it and the justification for those assumptions. I think these are all reasonable, so his model has a high prior probability of being accurate.
2. His model is well-calibrated based on previous predictions. As you say, this isn't a perfect measure, but it's still the best measure there is. It's certainly a better measure than cherry-picking the wrong predictions and claiming they prove his model wrong, or cherry-picking the correct predictions and saying they prove his model right.
3. His model has performed way better than any other. Predicting so many states right in 2008 and 2012 is obviously an example of this, but his 2016 forecast is an example, too. There was no other model that I know of that gave Trump a higher chance of winning. Even in 2020, when all models got the winner correct, his model thought there was still a significant chance that the election would be close. Others didn't seem to think so.
"I'm perfectly happy to accept that he got 2016 wrong if, by "wrong," you mean that his prediction didn't match the final results." Yes, that's what I mean. My target is people who don't think that Silver got 2016 wrong.
I think it's consistent to say that Nate Silver never gets anything right or wrong because he's not really in the business of predicting. It's also consistent to say that Silver got lots of elections right but got 2016 wrong. It's not consistent to say the former thing about 2016 but the latter thing about every other election.
Calibration is not the best we can do. Breier scores are better. See James Joyce's "A Nonpragmatic Vindication of Probabilism."
I think that a lot of the work of analyzing cause and effect that Silver does is extremely valuable. But that value comes from his attempts to understand and explain cause and effect. Capping things off with a probability estimate is of dubious value. What, if anything, does that number correspond to in reality? Cause and effect are real. Frequencies are real. Probabilities aren't.
I guess I think of a Breier score as just a more refined version of calibration. And if the critics of Nate Silver think he has a bad Breier score, they should actually prove that instead of just cherry-picking the elections he got (barely) wrong and yelling about them like a broken record. That's what really annoys me about them - the fact that they point out a few examples where he was wrong and completely ignore the probability he gave. In their minds, there is no difference between predicting a 49% chance of something that actually happens and a 0% chance.
> Cause and effect are real. Frequencies are real. Probabilities aren't.
This is itself a very dubious philosophical assumption. You seem to accept the existence of probabilities in cases like a card game. A probability is a quantification of uncertainty. We are uncertain about the result of a presidential election that hasn't happened yet, so it makes perfect sense to apply a probability to it.
I accept the existence of probabilities in a card game only to the extent that there is one ace of spades out of a deck of 52 cards. 1/52. Call that "probability" if you want. That's not a quantification of uncertainty, it's a ratio of material quantities, empirically verifiable.
It makes no sense to "predict a 49% chance of something happening." There is no state of affairs in the world that corresponds to a 49% chance of something happening, and thus it makes no sense to predict it. Things happen or they don't.
> I accept the existence of probabilities in a card game only to the extent that there is one ace of spades out of a deck of 52 cards.
This is not actually true. If you suspected that the dealer might be cheating and trying to give you the Ace of Spades, you certainly wouldn't say there's only a 1/52 chance of getting it. You only accept that the probability of drawing the Ace of Spades is 1/52 because you accept an additional premise: That all cards are equally likely. There's no way to make sense of that premise just as a ratio of material quantities.
And even if you don't like to quantify your uncertainty, it's absurd to say that just because you don't do it, no one can, and anyone who tries is doing something completely meaningless. If probabilities have no meaning at all, then you can't make sense of Breier scores - what on Earth are they measuring? You can't even make sense of any form of decision-making under uncertainty.
> There is no state of affairs in the world that corresponds to a 49% chance of something happening
There's no state of affairs that corresponds to a 1/52 chance of drawing the Ace of Spades either - it's either on the top of the deck, or it's not. The point is that you don't know which state of affairs you're in.
I don't accept that all cards are equally likely. I can understand the claim that there is a 1/52 chance that the top card is the ace of spades if that claim is equivalent to "There are 52 cards in the deck, one of which is the ace of spades." If that isn't true - because the dealer is cheating, or whatever - then there's not a 1/52 chance in any meaningful sense. If I think there is, I'm wrong. But I'm just wrong about the simple question of how many cards are in the deck.
I ultimately don't think Breier scores are any good either. They're just better than calibration if someone wants to go in for a measure of credal accuracy. And I don't really think there is a way to make sense of decision-making under uncertainty. I mean, we try to figure out what will happen, and we do the best we can. That's hopelessly vague, of course, but I don't think there's anything more that can be said. I think that all of formal decision theory is fundamentally nonsense.
The only possible state of affairs that could correspond to "a 1/52 chance of drawing the ace of spades" is the state of affairs where there's 52 cards in the deck and one is the ace. But yeah, it's either on the top or it's not and we don't know which.
More generally, I'm basically an error theorist about probability. There are some states of affairs that are kind of close to what we mean by probability, and out of a sense of charity I'll allow that there's a sense in which claims about probability can be true if we understand them as just being about those knowable, material states of affairs. But if you try to point out that those aren't REALLY what we mean by "probability," I'll just agree, because I don't think there's anything that corresponds to the common sense notion of probability. I've thought this through; I'm willing to bite all the bullets here, and I'm a little mystified why other people who've thought this through aren't willing to bite all the same bullets because it seems absurd to do otherwise.
Also, I'm not sure what good it is to critique Nate Silver's probabilistic model if you don't even believe in probability in the first place. If you don't believe in probability, and even Breier scores are nonsense to you, there's no point criticizing his model's performance because you'd think it's all BS no matter what.
> I don't accept that all cards are equally likely.
If you don't accept this claim, then you can't say anything about the probability of drawing the Ace of Spades, since all your reasoning behind that assumes that the cards are equally likely.
> I can understand the claim that there is a 1/52 chance that the top card is the ace of spades if that claim is equivalent to "There are 52 cards in the deck, one of which is the ace of spades."
But that claim is not equivalent. It's certainly not logically equivalent, and the only way to make it equivalent is under the assumption that all 52 cards are equally likely. Since you don't accept this assumption, it's never equivalent for you.
> If that isn't true - because the dealer is cheating, or whatever - then there's not a 1/52 chance in any meaningful sense.
But why does the dealer cheating mean there's no longer a 1/52 chance? By your lights, it changes nothing.
> But I'm just wrong about the simple question of how many cards are in the deck.
The dealer cheating doesn't mean there's a different number of cards in the deck! The dealer can cheat by covertly arranging the cards in the order he wants them.
> And I don't really think there is a way to make sense of decision-making under uncertainty.
This is a crazy admission to make. Since all practical decision-making involves uncertainty, an inability to make sense of it means we can't make sense of any real decisions. Rejecting that type of analysis would make us completely ineffective at doing any practical tasks whatsoever.
> I mean, we try to figure out what will happen, and we do the best we can. That's hopelessly vague, of course, but I don't think there's anything more that can be said. I think that all of formal decision theory is fundamentally nonsense.
I'm not sure how you can call formal decision theory "fundamentally nonsense" when your alternative is a maximally vague statement that avoids saying anything meaningful about decision-making. I don't know how it can get more "nonsense" than a non-answer. And to whatever extent it is meaningful, it's because it's a rough approximation to formal decision theory! "Figure out what will happen" is the informal version of "evaluate probabilities," and "do the best we can" means to perform some type of utility maximization. The only difference is that it's done informally, rather than with numbers. The best humans can do without computational tools is to trust their informal judgements, but once you have those tools, there's no reason you can't start using numbers for the things you can quantify.
> The only possible state of affairs that could correspond to "a 1/52 chance of drawing the ace of spades" is the state of affairs where there's 52 cards in the deck and one is the ace. But yeah, it's either on the top or it's not and we don't know which.
But neither of these states of affairs corresponds to a 1/52 chance. Out of the 52! possibilities this encompasses, there are 51! that correspond to a 100% chance of drawing the Ace of Spades and 51*51! that correspond to a 0% chance. Or at least, by your understanding of the probability, that should be the case. After all, if you knew that the Ace of Spades was on top, you wouldn't still say that there's only a 1/52 chance of drawing it.
"There's no state of affairs that corresponds to a 49% chance of something happening," applies to a deck of cards just as much as it does to who will be the next president. Probability is not a property of a state of affairs, but of the information we have about it.
> I've thought this through; I'm willing to bite all the bullets here, and I'm a little mystified why other people who've thought this through aren't willing to bite all the same bullets because it seems absurd to do otherwise.
I think it's obvious why most people don't want to bite a bullet so big that it blows your head off. Anyone who actually behaved as if there was no such thing as probability would be completely insane. There's not even any way to account for the existence of beliefs without probabilities - What is a belief other than something you judge to be more probable than not? And without beliefs, you can't even follow the vague account of decision-making you gave. Even if you somehow can account for beliefs but still reject credences (which you'll have to do if you think probability is nonsense), then there's no possible way to explain what risk is and no possible way to incorporate it into your decision making. Nor is there any way to explain why it's better to make a decision that you're more certain will benefit you even if you think the less certain benefit would be slightly better, etc.
And that's just the implications for practical decision-making. "Probability is nonsense" also undermines just about every piece of scientific knowledge we have. The entirety of the modern scientific method relies on statistics that assumes the existence of probability (p-values, uncertainty intervals, a "5 sigma result," etc. are all completely meaningless is probability is nonsense). Even something as simple as a survey means nothing without probability.
The reason to bite even just one of these bullets would have to be massive, given how much they undermine and how absurd their implications are. But the only reasons you've given are stunningly weak - they amount to a mischaracterization of what probability is and incredulity about what it could be.
I can see both sides of this, but have a few thoughts.
I think the debate reveals something deeper about people. Predictions aren’t prophecy, just more or less educated guesswork. And we all engage in predictions daily. But we rarely examine the assumptions that went into our predictions unless (sometimes even if) we turn out to be catastrophically wrong (e.g. get in a traffic accident, lose our job, or end up in jail). And then only in hindsight. Silver does that examination for a living, and before knowing the outcome.
This is really the essence of luck – getting our predictions wrong. People who go more or less all in – in poker, traffic, businesses – when they have a 28% chance of losing, have a lot of “bad luck”. People and businesses that are too cautious miss out on “good luck”. But people and businesses that are well calibrated, and properly read tricky situations, will do better in the long run. (Of course, in order to break out of average, you need to make some bets that look to others like long shots, but good long shots are based on unique insight, not lucky gambles.)
Which brings me to my next point: What is news/media (and Nate Silver’s role in it) *for*? What are we paying for? Many people follow the news mostly for the entertainment value and to have something to talk about. They like to read true crime and true horror stories, and to talk about silly celebrities (including business and political celebrities, and prophets like Silver). To them, Silver was wrong. He failed to prophesize Trump's victory.
But there’s more value in news/information that helps you make good decisions – to decide whether to move production to China, to rent or buy, to invest in Pharma companies or chip manufacturers, to take a short cut or stay on the highway, to plant your tomatoes or wait another week or two. To people who use news to make decisions, Silver was right. If you made good sets of decisions based on a 28% chance that Trump would win, hedging and managing risk properly, you’d have beaten the market. (Of course, people were also wrong about what a Trump win would mean, and there was a lot of bad decision making going around, but not because of Silver.)
So you're right that Silver is a bad prophet, but I think he might be a good advisor.
I think this is missing the most important thing to be said in defense of Silver. He gave Trump a much higher chance of winning than pretty much everybody else who was putting numbers to these things. Sam Wang, for example, really was massively discredited by 2016. He gave Clinton a >99% chance of winning. And the reason why Silver's probability was so much lower was based on genuine insight. Roughly, he appreciated how various states were correlated with each other, so he recognized the possibility of a systemic error (e.g., we're underestimating Trump support in the midwest) that Wang's model implicitly ruled out (by treating probabilities of various states going Trump as independent, so that the probability of *all* of them going Trump was vanishingly small). It's stuff like that that makes me think 2016, in retrospect, looks comparatively pretty good for Silver. Most people aren't in the same game as him--making a lot of forecasts and putting numbers on them. The people that are, did worse.
I'm on board with saying that Silver did better than other pollsters in 2016, in some important sense. "Silver saw that results in the upper midwest could be correlated" is a genuinely good defense of Silver's punditry in 2016. "Things that are 28% likely happen 28% of the time, so he wasn't wrong" is not.
Exactly. And that scales to his entire body of work in this space. Silver isn't a soothsayer, his work has nothing to do with determining who is going to win per se. The project is HOW TO THINK ABOUT THE QUESTION of who is going to win, and in that he is a transformational figure toward greater analytical rigor and thereby accuracy.
To over-do a metaphor, forget the fishes he's given us, it's that he's taught us how to fish that's important. Just average all the polls. Average all the polls with credence to the margin of error and you're doing dramatically better than almost all of the punditry in modern American political history (which was not less-good 2016 models, it was "I saw a lot of yard signs in my neighborhood"). Everything from there is ultimately just fiddling on the margin.
I'm not persuaded by this. If Silver's project is just "teaching people how to think about the question," then why does he describe it as forecasting and why does he regularly publish things that he calls forecasts? He has lots of meta-level discussion that is at least an improvement on the meta level commitments of other forecasters. But he's also making object- level claims that are, I think, conceptually confused. He's not unique in being confused - far from it! I beat up on him a bit because he's a very prominent exemplar of this particular conceptual confusion.
I don't think the claims are conceptually confused, they're just somewhat more humble than how they get used.
You are right to say "he didn't get 2008 'right' either", it's all just creating probabilities for very low sample size events.
But where we were in 2008 there was a real wonder in the discourse whether the Obama phenomenon could actually be real. Silver's contribution was just clarity in saying "yes, reams and reams of opinion polling are crystal clear on the overwhelming likelihood of this young African-American liberal Democrat winning these unimaginable demographics in these unimaginable places". Sounds stupid, but we were operating at a much lower level of sophistication even that recently. Nate Silver changed that.
Yeah, I'm with Daniel. Basically everybody got 2016 wrong; Silver got it less wrong than most did, while having got 2008 brilliantly right, which is why I still think he's probably the most trustworthy forecaster out there. The 2016 response is not "he wasn't wrong", it was "he was less wrong."
The people I'm criticizing are not the ones saying "he was less wrong." They're the ones saying "he wasn't wrong and if you think he was, you don't understand probability." I'm pretty sure Silver himself is in that camp.
Legit. I just want to make that clarification on behalf of the "less wrong" camp, seeing as I'm in it. :)
I strongly recommend checking out this link, which has become my go-to whenever this subject comes up: https://goodreason.substack.com/p/nate-silvers-finest-hour-part-1-of
Note that it does cite pre-2016 evidence to suggest that yes, in fact he was giving probabilities all along and the people thinking this was the same as predictions were misreading him even in the face of his baldly stating otherwise. e.g.
"I hope you’ll excuse the cliché, but it’s appropriate here: in poker, making an inside straight requires you to catch one of 4 cards out of 48 remaining in the deck, the chances of which are about 8 percent. Those are now about Mr. Romney’s chances of winning the Electoral College, according to the FiveThirtyEight forecast.
As any poker player knows, those 8 percent chances do come up once in a while. If it happens this year, then a lot of polling firms will have to re-examine their assumptions — and we will have to re-examine ours about how trustworthy the polls are. But the odds are that Mr. Obama will win another term."
Silver always uses poker as an analogy (he's a hardcore poker player), but this is precisely the kind of slipperiness I'm criticizing! Like, hitting that inside straight is about 8%. If you realize that, play accordingly, and then the inside straight hits, you wouldn't have to re-examine any of your assumptions. That's just what happens! Sometimes the straight hits! And so if the 2012 election were precisely analogous, why would pollsters have to re-examine any of their assumptions if Romney won? Why would 538 have to reassess, too? (That's just what happens! Sometimes Romney wins!)
You're right in a sense that simple examples of probability usually just assume a known distribution, and so questioning the distribution doesn't really come up. If we really wanted a more rigorous poker analogy, we'd probably have to talk about how poker players guess the probability of different hands their opponents might have (something that can be estimated from general player tendencies, frequency of specific hands dealt, and known information about a specific player, but which ultimately has one "correct" answer in a given hand)
I guess a simpler analogy might be if you had a coin but you don't know if the coin is biased or fair. Guessing the initial flip is no different from a known fair coin (unless you have specific reason to believe a hypothetical bias is more likely to tilt one way or the other) but if you start seeing enough flips go the same way this should affect estimated odds on future flips.
If we change the problem in this way...I guess I'm not fully seeing the line of argument here. Is this an argument against Bayesian reasoning entirely?
I don't know what "Bayesian reasoning" is supposed to be. Bayes' Theorem is a theorem; it's trivial given the ratio definition of conditional probability. If Bayesian reasoning is just reasoning that takes Bayes' Theorem into account, then that's all reasoning that's not logically contradictory. When I start asking for a better account of what Bayesian reasoning is supposed to be, people just start describing inference to the best explanation.
This is a bad way to stretch the analogy. The reason he says that pollsters would have to re-examine their assumptions is because the only way that the 8% chance occurs is if the polls are very far off from the actual result, and that only happens if the pollsters make mistaken assumptions. Silver's model takes into account the possibility that they make incorrect assumptions - that's why he assigns a nonzero probability to this.
In the case of an inside straight, the 8% chance is purely based on which cards happen to be dealt, not on whether some external data source has made the correct assumptions.
As for whether 538 would have to re-assess, too: Yes, he explicitly says that he would. "We will have to re-examine ours about how trustworthy the polls are." This would presumably come in the form of higher uncertainty about how close the polling results will match the final results.
This just affirms my larger point, which is that Silver is wrong to rely on the gambling analogy, since the case of the inside straight and the case of election forecasting are strikingly different in many significant ways. His "I'm a gambler, and I'm just telling you the odds" schtick doesn't withstand scrutiny.
As one of the Nate Silver defenders, I don't think this article really addresses the point well. You make a big deal out of the distinction between predicting something with X% confidence, and saying that there's an X% chance of that thing happening. But there is no distinction here, and if there is, it's a distinction without a difference - if I think Y has an X% chance of happening, then I predict that it will happen with X% confidence. It doesn't matter whether you want to call his 2016 forecast a prediction or not. The point is that he was not very confident in it, so pointing out that the prediction was wrong does very little to affect his credibility. But the "unwashed masses" completely ignore this. Every time Nate Silver says anything, they just shout, "He was wrong about 2016!", while completely ignoring all his correct predictions and the low degree of confidence in this one. Nate Silver never claimed to be a psychic, but they treat him as if he had.
I'm perfectly happy to accept that he got 2016 wrong if, by "wrong," you mean that his prediction didn't match the final results. The problem is that the Silver haters don't mean this when they say he got 2016 wrong. When they say this, they mean that his model is wrong, or that the probability he gave must not have been justified by the available data. After all, they are trying to use this to discredit his predictions. But neither of these claims are supported by the fact that an event which he thought had a 29% chance of happening ended up happening. Based on the available data, it would have been unreasonable to predict a Trump win, even though that was the final result, because none of the data suggested that was more likely to happen than a Clinton win. And a p-value of 0.29 is not enough to reject the reliability of his model by any sane standard, especially when that comes from cherry-picking the single example where he got it wrong (i.e., it's p-hacked). If he had given a much higher than 71% chance of Clinton winning, then it would be perfectly reasonable to conclude that Nate Silver's model was wrong and shouldn't be trusted to give accurate probabilities in the future. This is the case, for example, for models that predicted a >99% chance of Clinton winning.
Regarding the 2008 prediction specifically, as well as 2012 where he predicted every single state correctly, it's true to say that each individual state prediction is only weak evidence that Silver's model is accurate, in the same way that the 2016 prediction is only weak evidence for its inaccuracy. In fact, if his model consistently performed as well as it did in 2012 and 2008, that would actually be evidence that his probabilities are wrong, since they would be underconfident. But as Nate Silver said after those two elections, his model wouldn't always get that many states correct - it just got lucky those times. And to tell whether his model is good or not, you have to look at the aggregate of all the predictions it made, not just single one out and endlessly repeat, "But he got that one wrong!" no matter what other evidence is pointed out.
I trust Silver's model more than any other election prediction for three reasons:
1. He explains all the assumptions that go into it and the justification for those assumptions. I think these are all reasonable, so his model has a high prior probability of being accurate.
2. His model is well-calibrated based on previous predictions. As you say, this isn't a perfect measure, but it's still the best measure there is. It's certainly a better measure than cherry-picking the wrong predictions and claiming they prove his model wrong, or cherry-picking the correct predictions and saying they prove his model right.
3. His model has performed way better than any other. Predicting so many states right in 2008 and 2012 is obviously an example of this, but his 2016 forecast is an example, too. There was no other model that I know of that gave Trump a higher chance of winning. Even in 2020, when all models got the winner correct, his model thought there was still a significant chance that the election would be close. Others didn't seem to think so.
"I'm perfectly happy to accept that he got 2016 wrong if, by "wrong," you mean that his prediction didn't match the final results." Yes, that's what I mean. My target is people who don't think that Silver got 2016 wrong.
I think it's consistent to say that Nate Silver never gets anything right or wrong because he's not really in the business of predicting. It's also consistent to say that Silver got lots of elections right but got 2016 wrong. It's not consistent to say the former thing about 2016 but the latter thing about every other election.
Calibration is not the best we can do. Breier scores are better. See James Joyce's "A Nonpragmatic Vindication of Probabilism."
I think that a lot of the work of analyzing cause and effect that Silver does is extremely valuable. But that value comes from his attempts to understand and explain cause and effect. Capping things off with a probability estimate is of dubious value. What, if anything, does that number correspond to in reality? Cause and effect are real. Frequencies are real. Probabilities aren't.
I guess I think of a Breier score as just a more refined version of calibration. And if the critics of Nate Silver think he has a bad Breier score, they should actually prove that instead of just cherry-picking the elections he got (barely) wrong and yelling about them like a broken record. That's what really annoys me about them - the fact that they point out a few examples where he was wrong and completely ignore the probability he gave. In their minds, there is no difference between predicting a 49% chance of something that actually happens and a 0% chance.
> Cause and effect are real. Frequencies are real. Probabilities aren't.
This is itself a very dubious philosophical assumption. You seem to accept the existence of probabilities in cases like a card game. A probability is a quantification of uncertainty. We are uncertain about the result of a presidential election that hasn't happened yet, so it makes perfect sense to apply a probability to it.
I accept the existence of probabilities in a card game only to the extent that there is one ace of spades out of a deck of 52 cards. 1/52. Call that "probability" if you want. That's not a quantification of uncertainty, it's a ratio of material quantities, empirically verifiable.
It makes no sense to "predict a 49% chance of something happening." There is no state of affairs in the world that corresponds to a 49% chance of something happening, and thus it makes no sense to predict it. Things happen or they don't.
> I accept the existence of probabilities in a card game only to the extent that there is one ace of spades out of a deck of 52 cards.
This is not actually true. If you suspected that the dealer might be cheating and trying to give you the Ace of Spades, you certainly wouldn't say there's only a 1/52 chance of getting it. You only accept that the probability of drawing the Ace of Spades is 1/52 because you accept an additional premise: That all cards are equally likely. There's no way to make sense of that premise just as a ratio of material quantities.
And even if you don't like to quantify your uncertainty, it's absurd to say that just because you don't do it, no one can, and anyone who tries is doing something completely meaningless. If probabilities have no meaning at all, then you can't make sense of Breier scores - what on Earth are they measuring? You can't even make sense of any form of decision-making under uncertainty.
> There is no state of affairs in the world that corresponds to a 49% chance of something happening
There's no state of affairs that corresponds to a 1/52 chance of drawing the Ace of Spades either - it's either on the top of the deck, or it's not. The point is that you don't know which state of affairs you're in.
I don't accept that all cards are equally likely. I can understand the claim that there is a 1/52 chance that the top card is the ace of spades if that claim is equivalent to "There are 52 cards in the deck, one of which is the ace of spades." If that isn't true - because the dealer is cheating, or whatever - then there's not a 1/52 chance in any meaningful sense. If I think there is, I'm wrong. But I'm just wrong about the simple question of how many cards are in the deck.
I ultimately don't think Breier scores are any good either. They're just better than calibration if someone wants to go in for a measure of credal accuracy. And I don't really think there is a way to make sense of decision-making under uncertainty. I mean, we try to figure out what will happen, and we do the best we can. That's hopelessly vague, of course, but I don't think there's anything more that can be said. I think that all of formal decision theory is fundamentally nonsense.
The only possible state of affairs that could correspond to "a 1/52 chance of drawing the ace of spades" is the state of affairs where there's 52 cards in the deck and one is the ace. But yeah, it's either on the top or it's not and we don't know which.
More generally, I'm basically an error theorist about probability. There are some states of affairs that are kind of close to what we mean by probability, and out of a sense of charity I'll allow that there's a sense in which claims about probability can be true if we understand them as just being about those knowable, material states of affairs. But if you try to point out that those aren't REALLY what we mean by "probability," I'll just agree, because I don't think there's anything that corresponds to the common sense notion of probability. I've thought this through; I'm willing to bite all the bullets here, and I'm a little mystified why other people who've thought this through aren't willing to bite all the same bullets because it seems absurd to do otherwise.
Also, I'm not sure what good it is to critique Nate Silver's probabilistic model if you don't even believe in probability in the first place. If you don't believe in probability, and even Breier scores are nonsense to you, there's no point criticizing his model's performance because you'd think it's all BS no matter what.
> I don't accept that all cards are equally likely.
If you don't accept this claim, then you can't say anything about the probability of drawing the Ace of Spades, since all your reasoning behind that assumes that the cards are equally likely.
> I can understand the claim that there is a 1/52 chance that the top card is the ace of spades if that claim is equivalent to "There are 52 cards in the deck, one of which is the ace of spades."
But that claim is not equivalent. It's certainly not logically equivalent, and the only way to make it equivalent is under the assumption that all 52 cards are equally likely. Since you don't accept this assumption, it's never equivalent for you.
> If that isn't true - because the dealer is cheating, or whatever - then there's not a 1/52 chance in any meaningful sense.
But why does the dealer cheating mean there's no longer a 1/52 chance? By your lights, it changes nothing.
> But I'm just wrong about the simple question of how many cards are in the deck.
The dealer cheating doesn't mean there's a different number of cards in the deck! The dealer can cheat by covertly arranging the cards in the order he wants them.
> And I don't really think there is a way to make sense of decision-making under uncertainty.
This is a crazy admission to make. Since all practical decision-making involves uncertainty, an inability to make sense of it means we can't make sense of any real decisions. Rejecting that type of analysis would make us completely ineffective at doing any practical tasks whatsoever.
> I mean, we try to figure out what will happen, and we do the best we can. That's hopelessly vague, of course, but I don't think there's anything more that can be said. I think that all of formal decision theory is fundamentally nonsense.
I'm not sure how you can call formal decision theory "fundamentally nonsense" when your alternative is a maximally vague statement that avoids saying anything meaningful about decision-making. I don't know how it can get more "nonsense" than a non-answer. And to whatever extent it is meaningful, it's because it's a rough approximation to formal decision theory! "Figure out what will happen" is the informal version of "evaluate probabilities," and "do the best we can" means to perform some type of utility maximization. The only difference is that it's done informally, rather than with numbers. The best humans can do without computational tools is to trust their informal judgements, but once you have those tools, there's no reason you can't start using numbers for the things you can quantify.
> The only possible state of affairs that could correspond to "a 1/52 chance of drawing the ace of spades" is the state of affairs where there's 52 cards in the deck and one is the ace. But yeah, it's either on the top or it's not and we don't know which.
But neither of these states of affairs corresponds to a 1/52 chance. Out of the 52! possibilities this encompasses, there are 51! that correspond to a 100% chance of drawing the Ace of Spades and 51*51! that correspond to a 0% chance. Or at least, by your understanding of the probability, that should be the case. After all, if you knew that the Ace of Spades was on top, you wouldn't still say that there's only a 1/52 chance of drawing it.
"There's no state of affairs that corresponds to a 49% chance of something happening," applies to a deck of cards just as much as it does to who will be the next president. Probability is not a property of a state of affairs, but of the information we have about it.
> I've thought this through; I'm willing to bite all the bullets here, and I'm a little mystified why other people who've thought this through aren't willing to bite all the same bullets because it seems absurd to do otherwise.
I think it's obvious why most people don't want to bite a bullet so big that it blows your head off. Anyone who actually behaved as if there was no such thing as probability would be completely insane. There's not even any way to account for the existence of beliefs without probabilities - What is a belief other than something you judge to be more probable than not? And without beliefs, you can't even follow the vague account of decision-making you gave. Even if you somehow can account for beliefs but still reject credences (which you'll have to do if you think probability is nonsense), then there's no possible way to explain what risk is and no possible way to incorporate it into your decision making. Nor is there any way to explain why it's better to make a decision that you're more certain will benefit you even if you think the less certain benefit would be slightly better, etc.
And that's just the implications for practical decision-making. "Probability is nonsense" also undermines just about every piece of scientific knowledge we have. The entirety of the modern scientific method relies on statistics that assumes the existence of probability (p-values, uncertainty intervals, a "5 sigma result," etc. are all completely meaningless is probability is nonsense). Even something as simple as a survey means nothing without probability.
The reason to bite even just one of these bullets would have to be massive, given how much they undermine and how absurd their implications are. But the only reasons you've given are stunningly weak - they amount to a mischaracterization of what probability is and incredulity about what it could be.