Why Pundits And Politicians Hate NYT Election Forecaster Nate Silver

The New York Times election oracle, Nate Silver, who in his fivethirtyeight blog correctly predicted 49 of 50 states in the last election, predicts that Obama could breeze to victory with a 75 percent chance of winning. Despite his relatively simple method of averaging polls to predict winners, he’s become a punching bag for pundits and politicians who label him a fradulent snakeoil salesman.

Why does Silver, who is really just an apartisan puzzle-solver, inspire so much loathing? Because his results reveal a psychologically disturbing fact: we live in an uncontrollable, unpredictable world. Obama is a moderately popular incumbent running against a relatively uncharismatic one who’s not that well liked even among conservatives. A rainy election day and upswing in the economy could do more to affect the small slice of undecided voters in swing states than all the newspaper endorsements and billion-dollar campaigns put together.

“Both sides understand that it is close, and it could go either way. And anybody that thinks that this race is anything but a tossup right now is such an ideologue, they should be kept away from typewriters, computers, laptops and microphones for the next 10 days, because they’re jokes,” said MSNBC’s hotheaded morning political pundit, Joe Scarborough, in a pointed rant against the humble New York Times statistician.

Buzzfeed thinks that Silver’s critics target him because he favors Obama. Washington Post Blogger, Ezra Klein, takes a more therapeutic interpretation, pointing to a caustic op-ed in Politico to argue that Silver threatens the very existence of pundits, since the success of his models make their opinions an antiquated information source.

Speaking as one of the few journalists with an advanced degree in mathematics, I think the most telling reason why so many criticize Silver was plainly stated by Democratic Minority Leader, Nancy Pelosi, when Daily Show host Jon Stewart asked her last week how she felt about Silver’s prediction that the House of Representatives would almost certainly be controlled by Republicans again. “That’s why we have elections,” she said dismissively.

Yet, Silver is most likely right about the House and it implies that Pelosi, as well as campaigns and all the well-paid political pundits, can do little to change the outcome.

Uncontrollability

Campaigns and the media do very little to nudge voters, especially for federal elections. For instance the alleged “youthquake” of young first-time voters Obama allegedly inspired though an unprecedented use of social media in 2008 was a big fat myth. Census results later revealed that the entire presidential circus only boosted youth turnout by a meager 2.1 percent, so little that ““If no one under the age of 30 had voted, Obama would have won every state he carried with the exception of two: Indiana and North Carolina,” wrote Chuck Todd and Sheldon Gawiser in How Barack Obama Won.

Indeed Obama was locked in a thrilling neck-and-neck race with McCain right up until the economy tanked in the fall of 2008, a factor completely outside either senator’s control.

Elections are decided by a disturbingly slim margin of the population: a combination of partisan couch potatoes who need encouragement to go vote and undecided voters residing in a few swing states who are over the age of 18, eligible to vote, can make it out on Election Day, and actually have their ballot counted. (For a hilarious take on this fact, watch the SNL clip below.)

Campaigns do matter, but far less than we imagine. Indeed, looking at the fivethirtyeight.com forcasts since June, predicting Obama’s win looks much like it does now, despite the rollercoaster ride in between. And, the factors we can’t control can mean much more than those we can. As a species primed to believe in free-will and in control of our destiny, that’s a very disturbing fact.

Moderation And Uncertainty

“The thing that people associate with expertise — authoritativeness, kind of with a capital ‘A’ — don’t correlate very well with who’s actually good at making predictions,” Silver told TechCrunch. Celebrity pundits make their careers carefully tracking daily polls and offering bold headline-catching interpretations as to why the American electorate is turning away from a particular candidate.

In reality, the most surprising polls of the day are usually the most inaccurate. A poll’s margin of error, usually around +/- 5 percent, means that out of every 100 polls, around five will show results much higher or lower than the actual population. Given the inherent volatility in prediction, Silver always expresses his prediction in terms of a probability rather than a grand “yes” or “no” prediction. It’s difficult to make a TV career on humility.

Moreover, he bases his predictions on polls themselves, with relatively less influence from variables like the state of the economy or political beliefs of the candidate. As he argues in his new book, The Signal and the Noise, even the smartest political scientists have failed miserably at predicting elections based on anything we know about human behavior and political preference. If polls are still the only reliable source of prediction, then it proves how dumbfounded we are with voter behavior, making pundit forecasting little more accurate than monkeys rolling dice with catchphrases.

Silver gets guff for the same reason I do when I call out tech companies for using poor research; when sound statistical methods are employed, the results are almost invariably tiny and there’s more left unknown than known. In a world where certainty is a tradeskill, statisticians reveal how little we definitively know about the world — a threatening concept indeed.