The Polls Weren’t Incorrect, We Read them Incorrectly
Op-Ed
It was staring us in the face this entire time. We weren’t paying attention.
The polls were accurate. At least the data were. The people telling you otherwise are the people who read it wrongly in the first place.
Consider the “gold standard” poll: The NBC News/Wall Street Journal poll, conducted jointly by the Democratic firm Hart Research Associates and the Republican firm Public Opinion Strategies, both considered giants in the industry, released the weekend before the election. The results: 44% Clinton, 40% Trump, 6% Johnson, 2% Stein.
It’s right there, waving a red flag at us.
No, not in the WSJ’s write-up of the survey:
Hillary Clinton holds a 4-point lead over Donald Trump in the waning days of a volatile campaign that has left many voters dissatisfied with both candidates and the political system, a new Wall Street Journal/NBC News poll found.
They buried the lede. They never even get around to mentioning the lede. There’s 8% undecided voters, double the amount of Clinton’s lead, just days out from the election. In reality, there’s even more than that.
Early on in 2016, some polls showed Gary Johnson with support in the double-digits. With him appearing on all 50 ballots, pollsters were left in a quandary. Including an unknown third-party candidate risks offering respondents an “outlet” to get out of choosing between two candidates viewed highly negatively, thus inflating his support. Not including him risks not accurately showing the race. They added him. Shortly thereafter, they added Jill Stein as well.
This was a huge departure from the norm that went remarkably unremarked upon. As it turns out, concerns about including third-party support were for good reason. Heading into the fall, third-party support decreased as voters chose between the two major party candidates.
So let’s look again at the NBC/WSJ poll, but this time let’s cut third-party support in half (mirroring their actual national support) and put those respondents in the undecided column where they belong:
Clinton 44%, Trump 40%, Johnson 3%, Stein 1%, Undecided 12%
12 percent! Mere days from the election, 14 million votes are up-for-grabs. Nate Silver believes a 2-point swing would’ve swung the Electoral College to Clinton, and 6 times that amount were undecided just days before the Election. How is that not the headline of the survey?
NBC/WSJ wasn’t alone. Look at the last week of live interview phone calling (excluding online and IVR), using our adjusted undecided total by halving third-party support:
| Clinton | Trump | Johnson | Stein | Undecided | Adjusted | |
| McClatchy/Marist | 44% | 43% | 6% | 2% | 5% | 9% |
| Franklin Pierce University/Boston Herald | 48% | 44% | 4% | 2% | 2% | 5% |
| CBS News/New York Times | 45% | 41% | 5% | 2% | 7% | 11% |
| NBC News/Wall Street Journal | 44% | 40% | 6% | 2% | 8% | 12% |
| ABC News/Washington Post | 47% | 43% | 4% | 1% | 5% | 8% |
| Fox News | 48% | 44% | 3% | 2% | 3% | 6% |
| IBD/TIPP | 41% | 43% | 6% | 2% | 8% | 12% |
| Monmouth University | 50% | 44% | 4% | 1% | 1% | 4% |
| Bloomberg News/Selzer | 44% | 41% | 4% | 2% | 9% | 12% |
That’s an average of 9% undecided. What does that tell us? That there is a lot of uncertainty in the outcome of the election. At the very least, 9% is more than the margin between Clinton and Trump. If Trump wins 60% of undecideds, the final tally is 49% – 47% for Clinton – which is the projected final tally.
The same goes for state polling. In Wisconsin, where the final margins between polls and election results were furthest off, three of the four last polls showed adjusted undecided voters in the double-digits.
This is just a huge amount of uncertainty, especially compared to four years ago. Looking at the last week of phone survey polling in 2012, the average undecided amount is only 3%:
| Obama | Romney | Undecided | |
| CNN | 49% | 49% | 2% |
| Gallup | 49% | 50% | 1% |
| Democracy Corps | 49% | 45% | 6% |
| Angus Reid Public Opinion | 51% | 48% | 1% |
| PPP/Americans United for Change | 50% | 47% | 3% |
| NBC News/Wall Street Journal | 48% | 47% | 5% |
| Pew | 48% | 45% | 7% |
| ABC News/Washington Post | 49% | 48% | 3% |
| PPP/Americans United for Change | 49% | 48% | 3% |
| Politico/George Washington University/Battleground | 48% | 49% | 3% |
(Note that this parenthetical aside is the only place in the article where we’ll discuss other baked-in polling error, like sampling error (which is accounted for in the margin-of-error, which was +/- 3.0%) and coverage error and non-response error (which are not). Also, your understanding of margins of error is likely wrong. A 44-40 Clinton lead with a +/- 3.0% MoE doesn’t mean her lead could be at 7-points or 1-point; it means that Clinton could have between 47% and 40% of the vote, and Trump could have between 43% and 37% of the vote. This accepted variance inherently adds more uncertainty on top of the large amount of undecided voters.)
(Also, the rise of IVR and online survey outfits, excluded from this analysis, is due to phone surveys becoming more expensive due to low response rates, the cost of calling cell phones, and slashed media budgets. There are many bad quality polls out there. Poll aggregates like RealClearPolitics can mitigate low-quality polling, though this does nothing to reduce the uncertainty due to undecided voters that is the topic of this post. Note that when we discuss the “polling industry”, this includes private firms, who use phone surveys, produce the highest-quality work, and are rarely released to the public.)
(Last aside: 24 of the final 41 Brexit polls showed Leave winning (by as much as 10%) or tied. The polling industry didn’t miss that one, either; conventional wisdom did.)
***
The headline of the Wall Street Journal article reporting the survey should’ve been “Anything Could Happen”. The lede paragraph should have been “With two days to go, enough votes remain up for grabs that either Hillary Clinton or Donald Trump could win.”
That’s what the data were saying. Screaming, really. Data can only be what they are; it’s up to pollsters and the media to report them properly.
But people don’t read polls for a forecast, and an open-ended one at that. They want something polls are not intended to be: they want a prediction. So pollsters give them one: Clinton by four (or “C+4” if you’re on Twitter). Then armchair pundits and prognosticators like you and I take that, along with the other polls showing a slight Clinton lead and say “Clinton sure seems to have a small but solid lead. I bet she wins.”
One of the few who resisted this urge is Nate Silver, who was called out by numerous peers saying he’s too bullish on Donald Trump’s chances. Silver, who embraced the uncertainty the polling data were showing him, had Trump’s chances of winning at 30%. That’s roughly the equivalent of flipping a coin twice and getting heads both times.
Most everyone else didn’t follow his lead. Clinton’s team thought they’d win. Trump’s team thought she’d win. Most posters here thought she’d win. I thought she’d win. It’s what the polls said, after all. We were reading them wrongly.
The polls were accurate. The rest of us were not.
Stephen Spiker is a former Bearing Drift contributor and a nationally recognized pollster.
