Can We Trust the Polls?

I recently had a conversation with a friend about the ongoing government shutdown. I pulled out my phone and pulled up the FiveThirtyEight website to look at Trump’s approval rating. My friend looked at me and said, “Yeah, but you can’t trust the polls.” I was initially set back by her response – FiveThirtyEight has a reputation for providing reputable data and tracks Trump’s approval rating based off of a collection of all polling data weighted by methodological standards and historical accuracy. How could she dismiss their work so readily? Then I remembered the dialogue that perpetrated the media after the election in 2016. After Donald Trump won the election many people were asking how the polls could miss the mark. People felt like the polls had betrayed them — just look at these articles from USA Today, The New York Times, and Politico.

Many people asked how the polls could have gotten it so wrong, but the short answer is that they didn’t. The national pre-election polls in 2016 indicated that Hillary Clinton would win the popular vote by a 3.3 parentage point margin. In the end, she won the popular vote by a 2.1 percent point margin. According to FiveThirtyEight data analysts, Carl Bialik andHarry Enten, a 2 to 3 percent polling error is fairly standard, which puts the national polls well within the margin of error. This error is also enough to account for Clinton’s loss in the Electoral College.

The majority of problems with polling in 2016 was on the state level which, according to research conducted by the Pew Research Center, “missed a late swing to Trump among undecided voters, and who did not correct for the fact that their responding samples contained proportionally too many college-educated voters (who were more likely to favor Clinton)”. Although these problems lead to an overstatement in Clinton’s lead and her position in the electoral college, they were never outside the realm of the standard statistical errors we have seen in electoral polling in the past.

But if the polls weren’t that far off base why were American’s so shocked by the outcome? 

I believe that part of that shock resulting from the 2016 elections was because of misdirected media representation and a general misunderstanding polling data. FiveThirtyEight contributor Nate Silver wrote that “Polls were never as good as the media assumed they were before 2016 — and they aren’t nearly as bad as the media seems to assume they are now.” The media narrative around election polling tends to focus on the numbers alone, commenting about who is in the lead, and not talking about the makeup of the poll itself. As a result, the general public is left largely unaware of what goes into election polling. The general public doesn’t understand the role response rates, statistical bias, collection methods, and other sources of error play into determining the precision of a particular poll. 

Instead of discounting the value of election polling it is important that we measure the value it offers against its limitations. We need to recognize that errors are not mistakes, but rather sources of uncertainty. Once we understand what creates that uncertainty we will be better equipped to understand polling data. 

Sources:

Nate Silver. “How Popular Is Donald Trump?” FiveThirtyEight, 20 Jan. 2019, projects.fivethirtyeight.com/trump-approval-ratings/?ex_cid=rrpromo.

Nathan Bomey. “How Did Pollsters Get Trump, Clinton Election so Wrong?” USA Today, Gannett Satellite Information Network, 9 Nov. 2016, www.usatoday.com/story/news/politics/elections/2016/2016/11/09/pollsters-donald-trump-hillary-clinton-2016-presidential-election/93523012/.

Lohr, Steve, and Natasha Singer. “How Data Failed Us in Calling an Election.” The New York Times, 10 Nov. 2016, http://www.nytimes.com/2016/11/10/technology/the-data-said-clinton-would-win-why-you-shouldnt-have-believed-it.html.

Shepard, Steven. “GOP Insiders: Polls Don’t Capture Secret Trump Vote.” POLITICO, POLITICO, 28 Oct. 2016, http://www.politico.com/story/2016/10/donald-trump-shy-voters-polls-gop-insiders-230411.

Carl Bialik. “The Polls Missed Trump. We Asked Pollsters Why.” FiveThirtyEight, FiveThirtyEight, 9 Nov. 2016, fivethirtyeight.com/features/the-polls-missed-trump-we-asked-pollsters-why/.

Kennedy, Courtney, and Courtney Kennedy. “Can Polls Be Trusted? Yes, If Designed Well.” Pew Research Center, Pew Research Center, 14 May 2018, www.pewresearch.org/fact-tank/2018/05/14/can-we-still-trust-polls/.

Nate Silver. “The Polls Are All Right.” FiveThirtyEight, FiveThirtyEight, 30 May 2018, fivethirtyeight.com/features/the-polls-are-all-right/.

“Sources of Error in Survey Research.” Qualtrics, 25 June 2018, http://www.qualtrics.com/blog/sources-of-error-in-survey-research/.