The polls got it wrong (again) but don’t lose faith in quantitative research

By: Jim
Like many, I woke unusually early on
Wednesday and reached nervously for my mobile phone. It was US election night
and I was eager to see if, from my perspective, crisis had been averted or the
world really had gone mad. Before I had a chance to tap my favourite news app I
noticed a message from my brother: ‘Another resounding victory for the
polls bruv!’ Detecting sarcasm (I’m smart like that) I knew this could
only mean one thing. Sure enough, Trump was well on course to a victory that
nobody, least of all the pollsters, was anticipating. For the third time in
eighteen months (following the UK general election and EU referendum) the
pollsters had got it wrong!
In the period since May 2015, I’ve had
countless debates with polling sceptics like my brother. His, fiercely
articulated, view is that polling is not simply inaccurate, it also has the
potential to sabotage itself. He’s not alone in this belief. Behavioural
economics shows that people generally wish to follow the herd. Therefore, a
poll showing that the majority think in a particular way is likely to
influence, albeit subtly, what they themselves believe. Furthermore, there are
those that cite the possibility that polls could impact rates of voter turnout.
After all, why bother to turn out to vote if the polls have created a strong
belief that your favoured candidate is either assured of victory or has no chance
of winning?
Polling, when first popularised by George
Gallup in the 1930s, was hailed for the positive contribution it made
to the democratic process. Gallup himself was, understandably, steadfast in
this belief. Elmo Roper, another pioneer of the public opinion poll, described
it rather hyperbolically as ‘the greatest contribution to democracy since
the introduction of the secret ballot’. 

But there have always been critics, and
the anti-polling arguments inevitably gain traction when the pollsters get it
wrong. Failure is not a modern phenomenon either. Immediately prior to the 1948
election George Gallup predicted that Dewey would beat Truman in the election
and stated, unwisely as it turns out, ‘We have never claimed
infallibility, but next Tuesday the whole world will be able to see down to the
last percentage point how good we are’. Dewey lost. The anti-polling lobby
had a field day.

So criticisms of polling aren’t new and,
let’s be honest, they would remain niche concerns if the polls were accurately
predicting results. But they’re not and on the back of a series of high profile
failures it’s increasingly common to deride polling as a ‘devalued
pseudo-science conducted by charlatans’. Yep, my brother again. I hate to give
him the last word so, in order to provide a flavour of wider opinion, I’ll
quote the Guardian’s post-election editorial instead. ‘The opinion polls
and the vaunted probability calculus rarely trended in his (Trump’s) direction;
both are discredited today.’
The purpose of this blogpost is not to
defend political polling; I have my own concerns in that direction and it’s
undeniable that the work of pollsters is becoming harder, due to a combination
of methodological issues and a more fluid, less predictable, political
landscape. However, for the sake of fairness I’d like to mention two things,
neither of which is intended to exonerate the practice.
First, most polls reflect public sentiment
within a nationally representative sample. In the main, but not exclusively,
the polls conducted immediately prior to the election found that, by a
relatively small margin, more Americans intended to vote for Clinton than
Trump. In this they were correct. At the time of writing, the figures show that
59,814,018 Americans voted for Clinton whilst 200,000 fewer (59,611,678) voted
for Trump. However, due to the distribution of votes and the vagaries of the US
political system, this translated into 279 Electoral College votes for Trump
and 228 for Clinton.
Second, most polls conducted by reputable
polling organisations produced results that placed the result well within the
margin of error. ‘What’s that’? I hear you ask. Well, tucked away at the
end of most reports based on a public opinion poll will be a small note about
margin of error. This margin will differ depending on the number of people
interviewed for the poll but, for a standard sample size of 1,000, the margin
of error is +/- 3.5%. This essentially means that if the poll results show that
Clinton is projected to win 47% of votes, the reality is likely to be somewhere
between 50.5% and 43.5%. Within this context, the result of the election was
well within the margin of error of most polls. It wasn’t so much the polls that
got it wrong, it was the reporting of the polls that failed to sufficiently
stress that the result really was too close to call. But people don’t like
uncertainty so these boring, statistical caveats tend to get overlooked.

OK, but I said this blog wasn’t designed to
defend polling. So what is it about? Well, I don’t feel the need to defend
polling because I’m not a pollster. However, I am a market researcher working
with quantitative surveys and, what concerns me, is the fear that growing
scepticism around polling will negatively impact trust in all forms of
numbers-based research into public attitudes. Maybe I’m just a worrier and
people are perfectly able to distinguish between different forms of survey
based research. However, my own experience suggests that isn’t always the case.
In May 2015 I was working at the Guardian.
The Guardian has invested significantly in data journalism over recent years
and coverage and analysis of polls was given a high degree of prominence in the
run up to the UK general election. At the Editorial conference, held the day
after the election, the mood was subdued. When the conversation turned to the
failure of the polls some journalists questioned the prominence given to
polling numbers, especially as those numbers didn’t chime with their instincts
and the evidence of their own, on the ground, experiences. The upshot was a
policy decision, only recently reversed, that editorial coverage of polling
should be suspended. The coverage of polls in the run-up to the US election was
reported under the banner ‘Sceptical polling’, which gives a pretty good
indication of the mood around the organisation.  
As Head of Consumer Insight at The
Guardian, a key element of my role was to advocate for use of consumer research
and promote evidence-based strategic decision-making. My internal clients were
ranged on a spectrum that ran from research enthusiasts to rejecters. This
latter group, a minority it should be said, believed there was little to gain
from engaging with research. The great polling disaster of 2015 provided a
tailor-made reason to disengage. After all, research had been shown, in the
most public way imaginable, to be unreliable and wrong! Hadn’t it?
I’m sure the Guardian is like most
organisations in having research stakeholders ranging from enthusiasts to
sceptics. To the latter group I would make this plea; don’t conflate political
polling and other forms of quantitative market research and do not deny
yourself and your business an incredibly powerful, consistently proven aid to
decision making simply because political polling has been shown to not be a
perfectly accurate crystal ball. As mentioned, polling isn’t quite as
inaccurate as some would have you believe. Furthermore, the stakes are simply
much higher for polling: A couple of percentage points either way (generally
within the margin of error, remember) is the difference between two
diametrically-opposed outcomes and the profound repercussions associated with
that. In contrast, if a representative survey of consumers in a particular
sector suggests that awareness of your brand currently stands at 34% whilst
that of a competitor is 64%, does it really make a huge difference to the
decisions your company will take if the reality is a couple of percentage
points either side?  
Of course, some decisions do require a
higher degree of accuracy. In these instances, market researchers have two huge
advantages over pollsters. We can increase the number of people interviewed in
the study, thus reducing the margin of error and increasing confidence levels.
We can also utilise robust sampling techniques such as random probability
sampling. Generally speaking, neither of these options is available to
pollsters because they are simply too time consuming. Pollsters are required to
provide an almost instantaneous reading of public sentiment, before new events
have a chance to change it, and anything that slows that process is, by
necessity, discarded. If pollsters were given the freedom to use these tools,
it’s likely they would provide far more accurate predictions. How do we know?
Well, following the 2015 general election most polling companies conducted
re-contact surveys with pre-election poll respondents to try and understand
what went wrong. What they discovered was that, even when conducting post-event
research, they were unable to accurately replicate the result. The inquiry
conducted by the Polling Council of Great Britain concluded that the reason was
their use of (attitudinally) unrepresentative samples drawn from panels and
that a random probability sampling approach (that gives every member of a target
population an equal chance of participating in the study) would counteract the
problem. Tellingly, the survey that best replicated the election result was the
British Social Attitudes (BSA) survey conducted by NatCen Social Research. Need
I say that BSA is based on a large sample (3,000) and utilises random
probability sampling?
I’ve rambled on too long and exceeded my
word count limit by a distance so I’ll finish by saying this: The great jazz
musician, Duke Ellington (or possibly Richard Strauss, it’s disputed) is quoted
as saying ‘there are only two types of music: good and bad’. Market
research is much the same. When done properly it is an incredibly powerful
diagnostic and forecasting tool that can provide a highly accurate picture of
consumer sentiment as it currently exists. Pollsters, through no fault of their
own, are sometimes unable to do it.
Researchers, however, can and do. 
Jim Mann is a senior quantitative director
at the numbers lab @ Firefish