Following the failure of the pre-election opinion polls to get anywhere close to the actual result of the 2015 General Election, the team of survey methodologists and statisticians commissioned by the British Polling Council and Market Research Society to investigate what went wrong has now presented its preliminary conclusions.

The biggest factors – they concluded – were simply that the polls had too many Labour voters in their samples, too few Conservative voters, and that the weighting schemes failed to correct for this.

One of the most striking findings in the presentation was that at an aggregate level, the polls appear to have had too few ‘older old’ people (ie 75+); and that politically engaged younger people were over represented.

Here is my take on the Inquiry’s interim conclusions and, in particular, what should happen next. It isn’t easy to predict a first-past-the-post Election with the precision required to be useful. And I stress that the Inquiry team have excellent technical skills, as do the industry pollsters.  But I am concerned that complacency might already be kicking in. I was particularly surprised to hear Professor Patrick Sturgis (Inquiry Chair) say on the Today programme:

“You could look at it in some ways as a bit unfortunate that they were three points over on Labour and three points under on the Conservatives, but the fact is that the polls are by far and away the best way of trying to figure out what the election result is going to be”.

Well, not this time. And ‘unfortunate’ displayed masterful understatement given the impact of the polls on the way the media discourse played out.   We are therefore entitled to be tough in our “ask” of the polling industry as it responds to the Inquiry, and to encourage the Inquiry to be tough in its final recommendations to the British Polling Council and Market Research Society in March.

Technical improvements are required:

  • The sample is the foundation of any research, and arguably getting the raw sample right is conceptually the least tricky bit. The skew against older old people should have been clocked before the Election by rigorously evaluating profiles against demographic statistics. More attention also needs to be paid to the profiling of samples against a wider range of non-demographic benchmarks – calibrating against national statistics and major high quality probability sample surveys, such as British Social Attitudes and the British Election Survey, which did not appear to suffer from the same systematic bias.
  • The industry needs to agree some quality measures for quota sampling. For example, publishing the range of weights, the average number sought for every person achieved, and the development of methodological standards in terms of attempting to find the more difficult to reach.
  • Weighting for inadequacies in achieved samples is a difficult task to get right. But pollsters could make their raw data sets available (not just the tabulations) so that sensitivity analyses can be rapidly conducted on weighting schemes by others.
  • Looking back to the Inquiry following the 1992 polling problems, the survey researcher Roger Jowell (who set up British Social Attitudes), concluded that pollsters should consider imaginative uses of random probability sampling as one way of addressing sample biases. True random sampling would be impractical if we want fast results. But there must be a halfway house: slowing research down a touch, and drawing on Jowell’s earlier research by incorporating wave analysis of those who were easier and more difficult to reach to inform the weighting scheme, and also to inform strategies for achieving better representation in the first place.

A more collaborative culture is needed

This can be achieved through ways of working, transparency and stronger governance:

  • The industry must assure the public that there will be step change in open collaboration as to the conduct and evaluation of polls ahead of any important election. This should include a clear and open focus on addressing the well-known risks.  The polling problems surprised everyone. And yet the Inquiry’s analysis shows that every set of pre-election polls since the mid-1980s has (on average) underestimated the Conservative vote. 1992 and 2015 were particularly bad examples, but the risk of systematic under-representation of Conservative voters should have been dissected more openly, to consider whether this risk was being tackled or, in fact, crystallising.
  • Aside from recovering of the painful irony of an ex city analyst calling the election correctly, the industry needs to learn from Matt Singh’s approach to evaluating the polls’ findings, with more triangulation from other data sources.
  • Polling is self –regulated, and it is unlikely that Lord Foulkes’ proposed Bill will change this. But we should note that the British Polling Council is light touch, and remarkably homogenous. Its gender bias (and this is difficult to achieve) is worse than the FA’s with not a single member represented by a woman, and not a single woman on its disclosure board. No -one is comfortable with all male boards: why are we not shouting about such male dominance in the political opinion polling wing of the research industry?

There are lessons for commissioners

  • The Inquiry found that 91 GB wide polls were conducted during the campaign period itself. Between 2010 and 2015, there were nearly 2,000 GB wide polls. The team estimated that this compares with around 3,500 from the entire period from 1945 – 2010. This step change in volume, as a result of online polling and generally reduced costs, has clearly not served the public well in terms of the prediction of the result. (Of course, that is not to deny a wealth of useful information within the polls).
  • And yet despite this enormous number of polls, there has been much bemoaning about how it would simply be too expensive to do decent research instead. The obvious fix would simply be to do fewer, and make them better – even if we have to wait just a touch longer for the results.

Some pollsters have apportioned some blame in the direction of the media. Of course, journalists should understand what they are reporting. And certainly, the media should not have fixated on only one potential outcome indicated by the pre-election polls. But you can hardly blame the user if your goods are not up to the job – and the truth is that in aggregate, the 2015 pre-election polls were faulty rather than unlucky.

So, there is a lot to fix before the next General Election of 2020. But what of the looming In/Out EU Referendum? It is somewhere between unlikely and inconceivable that – even if the methodology is better – trust in methods and findings will be restored ahead of the EU referendum. So as well as taking rather more notice of methodology, we should also expect more focus on the implications of the vote, rather than simply trying to predict the outcome or the comment on the implications for the personalities involved.

Penny Young is Librarian and Director General, Information Services, House of Commons. Previously she was Chief Executive, NatCen Social Research, which conducts the long running British Social Attitudes survey.

Picture Credit: David Cameron