Since Elections in Wales burst upon an unsuspecting and defenceless public in July 2013, one of the things I’ve sought to do with the blog is promote an awareness and intelligent discussion of opinion polls. By that I mean not only publishing the findings of polls, and trying to make the details available for people to examine; I also mean a discussion of the potential limitations of surveys and polling.
Some of those limitations were, of course, very publicly demonstrated at the general election. The errors made by pretty much all the pollsters in their estimates of Labour and Conservative party support have prompted several enquiries and plentiful discussion about opinion polls. Some of those enquiries have now issued reports. The interim presentation made by the official British Polling Council inquiry is available here. There have also been some very interesting reports published in recent times by the British Election Study; by the British Social Attitudes study; and by YouGov. There has also been a very interesting three-part Radio Four series presented by David Cowling.
I’m not going to attempt a detailed summary of all these reports here. If you can’t quite bear plunging straight into them, the radio shows offer, I think, a very good introduction to the pertinent issues. Another very helpful introduction has been offered by YouGov’s Anthony Wells at his excellent UK Polling Report blog here. Put very simply, the main problem with the polls appears to have been with the construction and weighting of the samples, which have not been sufficiently representative of the electorate – and, most particularly, that section of the electorate that end up voting.
Finding consistently reliable fixes to these problems may be far easier said than done. Bluntly, if the problems were easy to fix then I suspect most of them would never have arisen in the first place. Working more closely with various polling companies in recent years I’ve become acutely conscious that polling is not at all an easy thing to do well. Moreover, in some respects it is getting harder: telephone polling, in particular, is getting ever more difficult as more and more people screen out calls or simply refuse to respond to surveys. I would certainly agree with Lord Ashcroft, who observed on his polling website recently that “the polling world has been remarkably open and honest in facing up to what happened and trying to put it right”. Still, openness and honesty don’t actually provide solutions to the problems.
But while the pollsters have been getting plenty of flak over recent months (some of it, at least, probably deserved), I don’t think they are the only ones who have lessons to learn from last year. There are also important lessons for political journalism. Opinion poll findings should be reported to the public: if this information exists then the people have a right to know about it. But reporting on opinion polls well requires attention not only to the immediate details of how polls are reported (i.e. getting the numbers right, and showing an awareness of ‘margins of error’, the importance of question wording etc etc). It is also important to consider how polling can inform or shape the broader narrative within which the election is reported and understood. In retrospect, I think many political journalists – who I fully recognise are also people with a very difficult job to do – would acknowledge that the polls probably became too central to the entire narrative within which much reporting of last year’s general election was conducted.
Does any of this have any specific implications for Wales – and particularly for this year’s Assembly elections? In general, polling in Wales faces the same challenges as elsewhere. But there’s a couple of issues I’d like to draw your attention towards.
The polls – or rather, the poll, the Welsh Political Barometer – actually did pretty well in Wales in 2015, as I have discussed previously. But the fact that we got pretty close to the final result in 2015 doesn’t diminish the innate limitations and difficulties of polling. Moreover, some of those problems may be all the more pertinent to lower-turnout elections: as I have pointed out recently, since YouGov started working regularly in Wales their final pre-election polls have been more accurate in general elections and rather more prone to error in lower turnout contests. I suspect this reflects simply the greater difficulty of estimating party support in circumstances when very large proportions of the electorate are actually not taking part. Of course, such difficulties may be very pertinent this year.
Second, the point about the poll may also be significant. YouGov are the only company regularly conducting political opinion polls in Wales. (Or, at least, the only company doing polls whose findings are published). YouGov are an excellent company, and I’m delighted to work with them on the Barometer polls. But it’s no criticism of them to say that we should be at least open to the possibility that polls from other companies might offer a slightly different picture.
For example, since last May’s UK general election the five highest ratings for UKIP in Britain-wide polls have all come from YouGov polls – YouGov are the only company to have put UKIP support as high as 17% or 18%. Moreover, these results have not been simply instances of the occasional ‘outliers’ that are all-but-inevitable in polling; in general, YouGov have been putting UKIP on a rather higher level of support than most other companies. (Since the 2015 general election, published polls by YouGov have given UKIP an average support level of 16.4%; those from ICM, by comparison, have put UKIP’s support at an average of 11.5%, and those from Ipsos-MORI have averaged UKIP at only 9.0%). Now that could well mean that YouGov are getting it right and other companies are under-stating UKIP’s current support. But it’s also at least possible that YouGov may be tending to over-state UKIP support. And if the latter were true for GB-wide polls it might very well also be the case for their Welsh polls. In short, were other companies to be conducting polls in Wales as well, it is quite possible that they might be reporting a slightly less positive picture for UKIP in Wales than recent Barometer polls have been suggesting. We might have had polls projecting UKIP to win 3-4 seats in the Assembly, rather than the nine suggested by the last Barometer poll. Who is doing the polling can sometimes play a significant role in shaping our expectations of an election.