Uncategorized

Error and Bias in Referendum Opinion Polls

During the current (2010-15) parliament there have been three major referendums on constitutional change: the March 2011 referendum in Wales on increased law-making powers for the National Assembly for Wales; the May 2011 referendum on the Alternative Vote; and the September 2014 referendum on Scottish independence.

Much has been said about many aspects of these referendums; and doubtless, much more will be said in the future. In this short piece I’d like to look at the record of opinion polls in the referendums. Overall, how successful were they at predicting the final outcome of the referendums?

First, though, a few words of definition and clarification.

Prediction: Prediction is not, in general, what opinion polls should be understood as trying to do. A poll is a measure – an attempt to assess the relative frequency of different attitudes within a given population. Of course, such measures can be used to derive predictions of some form, or at least projections (such as the uniform national swing projections I regularly make of the voting intention polls conducted in Wales.) Sir Robert Worcester, founder of the MORI polling agency, puts the point well: ‘polls don’t predict, even if many pollsters do’. Nonetheless, the final polls conducted by various companies shortly before a vote ought to be able to get close to the final result, unless there are some very dramatic late swings. In that sense, we might reasonably take an immediately pre-election or pre-referendum poll from a polling agency as their prediction as to the outcome of a vote.

Error and Bias: In looking at how close final polls before referendums may have come to the actual result, I want to distinguish two ways in which polls may get things wrong. When talking about the degree of Error in a poll, I will be concerned simply with the extent to which they diverge from the actual outcome of the referendum. Bias is a rather different concept: it concerns whether the polls are tending systematically to be wrong in a particular direction. As I’m using it here, the term bias does not imply any deliberate attempt to make the polls wrong in a particular direction. Deliberate manipulation might, I suppose, be one reason why the polls might tend to err in a particular direction. But there are plenty of others. It is, therefore, very possible that polls might tend to have substantial errors associated with them, but without any bias at all. It is also possible – as, indeed, we will see – for the polls to have only small errors, but with a clear bias.

So, how well have the pollsters done in the three referendums? In the tables below I’ve listed all the final, publicly-reported polls from British Polling Council pollsters where they conducted a poll in the final week of the campaign, for each of the three referendums.

Wales, March 3, 2011

Poll (including survey end dates) % Yes % No % Error
RMG/Media Wales, February 28 69 31 5.5
YouGov/ITV-Wales, March 1 69 31 5.5
ICM/BBCWales, March 2 69 31 5.5
FINAL RESULT 63.5% 36.5%

Mean error = 5.5%

AV, May 5, 2011

Poll (including survey end dates) % Yes % No % Error
ComRes/Independent, May 1 34 66 1.9
ICM/Guardian, May 3 32 68 0.1
YouGov/Sun, May 4 40 60 7.9
Angus Reid, May 4 39 61 6.9
FINAL RESULT 32.1% 67.9%

Mean error = 4.2%

Scotland, September 18, 2014

Poll (including survey end dates) % Yes % No % Error
Opinium/Telegraph, September 15 48 52 3.3
ICM/Scotsman, September 16 48 52 3.3
Panelbase, September 17 47 53 2.3
YouGov/Times/Sun, September 17 48 52 3.3
Survation/Daily Record, September 17 47 53 2.3
Ipsos MORI/Standard, September 17 47 53 2.3
FINAL RESULT 44.7% 55.3%

Mean error = 2.8%

What do we see? A first thing to observe is that we see the most error in the first of the three referendums and the least in the most recent vote. That may indicate a tendency for the pollsters to be getting better over time. It could be that the three referendums saw progressively more resources invested in them by the pollsters, reflecting the respective extent of UK-wide interest in the three votes. Or, quite possibly, the level of error may be related to turnout. Turnout was by some way the lowest, at 35.6%, in the Welsh referendum. It may simply be rather easier for the pollsters to get things right when more people are participating in a vote.

Given the standard margin of error of +/- 3%, the ‘mean error’ performance of the polls in the first two referendums, and that in Wales in particular, was undoubtedly somewhat disappointing. However, we might bear in mind that the mean error figure for the AV referendum is actually rather misleading: two of the final polls came very close to the actual outcome, while two of them were some way from the truth. For the Scottish referendum, the mean error was below 3 points, while none of the polls were significantly further away than that. More problematic is that while there was limited error in the polls, there was clear bias: all the final polls over-stated the final Yes vote, and under-stated the final No vote.

One might think that this bias in the Scottish polls reflected some factors specific to that vote – such as a late swing induced by the high-profile promises of more powers for the Scottish Parliament made by the three UK party leaders and by Gordon Brown. Yet when we look across all three referendums, we actually see that this problem with the polls in Scotland was not the exceptional case, caused by the exceptional circumstances of that vote. Actually, what we saw in Scotland was the norm. In all but one of the thirteen final week referendum polls conducted, across all three votes, the support for constitutional change was over-stated.

These findings actually fit in with broader international experience. As my colleague Alan Renwick from Reading University has shown, there is a general pattern that polls ahead of referendums tend to over-state the final level of support obtained for constitutional change in referendums. The main reason for this, Alan suggests, is “uncertain voters typically end up sticking with the devil they know. If you are unsure quite what effects a change will have, then it is safer to hold to the familiarity of the status quo.”[1]

Two broad conclusions, I think, follow from this. The first is that the opinion pollsters’ performance in the recent Scottish referendum was rather better than some critics have suggested. However, the polls did tend to over-state support for change. From this follows the second conclusion. As Alan Renwick puts it “unless you are already way ahead in the polls, you should be cautious of advocating a referendum on your pet reform idea.” The implications of this for potential referendums in Wales will be discussed in my next Blog post.

 

[1] This quote, and the following one, both come from the following: Alan Renwick ‘Don’t Trust your poll lead: how public opinion changes during referendum campaigns’, in (P. Cowley and R. Ford, eds.) Sex, Lies and the Ballot Box:(50 surprising facts about Britain’s politicians and voters) (London: Biteback, 2014), p.81.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *