One final note on this. As I said two posts ago, our Barometer poll asked the following question: “Now that Britain has voted to leave it will need to negotiate the future relationship it has with the European Union. With this in mind, which of the following comes closest to your view?” getting the following pattern of results:
|The priority should be for Britain to have full control of their borders even if it means it is no longer able to trade freely with the EU||17%||77%||47%|
|The priority should be for Britain to trade freely with the EU even if that means it does not have full control over its borders||53%||8%||26%|
|Neither of these should be the priority||17%||7%||12%|
However, in December the Western Mail published results from a question that they ran on a Beaufort face-to-face poll conducted in November, which appears to show a somewhat different state of affairs. The Beaufort/Western Mail question asked:
“Looking at this card, please tell me which one of these is more important to you in the negotiations about Brexit (or the UK leaving the European Union)?”
Respondents were then given the following options:
- Keeping free access to the European Single Market, so that the UK can continue to sell goods and services without incurring tariffs or other obstacles to trade, but means they must allow citizens from all EU countries to work in the UK
- Restricting the right of people from EU countries to work in the UK, but not having free access to the European Single Market
- Don’t Know
So we can see that while this question was clearly getting at similar things to the one that we used on our Barometer poll, it was worded with some potentially significant differences. There was only one non-committal response – Don’t Know – while the wording of both the more definite responses in the Beaufort poll was substantially different.
And perhaps because of those differences in wording, the two polls got very different patterns of responses. The table below summarises the patterns across the two samples:
|Priority||Western Mail/Beaufort||Welsh Political Barometer|
|Single Market Access||54%||26%|
So what do we make of this?! One response might be the standard one of some people, that “all polls are rubbish”. But universal dismissal is no more thoughtful or wise an attitude than universal credulity. A more thoughtful approach might suggest a number of possible factors behind the differences:
- Probably the least likely candidate is genuine change in attitudes over time. I think this is unlikely for two reasons. First, as a general point, public opinion on such questions normally changes rather more slowly. And second, the findings in our Barometer poll in January were pretty similar to those we found in September. It would be strange indeed for opinion to have shifted radically between September and November, and then shifted right back by early January.
- Another possible factor, which may explain some of the differences, is sampling variation. This may be the random variation that statistical theory shows we can get from sample to sample, even when everything is conducted perfectly. Or it may be something more systematic, such as differences between the sorts of samples that YouGov and Beaufort are able to conduct using their different approaches.
- The most likely factor to be accounting for most of the difference, though, is question wording. Simply asking about the issue in these different ways appears to elicit quite different responses.
The best way to test such question wording effects will be to run a split-sample experiment on a future poll – giving a randomly chosen half of the sample one version, and the other half the other version. If the same poll, run at the same time by the same company, produces systematically different results for two different wording of a question, then we can be fairly sure that it was the ‘wording wot done it’.
Talking about such question wording effects may prompt some people to debate what is the ‘best’ or ‘correct’ wording. I’m not sure that it makes much sense to argue in those terms. What I think these findings may point to is the contestability of the issues of market access and border/immigration controls, and their overall place within public debate about the Brexit process. Considered in isolation, both ‘market access’ and ‘border controls’ would probably sound to many people like Good Things. The polling questions are trying to explore what might be peoples’ attitudes if they have to choose between the two of them. But the terms in which such a choice should be presented are not self-evident.
Much of political debate is about the terms in which issues are debated: try to associate your own policies with good things, and associate opponents and their ideas with bad things. And much of the rhetorical debate around Brexit in the coming months will likely be around trying to define the terms through which different options and possibilities are understood. Although there is much about Brexit where many people seem to have fairly fixed views, here at least things still appear to be up for grabs.