Skip to main content
Coronavirus (COVID-19): Latest updates.


Mr Spoil-Sport

3 March 2014

I love my job. But one of the slightly less enjoyable bits is playing Mr Spoil-Sport in the interpretation of opinion poll results. I often find myself telling people (quite often politicians or political activists excited to see a poll apparently showing rising support for their party, or a view they support) that ‘it may not be as simple as that’, or words to that effect. I’m afraid that I’m going to have to do it again.


Last week’s BBC/ICM poll included a ‘constitutional preferences’ question. Such questions have been a staple of polls in Wales since at least the mid-1990s (and featured occasionally in polls conducted even earlier). Essentially, this form of question asks respondents to indicate their most preferred form of government for Wales from among several options. The new BBC poll had an interesting variation (used only very occasionally in previous studies): in addition to a standard constitutional preference question it had a follow-up which offered people the same set of options in the (currently!) hypothetical context of Scotland having become independent. The main point to note from the latter question was how little responses changed from the original question. But it is responses to the former, the standard constitutional preference question, that have attracted some interest and comment.


The interest arises from two features of the results. First, the poll showed support for independence at only 5%, the lowest that I can recall seeing in any such survey question since 1997. Second, support for the No Devolution option was 23%: the highest I am aware of in any survey since the Welsh Assembly Election Study of 1999, and standing in stark contrast to the Silk Commission survey, conducted by Beaufort Research and reported last summer, which found support for a No Devolution option at only 9%. In a future post, I intend to assess the recent BBC/ICM findings in conjunction with other recent evidence. Here, I will focus simply on how the BBC/ICM poll compares with previous BBC/ICM studies.


Last week’s BBC/ICM poll appears to show rising opposition to devolution. The percentage supporting the No Devolution option, at 23%, is fully 10% higher than in the equivalent survey conducted in 2010. That looks like a very important finding. It is clearly relevant to current constitutional debates, and runs against the main trend in public attitudes since 1999: the decline in opposition to devolution. A number of people have been quick to suggest possible causes for such an apparent rise in opposition to devolution: those I have seen include:


  • The Scottish referendum: with support for independence being low in Wales, some may look at Scotland and recoil from Wales heading along any similar ‘slippery slope to separatism’.
  • The current Welsh government’s difficulties and apparent failings in key policy areas. As discussed recently on this blog, many people (even many Labour supporters) are not impressed with what devolution has delivered in areas like health and education; one very plausible response might be to support getting rid of the Assembly.


I would not dismiss those arguments out of hand. However, the truth may just be rather more prosaic. To understand why, we need to probe deeper into what current and past BBC/ICM surveys have asked, as well as what they have found.


First, I’m afraid our old friend ‘question wording’ enters into the picture. I’ve mentioned previously on the blog that apparently innocuous changes in question wording can produce surprisingly big effects; that may be the case here as well. The original ‘stem’ of the constitutional preference question seems to have remained constant throughout the last five BBC/ICM surveys. Survey respondents were asked: “Which one of these statements comes closest to your view?” However, the set of response categories differs in subtle ways. Below is the set of responses offered in 2010 and 2011:


  • Wales should become independent, separate from the UK and the European Union
  • Wales should become independent, separate from the UK but part of the European Union
  • Wales should remain part of the UK with its own Assembly which has full law making powers and some taxation powers
  • Wales should remain part of the UK with its own Assembly which has full law making powers but no taxation powers
  • Wales should remain part of the UK with its own elected Assembly which has limited law making powers only (as it has now)
  • Wales should remain part of the UK and the Assembly should be abolished.
  • None of these
  • Don’t Know

Got that?!


But in 2012, 2013 and this year, the following set of responses was used:


  • Wales should become independent, separate from the UK
  • The Welsh Assembly should have more powers than it currently has
  • The powers it has are sufficient and it should remain as it is now
  • The Welsh Assembly should have fewer powers than it currently has
  • Wales should remain part of the UK and the Assembly should be abolished
  • None of these
  • Don’t Know


This latter response set clearly differs in several, potentially important ways:


  • First, note that we have switched from two to one ‘independence’ categories. This alone very possibly accounts for the decline in support for independence in the polls after 2011 (see tables below).
  • Second, the ‘more powers’, ‘remain as now’ and ‘fewer powers’ options are all worded much differently from previously, with the more powers options also having (in effect) declined in number from two to one.
  • Third – and I strongly suspect that this is important – the ‘Assembly abolished’ option is now the only one to specifically mention Wales remaining in the UK, whereas previously it was explicitly mentioned in all but the independence options.


In short, it is very hazardous to compare the results of the constitutional preference question in this year’s BBC/ICM survey with those prior to 2012. We really have two sets of results: those for 2010 and 2011, and those for 2012-2014.


What did the respective surveys find? First, I re-produce the findings for 2010 and 2011.





Independence outside EU



Independence within EU



Full Law & Tax powers



Full Law but not tax powers



Limited Law Powers (as now)



Assembly abolished






Now the findings for the last three years:








More Powers




Remain as now




Fewer Powers




Assembly Abolished








Why would the question wordings have been changed? I was not involved in the discussions around changing the question wording, so cannot comment on precisely what the reasoning would have been. (Full disclosure: I did provide the BBC with some advice on the wording of this year’s survey). But obviously the 2011 referendum result changed the ‘status quo’ around which a question like this has to be oriented. We must also remember the real-life difficulties of survey companies like ICM, trying to meet right deadlines amidst a generally declining willingness of the public to participate in telephone surveys. In that sort of context, changing and simplifying the wording of a question can make a lot of sense.


However, it is clear from this examination of the details of the recent surveys that making comparisons between the 2010/11 constitutional preference findings and those from the last three years of BBC/ICM poll is at best very hazardous, and possibly downright foolhardy. Including ‘remain in the UK’ only in the final response option is a choice I would find difficult to commend; this is very plausibly a significant part of why the BBC/ICM figures for a No Devolution option have been higher than those reported in some other surveys. (As I will explore soon in a future blog post). Now that this wording has become established as standard in the BBC/ICM poll, it may well be best to retain it in order that any trends over time can be explored. But none of the figures produced by this question for either 2013 or 2014 differ by more than the standard margin-of-error from those produced in 2012.

In short, it may be that there is growing support for abolishing the Assembly. But it is far from clear that this is the most plausible interpretation of the available evidence.


  1. Roger Scully

    By the way – apologies to all that the formatting on the second table came out all wrong.

    What can I say – Microsoft strike again…

  2. Jon Jones

    A good example of the best way to establish opinion; KISS. The fewer the options the more clear the meaning of the responses; giving lists to people and asking them to respond increases the risk of random responses and an inclination to avoid the extremes of opinion.

    Ultimately the various responses can be encapsulated in a form that would be offered in a referendum:
    “Do you think that the Welsh Government should be given more powers than it already holds; Yes/No.”

    Over the last three years the answer would clearly be “NO”. Of course a sustained campaign by all political parties acting together might well change that, after all, it worked last time.

    Then again, they say you can fool all of the people some of the time…but maybe all of the time if every political party and all of Academia plus the Great and the Good put their minds to it.

Leave a reply

Your email address will not be published. Required fields are marked *