Uncategorized

So How Did We Do?

 

On several occasions this Blog has celebrated the creation of the Welsh Political Barometer, the polling collaboration between Cardiff University’s Wales Governance Centre, ITV-Cymru Wales, and YouGov. Providing regular polling in Wales can, I believe, better inform political debate and discussion. But a reasonable response to this might be that polls can only properly inform discussion if they are accurate. The European election provides a first opportunity to check how well calibrated is the Barometer’s measurement system. So how did we do?

Before we look at this in detail, though, there are a few things worth remembering:

  • First, that even if all polls were conducted perfectly, the maths of sampling theory suggests that polls of around 1000 respondents should only expect to estimate support for the different parties (or whatever else we are trying to estimate) to within 3% of the true figure, and that only 95% of the time. ‘Margins of error’, and the occasional outlier, are unavoidable.
  • Second, a poll seeking to get the final voting percentages right for each of the parties should really be conducted as late as possible before the act of voting itself. (The ideal would be to speak to voters as they were actually walking into the Polling Station, or just about to complete their postal ballot). But for reasons I won’t bore you with, sampling for the final Barometer poll prior to the European election in Wales was conducted 8-10 days prior to the vote. That might be OK with postal voters, but still leaves room for problems if there is a late swing between the parties – which, as we will see below, quite possibly happened in Wales in 2014.
  • Third, one thing that has already established about the GB-wide polls is that YouGov (who conduct the Barometer polls) were the most accurate company in estimating vote shares between the parties. So on a GB-wide basis at least, there is clearly little that is innately wrong with YouGov’s methodology.

The table below shows all the polls conducted on EP voting intention for this year’s election. (For the final poll I have reported two sets of figures on voting intention: those offered by all respondents, and those offered by respondents declaring themselves certain to vote).

 

Poll

Lab

Con

Lib-Dem

Plaid

UKIP

others

ItV-Wales/YouGov, February 2013

44

23

7

14

9

3

ITV-Wales/WGC/YouGov, Dec 2013

41

20

9

13

13

5

ITV-Wales/WGC/YouGov, Feb 2013

39

17

7

12

18

7

FoES/YouGov, April 2014

39

18

7

11

20

6

ITV-Wales/WGC/YouGov, May 2014

(Certain to Vote)

33

32

16

16

7

7

15

17

23

22

7

5

ELECTION RESULT, MAY 2014

28.15

17.4

3.95

15.3

27.55

7.6

The first poll to ask about European voting intentions was one conducted by ITV-Wales and YouGov prior to the establishment of the Barometer, in February last year. Some might well question how much value there could be in a separate question about European voting intentions so far out from the European elections, or even in the first Barometer poll back in December last year. It’s a fair question: we know that European elections have very low salience among most of the population. Still, given that past European elections had shown some people apparently voting differently than for general and devolved elections, I think it was fair to at least ask whether voters had yet developed distinct voting preferences for the European ballot (even if the answer, at that time, was that few people had).

There are two main trends that we see in the run of polls up until the final, pre-election one: a steady decline in Labour support, and an even greater rise in UKIP support. I would urge great caution about inferring a simple transfer of Labour voters straight to UKIP: net aggregate shifts can disguise much more complex patterns of voter ‘churn’ between the parties. It’s also the case, as I will elaborate on next week, that Labour’s recent decline in support has been a broader trend, not confined only to polls about European voting intention. However, it is clear that as the European election started to come into at least some people’s minds, UKIP support grew steadily. Moreover, this rise was greater than any UKIP has seen in voting support for general and devolved elections in Wales.

And how did the final poll compare to the result? One thing that has been commented on is that turnout in the election (at 32.1%) was much lower than the percentage of the final poll who claimed to be certain to vote (55%). I’m actually very comfortable with this. Polls (whether conducted face-to-face, by telephone, or via the internet) always over-estimate turnout; there are a number of well-known reasons for this, including:

  • The official turnout figure is always an under-statement. Even a frequently updated electoral register (as we now have in the UK) will still include some people unable to vote because they have recently moved and are not yet re-registered at their new address, and others unable to vote because they are too busy being dead.
  • The sort of people who respond to polls are also the sort of people more likely to vote.
  • And some people may feel uncomfortable admitting that they did not vote; taking part in an election may be perceived as the more ‘socially desirable’ thing to do.

Over-estimation of turnout is now a well-known phenomenon, and not one that I personally see as that problematic. What does matter is that within those who don’t vote, and those who do, we get a reasonably representative sample.

Looking at the performance of the main parties, how did we do there? Well, clearly the final poll was well within the ‘margin of error’ for the Conservatives and Plaid Cymru. It was also very close for the Greens, who got 4.5% of the vote (compared to 6% in the final poll, or 5% among those declaring themselves certain to vote). The final poll also (just about) estimated the Liberal Democrats support within 3% of the true figure.

Where things look a little more problematic is with regard to Labour and UKIP. The final poll was not wildly inaccurate – it’s not like we suggested Labour might get double the support level they did! – but it was significantly less than perfect. Labour support was over-stated by 4-5 percentage points, and UKIP support under-estimated by a similar amount.

One potential explanation, to which I would certainly give some credence, is that there was a further swing from Labour to UKIP between the final poll and the actual vote. Given that the polls were moving in this direction consistently up to the point of the final poll, this seems very plausible.

Indeed, the only clear reason I can find to doubt this explanation is the study conducted by Cardiff and Edinburgh Universities in April: sampling done (by YouGov) about five weeks before the European election estimated UKIP support for the European election at the following levels for the three British nations: England, 29%; Wales, 20%, Scotland, 10%. For both England and Scotland, the survey was very close to UKIP’s final vote share; for Wales, it was under-stated by more than 7%. Was there really a swing to UKIP that occurred only in Wales? Perhaps so, although we cannot be certain.

For Labour, at least, there is one other possible explanation for the error in the final poll’s estimation of its support (though an explanation that is not necessarily mutually exclusive with the one outlined above). This is that the Barometer polls (and others conducted by YouGov) may have been tending systematically to slightly over-state Labour support. As I’ll discuss next week in exploring broader polling trends in Wales, it is notable that the only two non-YouGov polls reported in Wales in 2014 have been the ones producing the lowest Labour vote intention for Westminster. Let’s be clear: that doesn’t necessarily mean that these other polls were right and YouGov were wrong. It merely raises a possibility, which may be worthy of further exploration.

To conclude: one of the general difficulties of polling is that we are using sample surveys to try to estimate attitudes in the general population when the ‘true’ figure in that population is mostly unknowable. Election results thus provide a very valuable ‘reality check’, and can help polling companies refine methods for the future. Of course, even that is imperfect. It means that you are using the past – albeit the very recent past – to guide the future, and the past is always an imperfect guide: as Harold Macmillan once delightfully put it, it is like travelling by rail using last year’s Bradshaw!

Overall, I think the Welsh Political Barometer has done pretty well in its first encounter with voting reality. The specific lessons we can learn is that there is a possibility – which we need to explore further – that Labour support has been marginally over-stated; and secondly that we need the final poll to be as close to the election date as possible. For next year’s general election, we’ll make every effort to have the sampling for that last poll conducted much closer to election day itself.

Comments

  • J.Jones

    One thing that I have pondered is this thorny question: What would the EP election outcome have been without the Yougov polling?

    I can’t remember the previous history of uniquely Welsh polling in the run up to an EP election but I did notice that for Plaid in particular finding their polling figures at 12% and 11% focussed their support enormously. I wonder if Labour seeing their support at 39% actually relaxed. As for UKIP…..I remain amazed but I was pointing out to you before the election that there were strong anti immigration attitudes in Wales.

    I did speak to one UKIP supporter before this election to urge him to vote Labour, he told me that UKIP expected to take two MEPs in Wales!

    • Roger Scully

      Interesting one, Jon, and of course we can never quite know the answer to such counter-factuals.

      ITV-Wales and us certainly didn’t start the Barometer polls with the aim of influencing the Euro elections in a specific way! We did want to try to contribute to people being rather better informed – and specifically to make sure that there would be REGULAR polling. In this particular instance, it is possible that it has influenced things in the way you describe; doubtless, at other times it might help someone else.

      But, on the whole, I think it is far, far better to have regular polling – and with all the acknowledged imperfections and limits of polling – than nothing. Imagine if all we had to go on were rumours of private polling, and endless Twitter statements about the ‘fantastic reception on the doorstep’…

  • Vaughan Williams

    I feel Labour should be quite concerned, they seem to have slumped in recent opinion polls in Wales. The UKIP vote seemed VERY high in traditional Labour areas suggesting a loss of support – we do have evidence of this after the Ynys Mon by-election where in Holyhead (being a native from that town I have prior knowledge) they are into the Labour vote substantially. It remains to be seen what support UKIP will have come the next election but ‘safe’ Labour seats may now be up for grabs.

    As for Plaid Cymru, in such an anti-EU election and UKIP having SO much media attention, I think 15% of the vote 111,000+ votes was quite an achievement.

    • Roger Scully

      Thanks for the interest in the Blog, Vaughan.

      I think I largely agree with you, and I’ll be blogging next week about the recent trends in opinion polls for both the general election and the Assembly constituency vote. Neither of these look great for Labour – although their saving grace is the lack of a single strong challenger.

      Re: Plaid – I am in the ‘half-full’ rather than ‘half-empty’ camp on their performance. I think they did fairly well given a difficult context, though I wouldn’t put it any stronger than that.

  • Carl

    There is a dire need for more polling in Welsh politics. The Wales Governance Centre, ITV Cymru-Wales and YouGov deserve a great deal of praise for this collaboration.

Leave a Reply

Your email address will not be published. Required fields are marked *