All That is Solid…

I ended part two of my recent Electoral Review of 2015 by observing, rather portentiously, that “All that is solid about an opinion poll lead can melt into the electoral air…” I had been reflecting, as you may recall, on how an apparently sizeable opinion poll lead for Labour prior to the 2014 European election had largely disappeared by election day.

Looking back over the opinion poll numbers for that piece, however, set me thinking along slightly different lines. Even after seeing its poll rating edging downwards as the European election approached, Labour still remained some points ahead of its nearest challenger for those elections, UKIP. But on the day, Labour substantially under-performed its final poll rating in the election itself. And as I checked over some other numbers, it became clear that this was not a wholly isolated incident.

Comparing very different polling companies and methods over time can be hazardous. Moreover, it is only since YouGov began fairly regular Welsh polling in late 2009 that we in Wales have consistently had pre-election polls which we can compare to the actual election results. So in the table below I’ve looked only at the performance of YouGov in Wales, compared to the election results, from the 2010 general election onwards. I’ve taken the last pre-election poll prior to each significant electoral contest, and compared the figures produced from the standard voting intention question with the final result. I think the table is quite revealing:


UK General Election Polls Lab Con Lib-Dem Plaid UKIP Others*
ITV-Wales/YouGov, 1-3 May 2010 35 27 23 10 5
ELECTION RESULT, MAY 2010  36.2 26.1 20.1 11.3   6.2
ITV-Wales/WGC/YouGov,  May 2015 38 26 7 12 13 4
ELECTION RESULT, MAY 2015 36.9 27.2 6.5 12.1 13.6 3.7
 EP Election Polls
ITV-Wales/WGC/YouGov, May 2014 33 16 7 15 23 7
ELECTION RESULT, MAY 2014 28.15 17.4 3.95 15.3 27.55 7.6
 Local Elections            
WGC/YouGov, May 2012 48 17 7 14 15
ELECTION RESULT, MAY 2012# 42 13 8 14   22
 NAW Constituency Vote            
ITV-Wales/YouGov, 2-4 May 2011 47 20 9 18 6
ELECTION RESULT, MAY 2011 42.3 25.0 10.6 19.3   2.8
 NAW List Vote            
ITV-Wales/YouGov, 2-4 May 2011 43 19 8 18 13
ELECTION RESULT, MAY 2011 36.9 22.5 8.0 17.9   14.7

*Including UKIP until 2014   #Local elections covered all of Wales except for Ynys Môn.


Overall, the performance of these final polls was pretty good, without being flawless. As one would probably expect, the polls were more accurate in the highest turnout elections – the two general elections, where there was a mean average error per party of 1.5% in 2010 and 0.82% in 2015. (Rather ironically, it was in 2015 – supposedly the disastrous year for the pollsters – that YouGov’s Welsh polls were the most accurate of all the six ballots observed here!) The final poll was least accurate in the lowest turnout election, the 2014 European Parliament one, where there was a mean error per party of 4.31%.

But where did these errors come from? There are some fascinating differences that emerge when we compare the parties and how their electoral performance relates to the final opinion polls. Labour has under-performed its poll rating in all of the elections except that of 2010; for the Conservatives, the exact opposite was the case apart from the 2012 local elections. (And local elections may be a special case for the Conservatives, given the strong continuing tradition of small-c conservative Independent councillors in much of Wales). Plaid Cymru’s support has thus far always been pretty accurately estimated by YouGov, who twice have been within 0.1 percentage points of their vote share, and never more than 1.3% out; however, on four out of the six instances Plaid have been (slightly) under-estimated. The Liberal Democrats’ support has been over-estimated slightly more often than not, while on both occasions since they emerged as a major party UKIP’s vote share has been under-stated by YouGov – although the extent of the under-statement was much smaller in the general election than in 2014’s European election.

That 2010 was the one election in which Labour support was under-stated, and Conservative support over-stated outside of local elections, could well be more than just a coincidence. Before the May 2015 general election, Matt Singh of Number Cruncher Politics conducted an extensive review of the performance of the GB-wide polls against final election vote shares over recent decades. This review enabled Matt to be just about the only person I know of who publicly predicted, before the fact, the errors made by the opinion polls in last year’s general election. (Indeed, Matt not only predicted that the polls would get things wrong, but predicted almost precisely the magnitude of their error. And he apparently backed this up with some bets, so making money out of it!). Matt suggested that 2010 was something of an outlier to much longer-standing patterns; perhaps because of the turbulence caused by the mid-campaign eruption of ‘Clegg-mania’, it was atypical to the normal relationships between opinion polls and general elections outcomes, patterns which emphatically re-asserted themselves in 2015.

The 2010 general election aside, Labour’s recent record in Wales has been one of under-performing its opinion poll rating in elections. Moreover, that tendency appears more marked in lower-turnout contests like the Assembly and European elections. The Conservatives, by contrast, tend to do better than the polls suggest.

I’m certainly not saying that this is some sort of immutable, iron law of Welsh electoral politics. We don’t have nearly enough data points to indicate that. Moreover, recent changes by YouGov to their methodology – which should be rolled out in full for the first time in Wales with our next Barometer poll – may help to reduce the disparities we observe here.

What I am saying is that the evidence seems to suggest that Welsh Labour have been having some problems getting their vote out in recent years; moreover, those problems seem to have been of greater magnitude in those elections that generally attract a lower voter turnout. Unless Labour can successfully address that problem before May, this year’s National Assembly election might just turn out to be a little more competitive than the polls would currently seem to indicate is probable.


  • Degsy

    Brilliant analysis, Roger. Unfortunately for Wales, even if Labour’s vote does collapse in May, we will still almost certainly end up with a Labour FM, as Leanne Wood has telegraphed her support for the other “socialist” party.

    If Plaid lose enought seats on the list to UKIP, Leanne may do the decent thing and resign. Maybe a new Plaid leader will be the one to say “no” to propping up this failed Labour regime.

  • Nigel Marriott

    Hi Roger,

    Matt did a great job and I have met him since and remain in contact with him. However, he wasn’t the only one to predict before the election that the polls would be wrong. My own prediction was placed on YouTube 2 days before the election and I stated the polls would be wrong but I only predicted a 4 pt gap between Labour & Conservatives compared to the actual 7pt gap on the night and the 0/1 pt gap implied by the polls. This was based on general elections back to 1992 and also local & Eu elections in between 2010 & 2015. You can find the clip here https://www.youtube.com/watch?v=b9vl6xrTIc0&feature=youtu.be

    • Roger Scully

      Nigel – Thanks for that, I’ll watch it tonight with considerable interest. In my own defence, I did qualify my statement: Matt was ‘just about the only person I know of’, or something like that! There were bound to be some others – I just didn’t know of them. R

  • J. Jones

    Interesting “where did it all go wrong?” analysis coming out. The difficulty in finding Tories…are they more likely to be “busy” or to think that their voting intention is no one’s business but their own? And the pollsters “herding” instinct; the inclination to avoid being “out of line” with other pollsters. In Wales it really looks as though Plaid supporters are always available and always do what they say that they will do. Very cohesive, but does that mean that there aren’t many people who are on the verge of voting Plaid…maybe I shall maybe I shan’t?

    • Geraint

      With more parties standing in the PCC elections and there being more competative races in prospect it will be interesting to see how the second preferences work out.

Leave a Reply

Your email address will not be published. Required fields are marked *