Polling

Trying to Get it Right

Assiduous Blog readers (all of you, I’m sure) will recall July’s YouGov poll for Elections in Wales. One aspect which elicited comment, and to which I devoted a subsequent post, was the Assembly list vote results. These suggested that Labour support was 21% lower on the list vote than Assembly constituency vote; that Plaid Cymru were only 2% behind Labour on the list vote; and that UKIP were in third place on the list vote.

These findings struck many – including me – as implausible. One possible explanation was a change in question wording introduced by YouGov in April 2012. The new wording tried to better explain to respondents the two-ballot system used for devolved elections; but in doing so referred to the list vote as ‘your second vote’. This, I summised, might mislead some respondents into offering their second preference, thus distorting the overall results.

Working with YouGov on various studies since 2009, I’ve been very impressed by their willingness to test out ideas, and their determination to try to get things right. After discussions with them, we agreed an experiment to test whether my summise was correct.

Members of the YouGov Welsh panel all received the same (standard) question on Westminster voting intention. However, we then split the sample into three (roughly equal) groups.

 

Group 1 received the following question wording:

“If there were an election to the National Assembly for Wales tomorrow, and thinking about the constituency vote, how would you vote? And thinking about the regional or party vote for the National Assembly for Wales, which party list would you vote for?”

This is the wording YouGov used in Welsh polls from 2009 through to February 2012. The results of the 2011 Assembly election suggested that this wording tended to over-report Labour support (the final poll in the campaign reported Labour list vote intention at 43%; Labour’s actual list vote was 37%); however, this does follow a much longer tradition of Welsh polls tending to over-state Labour support.

 Group 2 received the following:

“In elections to the National Assembly for Wales you have two votes. One is for an individual member of the Assembly – or AM – for your constituency. The second is for a party list for your region. If there were a National Assembly for Wales election tomorrow, which party would you vote for in your constituency? Now thinking about your second vote, for a party list in your region, which party would you vote for?”

This is the wording that YouGov have been using in Wales since April 2012 – with the apparent problems discussed above.

 Finally, Group 3 received the following:

 “In elections to the National Assembly for Wales you have two votes. One is for an individual member of the Assembly – or AM – for your constituency. The second is for a party list for your region. If there were a National Assembly for Wales election tomorrow, which party would you vote for in your constituency? Now thinking about the regional or party vote for the National Assembly for Wales, which party list would you vote for?”

The wording here is subtly different from that used in recent polls: the last part removes the suggestion of the list vote being a ‘second’ vote.

So, what did we find? First, the findings on the Westminster and Assembly constituency voting intention:

Westminster

Group 1

Group 2

Group 3

Labour

44

40

43

Conservative

21

23

21

LibDems

5

5

4

Plaid

14

14

15

UKIP

10

13

13

Other

6

5

5

NAW Constituency

 

 

 

Labour

42

38

40

Conservative

19

21

19

LibDems

5

7

5

Plaid

23

22

23

Other

12

12

13

These findings should be read with a very strong qualification. For rather complex reasons to do with how we constructed the test, and the weightings of the sub-samples, these figures are NOT directly comparable with those of previous YouGov polls in Wales.  So, opponents of Labour who see Labour’s figures here apparently dropping from previous YouGov polls should not get too excited! What matters are the relative figures across the three groups. These suggest that Group 1 is marginally the most pro-Labour, and Group 2 marginally the least (and the most pro-Conservative). But they also suggest that the three sub-samples (each comprising well over 1000 respondents) are only slightly different.

Therefore, we should expect the three groups’ list voting support also to be similar – unless question wording is having a big impact. So what do we find on list voting intention? Here are the figures:

 NAW List

Group 1

Group 2

Group 3

Labour

39

19

35

Conservative

18

16

18

LibDems

4

8

5

Plaid

21

24

21

UKIP

9

20

14

Green

5

7

4

Other

4

6

4

 A YouGov colleague described these results as ‘frankly astonishing’, to which I can only agree. One reads about ‘question wording effects’ in social surveys, but you very rarely see them demonstrated so dramatically.

It is clear that the wording that has been used since April 2012 has been having a big impact on the findings produced. Of course, we don’t know for sure what is the ‘true’ figure for party list vote support. However, past experience from previous Assembly elections suggests that Labour does between 2-5% worse on the list vote than the constituency vote, not 19% worse! Given this, and in the light of these findings, it seems incontrovertible that the list vote question wording used in polls since April 2012 has been distorting the results.

In discussions with YouGov, we have agreed that the wording received by Group 3 is probably the best way forward. That is what we intend to implement in all polls from now on until the May 2016 National Assembly election – and I hope that such polls will soon become more regular events. In the meantime, much kudos to YouGov for expending the time and resources on making sure that we get as good a measure of party support as possible.

The obvious implication of changing the question wording is that reported levels of list vote support for Plaid Cymru and UKIP are likely to fall, and support for Labour to rise. Again, people should not get themselves too excited about this – it is a question wording effect. Once the new wording has bedded down in a few surveys, we will then be able to follow trends in party support.

Comments

  • Harry Hayfield

    Perhaps we need to drop words and show pictures instead in the form of a ballot paper? I have often thought that despite the fact it got it completely wrong, the 1992 general election exit poll (of getting people to mark a mock ballot paper) might actually be a way forward with regard to polling. The polling companies have so much information these days, I am sure they could easily post out or e-mail people with a mock ballot paper and say “We have been commissioned to conduct a poll on the Assembly of Wales. Please find enclosed three mock ballot papers and follow the instructions then then return them to us” with the mock ballot papers being a Westminster election, Assembly constituency and Assembly regional list.

  • Roger Scully

    Interesting idea, Harry.

    Of course, it wouldn’t work with telephone polls. But I believe that during election time some face-to-face pollsters have tried something like this – can’t recall if it made much difference to anything. Outside election time, it is more difficult – at least for constituency elections – because you don’t have the names of candidates confirmed.

    I’m not sure if YouGov have tried something like this. Of course, you’d still need SOME words, at least to explain what the ballot papers were for. And, arguably, asking the question on-line presents people with something that looks not that difference from a real ballot paper. Anyway, worth pondering a bit further.

  • Dafydd Trystan

    Very interesting post, and a welcome development. I’ve always been concerned by a massive shift between constituency and regional voting intention, when all of the election survey evidence has suggested at most one in four voters moving – and then in ways that often cancelled each other out. Da iawn yougov – a’r Athro Scully wrth gwrs!

  • J Jones

    A very welcome piece of research…..now I expect that you will be doing the same sort of analysis on the difference between the RMG/IWA poll on devolution of income tax varying powers and the remarkable (laughable) result arrived at in the Silk Commission polling. Only joking Roger….you can take academic integrity too far.

    • Roger Scully

      I’ll try to take this comment in the spirit in which I hope it was intended, Jon.

      I hope to have some time in the new year to explore the issues that you have raised. The autumn has not exactly been quiet.

      And I don’t think you can ever take academic integrity too far.

      • Jon Jones

        Yes Roger; just joking…I think. But the questions that are raised by polling can be bigger than the answers that they give us. It’s a serious business when the survey undertaken forms part of the evidence of an “independent” Commission that goes on to give in depth recommendations based on their research.
        It’s perhaps a good idea to ask beforehand what is wanted as the end result of any polling; do you want the “Opinion” (feeling at the time) of the respondents? Do you want “Prediction” (what will you do at the election?). Or do you want “Persuasion” (in the context of this poll, how can I influence you? Using this poll how can I influence others?).

        I really like the approach of people on “Politicalbetting” because they are on the whole trying to use polling as a predictive tool for the obvious reason that accurate prediction leads to financial gain. This is the most honest approach to polling analysis.

        I return to my now (tediously) familiar position; the Silk polls asked too many questions and the context influenced the outcome. The warning sign was the very obvious difference in the self reported ability to speak Welsh when compared to the factual situation….16% of adults speak Welsh to some degree. No one finds more than 25% of adults willing to say that they speak Welsh to some degree, let alone 40% or 60%.

        I could, I think, design an instant poll that shows the public in Wales as anti-further devolution:

        No offer of Welsh or English option for answering.
        Thinking about the economy:
        Do you feel that it has;
        Got better;
        Got Worse;
        Remained the same since Devolution?

        Thinking about Education in Wales:
        Do you feel that outcomes have got better;
        Got Worse;
        Remained the same since Devolution?

        Thinking about the NHS in Wales:
        Do you feel that it has got better;
        Got worse;
        Remained the same since devolution?

        In the future would you like to see;
        More powers devolved to the Welsh Assembly;
        Some powers returned to Westminster.
        Leave things as they are.

        That should do it. In the wake of PISA and publicity for a 6 hour wait in an ambulance outside a Welsh hospital I think that I’ve got a poll to show little enthusiasm for further devolution.

        In this case I’m depending on context both within the poll and outside the poll to influence the outcome.

          • Jon Jones

            I often feel frustrated that polls don’t ask questions that we (people in Wales) need to know the answers to. However it is possible to arrange this with Yougov……

            All you need to do Roger is think up a question, unrelated to voting intention and to be asked after all voting intention questions in a Yougov poll and put it up on your site. People like me, who would pay a fiver to see the answer to some burning question, would pledge the money to Yougov who, when they got enough money to make it worthwhile, would run the question.

            “Do you think that the Welsh Government should concentrate transport funding on East-West or North-South links in Wales?”

            Something of current interest. It would be interesting to see if you could actually access the voice of the mythical “Silent Majority”.

  • Michael Haggett @ Syniadau

    Very enlightening. Although the group 3 question is undoubtedly better, I’m concerned that it still contains the word “second”.

    At first I thought of replacing “the second” with “the other”. But this might lead some people to think that the regional vote must be for another party.

    So I’d like to suggest this as an alternative:

    “In elections to the National Assembly for Wales you have two votes: one for an individual member of the Assembly – or AM – for your constituency, and one for a party list for your region. If there were a National Assembly for Wales election tomorrow, which party would you vote for in your constituency? Now thinking about the regional or party vote for the National Assembly for Wales, which party list would you vote for?”

    • Roger Scully

      Interesting suggestion, Michael.

      For now, I think we will stick with what we have. It seems to be avoiding the problems with the former question.

      Still, all suggestions gratefully received!

  • Jon Jones

    Re-reading your piece (which I admit is a valuable, eye opening piece of research) it suddenly occurred to me that option 2 SHOULD be the one that is closest to voting intentions. Furthermore, if it was a genuine poll, it would tell us a lot about how people would like to see the Assembly made up.

    What, after all, is the point of 35% or 39% of voters voting Labour in the regional voting?

  • Tony Payne

    Before election day in Wales , and the public scrap between parties that will ensue up to election day. Let us take 1% of the 15billion annual budget ,and split it 6 ways between the opposing parties.
    Let each party prioritise their spending ,of the share allotted .and let the public decide who spent their one sixth of one percent in the most responsible way.
    Now we might get past the diatribe and earn ourselves a 1 to 6 odds result in financial order,which would be most useful when we vote in May.

Leave a Reply

Your email address will not be published. Required fields are marked *