OpinionPolitics This article is more than 1 year oldWhy do polling firms like YouGov tweak polls? Because they are scared of being wrong
This article is more than 1 year oldPeter KellnerThe pressure to avoid mistakes can force pollsters to second-guess themselves, just as YouGov did in 2017
- Peter Kellner is a former president of YouGov
Election polling is a tough business. My former YouGov colleague Chris Curtis let the cat out of the company’s bag today. On Twitter, he detailed how the team second-guessed their own polls showing a shrinking Tory lead and likely hung parliament before the 2017 election. After being off on several high-profile predictions they were put under enormous pressure to not get it wrong, and ultimately tweaked their methods in subsequent polls.
We now know that was incorrect. YouGov’s adjustment turned an excellent poll into a mediocre one. The Tory lead in votes was just 2.5 percentage points, and Theresa May lost her majority. So YouGov was mistaken to adjust its final poll, but was it a culpable mistake? This is a much harder question to answer.
I have been reporting or conducting polls for half a century. I was the chairman of YouGov from 2001 to 2007, and its president from 2007 until 2016. It is common knowledge that final election polls are sometimes tweaked. This is because pollsters are determined to produce figures as close as possible to the election result. If they are convinced by the data that there are good reasons to make minor methodology changes, that is a judgment call, not a criminal offence.
These decisions don’t take place in ideal conditions; there is immense pressure to get it right. In 2015, YouGov got it wrong, along with every other polling company. We showed the two main parties neck-and-neck. In fact, David Cameron led the Tories to outright victory with a 6.5 percentage point lead over Labour. As I was part of the BBC’s election night team covering the results, I had to own up live on air. Our previous polling successes counted for nothing. In 2017, every pollster hoped to do better.
In the lead-up to the election, YouGov published a series of polls showing (rightly) that the Tory lead was shrinking. One poll in particular attracted a huge amount of attention. On 31 May, the Times’s front page proclaimed that YouGov, using a huge sample and a brand-new polling methodology, showed that the election was heading for a hung parliament.
In a commentary accompanying the news story, Stephan Shakespeare, YouGov’s CEO, gave his “midpoint” projection: Conservatives 310, Labour 257. YouGov was mocked by the Tories – including by the YouGov co-founder, Nadhim Zahawi, who had left the company in 2010 to become an MP. Reportedly Zahawi called Shakespeare and cautioned him not to get the prediction wrong. (Zahawi said today that this wasn’t a threat but a joke between friends.)
On the afternoon of 7 June, YouGov had to deliver its final forecast to the Times. This was a conventional survey, and the one that would enter the records as YouGov’s formal prediction. I have been told that the figures, using YouGov’s normal methods of weighting the raw data to make sure the figures accurately reflected Britain’s electorate, showed the Conservatives three points ahead – the same as in the much-mocked poll the previous week.
The decision YouGov had to take was whether to go with these numbers or to adjust them. According to Curtis, “there were a few ‘minor’ methodology changes for the final poll which increased the Tory lead. This was done after pressure from high-ups (and despite protests from those of us who thought it wasn’t OK).”
The effect of the process was to move two percentage points from Labour to Conservative, and increase the Tory lead from three points to seven points. That evening I encountered one of the Times’s political staff in Westminster. They were mightily relieved by YouGov’s figures, for they felt exposed after their previous front page story. They were sure May would lead her party to a comfortable victory, and were reassured that their own pollster concurred. Of course, they were wrong. The final tally of the election: Con 318, Lab 262.
In 2017, YouGov was especially nervous. It had wrongly predicted a hung parliament in 2015 and said just after 10pm on 23 June 2016 that Remain had won the Brexit referendum. It did not want to be wrong for the third year running. If it stuck with a three percentage point Tory lead, it would vindicate YouGov’s consistent story in the final stages of the campaign. But if there was in fact a clear Tory victory, that would look really bad.
After the election, YouGov justifiably trumpeted its hung-parliament poll, which not only got the overall result right, but gave a good account of the kinds of seats that would change hands. It kept quiet about its final, seven-point lead poll.
That YouGov entered election day with, in effect, two different stories about what would happen was not really improper. But after the failures of the previous two years, it was certainly convenient.
Peter Kellner is a former president of YouGov
ncG1vNJzZmivp6x7tbTEoKyaqpSerq96wqikaJuforqmutOiqp%2BqlZp8c3yRa2ajrZ5kfXl7z6ijpaGenHqntdGmqmaxn6q0sMKMra6emZtivbC4y6w%3D