The result of last year’s General Election in the UK came as a surprise to almost everyone, most of all to those who were paid to predict it, the pollsters. Even the most successful pollster in the world, Nate Silver of FiveThirtyEight  (see Reading the Tea Leaves), got it spectacularly wrong. How could this happen?

Six months of official soul-searching has now, it seems, delivered the answer. The pollsters didn’t try hard enough.

Polling is difficult. You’d think that it would be easy enough to find a ‘random’ sample of the population and pop a question or two, but of course it’s not. Counterintuitive though it may seem, a ‘random’ sample has to be chosen very carefully if it’s to be representative of the whole country. You’ve got to start with the right list (a telephone directory isn’t, at least nowadays, a good place to start) and then iron out all the socioeconomic differences that your sample might reflect. I am no expert, but I suppose these might be gender, occupation, location, age, etc.

And it seems that in this respect the pollsters didn’t get it wrong. Their lists were good. What they got wrong was that they were not persistent enough in contacting everyone on their list. It turns out, following recent research, that Tories are harder to get hold of.

Various theories have been put forward for the unexpected election result – the theory of the ‘shy’ Tory (which I thought plausible), which suggests that some Tories don’t tell the truth to pollsters about their real voting intentions. Then there’s the theory of the ‘late swing’.

But it turns out, following more research, that when pollsters went back to the same samples they had used before the election and asked their victims how they had actually voted on Election Day, they got the same result (of course, it’s possible that the shy Tories were still lying).

But another crucial piece of research found that if pollsters were persistent in tracking down their original sample, not those they actually polled, but all of those they intended to poll, even if this took two weeks or more, they got a result that was close the final election result. Those hardest to track down, it seems, were more likely to vote Tory. Add this anomaly to the received wisdom that Labour voters are less likely than Tory voters actually to show up at the Polling Station on Election Day, and you have an ever more complex formula to devise. Tories are more elusive before the day, and less elusive on it.

The problem is that pre-election opinion polls must be conducted in under two days. You just don’t have two weeks to track down your Tories.

The whole business of polling is horribly complex. I’m reminded of a particular passage about a country-house astrologist in Aldous Huxley’s novel Crome Yellow:

‘Most of Priscilla’s days were spent in casting the horoscopes of horses, and she invested her money scientifically as the Stars dictated. She betted on football too, and had a large notebook in which she registered the horoscopes of all the players in all the teams of the League. The process of balancing the horoscopes of two elevens one against the other was a very delicate and difficult one. A match between Spurs and the Villa entailed a conflict in the heavens so vast and so complicated that it was not to be wondered as if she sometimes made a mistake about the outcome.’

Pollsters must be scratching their heads and wondering how to find the elusive Tory,  rather than extract the truth from the shy one. I’m wondering, though, why this wasn’t a problem in the past.

Or why not abandon the entire enterprise and buy a crystal ball?