What next to predict? I know. Elections!

So…

We can use monte carlo methodology to simulate the last few games of the Premiership and we’ve seen how what we produced was pretty much in line with what the betting markets were predicting (which is normally a good free market that tends to the actual outcome). The challenge then is to try and predict something else with the same methodology. What about the UK General Election in 2015?

Predicting elections is something that I’ve been doing for a while. I set-up Forecast UK to do exactly that for 2010 and I also played the Election Game for a year (coming second overall) before taking over hosting the game alongside Paul Maggs. My best result ever was getting the last Euro elections pretty well spot on for all four major parties (and I look forward to making lots of money again doing that this year).

My approach in the past has been to analyse the various poll results, produced a bespoke weighted average based on adjusting for the different methodologies, and then forecast swings in individual seats (taking account of incumbency) to arrive at an overall forecast. In 2010 I managed to be one seat out on the final Conservative tally, predicted Brighton Pavilion to within 500 votes, but got the Labour and Liberal Democrats horribly wrong (too low for Labour, too high for the Liberal Democrats). That said, everybody else completely over-forecast the Liberal Democrats since they polled really well in the campaign and then failed to deliver on the day of voting.

And so to 2015. This time I want to produce a probability forecast for the next General Election, indicating the likelihood of certain events happening (so, Conservatives winning, Labour and the Liberal Democrats having enough seats between them to form a coalition, that kind of thing). That’s not too hard to do, but there are certain things that need to be tackled first.

Creating a probability distribution isn’t tricky at all – simply calculate a covariance matrix of the different parties based on the latest poll results and apply that to the projected final poll rating for each party. Thing is, what is the final poll rating? Just because today, 14 months before the fixed date of the General Election, Labour is polling around 35.5% doesn’t mean that that’s where they will be in May 2015. To solve this problem I have, inspired by Stephen Fisher’s work here, built a statistical model to calculate the path from today to 2015, looking at how each party’s poll ratings tend to move in the final months before an election. This means that I can take the poll ratings today and make a very educated guess based on past performance on what that means they will look like in a year’s time.

Put that all together and you have a probability model for the 2015 General Election. Calculate the current covariance, test for normality in the opinion polls, project the mean of the multivariate normal probability distribution forward to May 2015 using the regression model and run a monte carol simulation. Easy peasy.

This is all done and works beautifully. The next thing to do is to calculate and incorporate incumbency effects, produce a sub-model for Scotland and put it all together.

Watch this space.