The Pragati Podcast is available on all major podcast apps.
Those of you who might be following me on social media (Twitter/Facebook/LinkedIN) might know that I’ve started a career in TV Punditry over the last week. Well, it’s not that much of a career – I still need to figure out how to get paid for it.
Anyway, so I was on News9 once on Saturday (analysing exit polls) and again on Tuesday (analysing the election results). It happened pretty much at random, from a random twitter conversation:
yeah there till 28th. fix one election TV appearance for me 😛
— Karthik Shashidhar (@karthiks) May 1, 2018
And so Mathang (who I’d first met in 2004 when he had interviewed me for Education Times) set me up with Anil Kumar from News9, who presently asked me for my number. A couple of twitter DMs, a couple of emails and a couple of phone calls later, I had been asked to come to the News9 studio at 5pm on Saturday.
Saturday’s session was really enjoyable, and I spoke a fair bit on the process of conducting an exit poll, the importance of sample sizes and representative samples, the process of converting votes to seats, etc. A 5 minute monologue on sampling process got the anchors interested in me, and they kept coming back to me. As is my wont, I summarised the import of my arguments for Mint.
And so I got invited again for Tuesday’s post-counting session, and I’m not sure I enjoyed it that much. As the elections threw up a hung assembly, the politicians on the panel spent their time shouting at each other. I was seated in an inappropriate place – right between a loud JDS spokesperson and a loud BJP spokesperson. I recused myself from much of the discussion and was only brought in because the anchors probably thought I should be “given some lines” – an opportunity I used to comment on the parties’ election strategies.
So two TV appearances later, I must say I quite like the format – it’s good footage (literally) if not anything else, but it can be a bit painful. Writing is easy in the sense that you just collect your thoughts and deliver them at a time.
Video means that you are virtually participating in a group discussion, and need to butt in to make your point. You might have something insightful to say, but need to wait for an opportune time to interject. You might be in the middle of a long point but get interrupted by another panellist. You might wait for ages to say something but the opportunity never comes. At other times, you might get a question that you’re not prepared for.
The worst thing as an analytical guy on TV is that you need to keep referring to your data, and your analysis. So there was one occasion on each session when the anchors asked me a question to answer which I’d to write some code to answer. So each time I mumbled something and bent down to my laptop, and got bailed out by the anchor who got someone else’s view in the time I took to get the requisite data.
In any case, I want to do more of this. I also hope that like with my writing, I can some day hope to get paid for TV appearances – this is a hard job since panellists representing political parties don’t charge anything – it’s in their parties’ interests to be represented on the show.
But, some day..
So for a change the opinion polls seem to have got it right. I’m talking about the just-concluded elections in the UK here, which has returned a hung parliament. The Tories have fallen just sort of a majority (in Kannada we’d call it “AJM“). It’ll be interesting to see how a government will be formed now.
Now, the thing is that the opinion polls got it right. While the Tories had started off with a big lead at the time the elections were announced, opinion polls over time showed that the race was getting a lot tighter. I’d piggybacked on the opinion polls to conduct my own analysis which got published in Mint.
Having shown off that I’d made the prediction correctly, let me get to my hypothesis of why the opinion polls got it right. Opinion polls in the UK have a greater chance of being right because because betting is legal here.
Howzzat! I'd predicted 3-4 days back, based on opinion polls that a hung parliament is likely in the UK! https://t.co/cWeoJ6jfzF
— karthik (@karthiks) June 9, 2017
I was walking around Central London yesterday when I saw this poster outside a betting shop.
Because betting is legal in the UK, betting houses take bets on just about anything, including the results of elections. The way betting works is that the betting houses make markets. They present odds for each side of the deal (in this case, let’s say Tory win, Labour win and hung parliament), and whenever a punter walks into the shop and places a bet, it’s the house that’s taking the opposite side of the bet.
What this implies is that the house better get the odds right, otherwise the difference in their odds and the actual results can wipe out the shop. And how does the betting house know where to set the odds? For something like an election, they rely on the opinion polls.
If the opinion polls get it wrong, the betting houses can end up losing a lot of money (like they evidently did last year during the Brexit vote which most pollsters got horribly wrong). So there is a legal entity which has real skin in the game in opinion polls being right.
I’m not sure of the ownership of the opinion polling companies here in the UK, but I won’t be surprised if they make plenty of money by selling their results to betting shops (at a more granular level than what they make public). And given the intense competition among pollsters here in the UK (at least 15 different agencies conducted opinion polls ahead of yesterday’s elections), there is a real incentive for a pollster to get it right – get it wrong and the betting houses might take their business elsewhere.
In case betting wasn’t legal (such as in India), polling agencies wouldn’t be able to legally sell their results to betting houses and punters, and their markets would be limited to media houses. Media houses don’t have that much of a skin in the game in the polls – their profits don’t depend on getting polls right as much as the profits of betting houses. And pollsters would have less incentive to get the polls right.
Going through an old piece I’d written for Mint, while conducting research for something I’m planning to write, I realise that I’d come rather close to staking claim as a great election forecaster. As it happened, I just didn’t have the balls to stick my neck out (yes, mixed metaphors and all that) and so I missed the chance to be a hero.
I was writing a piece on election forecasting, and the art of converting vote shares into seat shares, which is tricky business in a first past the post system such as India. I was trying to explain how the number of “corners of contests” can have an impact on what seat share a particular vote share can translate to, and I wrote about Uttar Pradesh.
Quoting from my article:
An opinion poll conducted by CNN-IBN and CSDS whose results were published last week predicted that in Uttar Pradesh, the Bharatiya Janata Party is likely to get 38% of the vote. The survey reported that this will translate to about 41-49 seats for the BJP. What does our model above say?
If you look at the graph for the four-cornered contest closely (figure 4), you will notice that 38% vote share literally falls off the chart. Only once before has a party secured over 30% of the vote in a four-cornered contest (Congress in relatively tiny Haryana in 2004, with 42%) and on that occasion went on to get 90% of the seats (nine out of 10).
Given that this number (38%) falls outside the range we have noticed historically for a four-cornered contest, it makes it unpredictable. What we can say, however, is that if a party can manage to get 38% of the votes in a four-cornered state such as Uttar Pradesh, it will go on to win a lot of seats.
As it turned out, the BJP did win nearly 90% of all seats in the state (71 out of 80 to be precise), stumping most election forecasters. As you can see, I had it all right there, except that I didn’t put it in that many words – I chickened out by saying “a lot of seats”. And so I’m still known as “the guy who writes on election data for Mint” rather than “that great election forecaster”.
Then again, you don’t want to be too visible with the predictions you make, and India’s second largest business newspaper is definitely not an “obscure place”. As I’d written a long time back regarding financial forecasts,
…take your outrageous prediction and outrageous reasons and publish a paper. It should ideally be in a mid-table journal – the top journals will never accept anything this outrageous, and you won’t want too much footage for it also.
In all probability your prediction won’t come true. Remember – it was outrageous. No harm with that. Just burn that journal in your safe (I mean take it out of the safe before you burn it). There is a small chance of your prediction coming true. In all likelihood it wont, but just in case it does, pull that journal out of that safe and call in your journalist friends. You will be the toast of the international press.
So maybe choosing to not take the risk with my forecast was a rational decision after all. Just that it doesn’t appear so in hindsight.
The Bruhat Bengaluru Mahanagara Palike (BBMP) went to polls on Saturday, and votes were counted today. The BJP has retained its majority in the house, winning 100 out of 198 available seats. While this is a downer compared to the 113 seats they had won in the previous elections in 2010, the fact that they have won despite being in opposition in the state is a significant achievement.
Based on data put out by Citizen Matters, here is some rudimentary analysis. The first is a choropleth of where each party has performed. Note that this is likely to be misleading since constituencies with large areas are over-represented, but this can give you a good picture.
Notice that there are a few “bands” where the BJP has performed really well. There is the south-western part of the city that it has literally swept, and it has done well in the south-eastern and northern suburbs, too. And the party hasn’t done too well in the north-west, the traditional “cantonment” area.
We can get a better picture of this by looking at the choropleths by assembly constituency. These shapes might be familiar to regular readers of the blog since I’d done one post on gerrymandering.
This tells you where each party has done well. As was evident from the first figure, the BJP has done rather well in Basavanagudi, Jayanagar and Padmanabhanagar in “traditional South Bangalore” and blanked out in Pulakeshi Nagar in the cantonment. In fact, if you try to correlate these results with that of the last Assembly elections, the correlation is rather strong. Most constituencies have gone the way of the assembly segments they are part of.
Then there is the issue of reservation – there were a lot of murmurs that the Congress party which is in power in the state changed the reservations to suit itself. Yet, there are a few interesting factoids that indicate that the new set of reservations were rather logical.
The next two graphs show the distribution of SC and ST populations respectively as a function of the reservations of the constituencies (population data from http://openbangalore.org).
Notice that the constituencies reserved for SCs and STs are among those that have the highest SC and ST populations respectively. The trick in gerrymandering was in terms of distributions between general and OBC constituencies, and among women.
I could put the performance of different parties by reservation categories, and on whether the reservation of a constituency has changed has any effect on results, but (un)fortunately, there aren’t any trends, and consequently there is little information content. Hence I won’t bother putting them in.
Nevertheless, these have been extremely interesting elections. All the postponement, all the drama and court case, and finally the underdog (based on previous trends) winning. Yet, given the structure of the corporation, there is little hope that much good will come of the city in the coming years. And there is nothing in the election results that can alter this.
Kindly note that this post falls under the category of “political gossip” and not under the category of “policy analysis” that this blog is mostly filled with
So the BJP has got trounced in the by elections that were counted yesterday. People have been quick to call this a referendum on Modi’s government and are asking him to change course (each commentator is calling for a change of course in a different direction – possibly with the vector sum of them being nothing). I got an email this morning asking for reasons of the BJP’s poor performance and this is what I wrote back:
So in that one vote that you have, you need to collectively express a range of emotions – like which party you want to form the government, who you want the prime minister to be, who will take best care of your community in your constituency, who is the best person to represent your constituency in the assembly, which local person you can turn to in times of trouble, etc. (it’s a very long list). So your vote is essentially a weighted average of your emotions in all these aspects.In the elections in May, thanks to the non-existence of a government for a very long time, the weights given to a stable and strong government at the centre and choice of prime minister shot up. Like crazy. And it was clear before the elections that there was only one party and one man who could offer this kind of a government.Since the weight given to this factor in the minds of people was so high, it trumped everything else, and even the proverbial lamppost on a BJP ticket (especially in Uttar Pradesh) managed to get elected! And thus we got a party with full majority. And we got the desired man as PM. And we will most likely have a stable government for the next five years.A bypoll is different – especially when you have a small number of by polls they simply don’t affect who forms the government and who the prime minister should be. Thus, the weight given to those elements of the vector, which were extremely high in May,were set to zero. Thanks to that, people voted based on the other components – like caste, local dominance, community support and all that. In that respect I’m not surprised at all in terms of the result.Also it’s not fair to compare the performance in these bye-elections to the party performance in the respective assembly segments in the lok sabha elections. What we should compare these bypolls to is to the parties that held these seats before they fell vacant. The media has once again succeeded in distorting the narrative to come to hopefully desired conclusions?
For those of you who are not particularly aware of it, for the last year and a half I’ve been writing this column called Election Metrics for Mint. It’s basically a quantitative take on elections, and in my estimate I should’ve written over 50 pieces for them so far.
The last two pieces, however, have been different in the sense that I have now moved beyond covering Indian elections to look at elections abroad. In my last but one post, published last month, i took a look at potential cheating in Afghan elections. (Now I remember linking to that piece from here).
Now, in the latest piece that was published today I look at the forthcoming Scottish referendum, and a recent poll by YouGov in which 47% of respondents said they wanted to vote in favour of independence. I use some binomial jugglery that shows that this translates to a 2.5% chance of a Yes vote, which while insignificant, is an order of magnitude higher than the 0.0004% chance of “Yes” that can be implied from an earlier poll.
I then use the “possible, plausible and probable” framework made famous by Bill Gurley and Aswath Damodaran in their “exchange” in July to show why this poll is significant (it shows that a “Yes” vote is “plausible”, while earlier it was possible but definitely not plausible).