The Pragati Podcast is available on all major podcast apps.
Those of you who might be following me on social media (Twitter/Facebook/LinkedIN) might know that I’ve started a career in TV Punditry over the last week. Well, it’s not that much of a career – I still need to figure out how to get paid for it.
Anyway, so I was on News9 once on Saturday (analysing exit polls) and again on Tuesday (analysing the election results). It happened pretty much at random, from a random twitter conversation:
yeah there till 28th. fix one election TV appearance for me 😛
— Karthik Shashidhar (@karthiks) May 1, 2018
And so Mathang (who I’d first met in 2004 when he had interviewed me for Education Times) set me up with Anil Kumar from News9, who presently asked me for my number. A couple of twitter DMs, a couple of emails and a couple of phone calls later, I had been asked to come to the News9 studio at 5pm on Saturday.
Saturday’s session was really enjoyable, and I spoke a fair bit on the process of conducting an exit poll, the importance of sample sizes and representative samples, the process of converting votes to seats, etc. A 5 minute monologue on sampling process got the anchors interested in me, and they kept coming back to me. As is my wont, I summarised the import of my arguments for Mint.
And so I got invited again for Tuesday’s post-counting session, and I’m not sure I enjoyed it that much. As the elections threw up a hung assembly, the politicians on the panel spent their time shouting at each other. I was seated in an inappropriate place – right between a loud JDS spokesperson and a loud BJP spokesperson. I recused myself from much of the discussion and was only brought in because the anchors probably thought I should be “given some lines” – an opportunity I used to comment on the parties’ election strategies.
So two TV appearances later, I must say I quite like the format – it’s good footage (literally) if not anything else, but it can be a bit painful. Writing is easy in the sense that you just collect your thoughts and deliver them at a time.
Video means that you are virtually participating in a group discussion, and need to butt in to make your point. You might have something insightful to say, but need to wait for an opportune time to interject. You might be in the middle of a long point but get interrupted by another panellist. You might wait for ages to say something but the opportunity never comes. At other times, you might get a question that you’re not prepared for.
The worst thing as an analytical guy on TV is that you need to keep referring to your data, and your analysis. So there was one occasion on each session when the anchors asked me a question to answer which I’d to write some code to answer. So each time I mumbled something and bent down to my laptop, and got bailed out by the anchor who got someone else’s view in the time I took to get the requisite data.
In any case, I want to do more of this. I also hope that like with my writing, I can some day hope to get paid for TV appearances – this is a hard job since panellists representing political parties don’t charge anything – it’s in their parties’ interests to be represented on the show.
But, some day..
So for a change the opinion polls seem to have got it right. I’m talking about the just-concluded elections in the UK here, which has returned a hung parliament. The Tories have fallen just sort of a majority (in Kannada we’d call it “AJM“). It’ll be interesting to see how a government will be formed now.
Now, the thing is that the opinion polls got it right. While the Tories had started off with a big lead at the time the elections were announced, opinion polls over time showed that the race was getting a lot tighter. I’d piggybacked on the opinion polls to conduct my own analysis which got published in Mint.
Having shown off that I’d made the prediction correctly, let me get to my hypothesis of why the opinion polls got it right. Opinion polls in the UK have a greater chance of being right because because betting is legal here.
Howzzat! I'd predicted 3-4 days back, based on opinion polls that a hung parliament is likely in the UK! https://t.co/cWeoJ6jfzF
— karthik (@karthiks) June 9, 2017
I was walking around Central London yesterday when I saw this poster outside a betting shop.
Because betting is legal in the UK, betting houses take bets on just about anything, including the results of elections. The way betting works is that the betting houses make markets. They present odds for each side of the deal (in this case, let’s say Tory win, Labour win and hung parliament), and whenever a punter walks into the shop and places a bet, it’s the house that’s taking the opposite side of the bet.
What this implies is that the house better get the odds right, otherwise the difference in their odds and the actual results can wipe out the shop. And how does the betting house know where to set the odds? For something like an election, they rely on the opinion polls.
If the opinion polls get it wrong, the betting houses can end up losing a lot of money (like they evidently did last year during the Brexit vote which most pollsters got horribly wrong). So there is a legal entity which has real skin in the game in opinion polls being right.
I’m not sure of the ownership of the opinion polling companies here in the UK, but I won’t be surprised if they make plenty of money by selling their results to betting shops (at a more granular level than what they make public). And given the intense competition among pollsters here in the UK (at least 15 different agencies conducted opinion polls ahead of yesterday’s elections), there is a real incentive for a pollster to get it right – get it wrong and the betting houses might take their business elsewhere.
In case betting wasn’t legal (such as in India), polling agencies wouldn’t be able to legally sell their results to betting houses and punters, and their markets would be limited to media houses. Media houses don’t have that much of a skin in the game in the polls – their profits don’t depend on getting polls right as much as the profits of betting houses. And pollsters would have less incentive to get the polls right.
Going through an old piece I’d written for Mint, while conducting research for something I’m planning to write, I realise that I’d come rather close to staking claim as a great election forecaster. As it happened, I just didn’t have the balls to stick my neck out (yes, mixed metaphors and all that) and so I missed the chance to be a hero.
I was writing a piece on election forecasting, and the art of converting vote shares into seat shares, which is tricky business in a first past the post system such as India. I was trying to explain how the number of “corners of contests” can have an impact on what seat share a particular vote share can translate to, and I wrote about Uttar Pradesh.
Quoting from my article:
An opinion poll conducted by CNN-IBN and CSDS whose results were published last week predicted that in Uttar Pradesh, the Bharatiya Janata Party is likely to get 38% of the vote. The survey reported that this will translate to about 41-49 seats for the BJP. What does our model above say?
If you look at the graph for the four-cornered contest closely (figure 4), you will notice that 38% vote share literally falls off the chart. Only once before has a party secured over 30% of the vote in a four-cornered contest (Congress in relatively tiny Haryana in 2004, with 42%) and on that occasion went on to get 90% of the seats (nine out of 10).
Given that this number (38%) falls outside the range we have noticed historically for a four-cornered contest, it makes it unpredictable. What we can say, however, is that if a party can manage to get 38% of the votes in a four-cornered state such as Uttar Pradesh, it will go on to win a lot of seats.
As it turned out, the BJP did win nearly 90% of all seats in the state (71 out of 80 to be precise), stumping most election forecasters. As you can see, I had it all right there, except that I didn’t put it in that many words – I chickened out by saying “a lot of seats”. And so I’m still known as “the guy who writes on election data for Mint” rather than “that great election forecaster”.
Then again, you don’t want to be too visible with the predictions you make, and India’s second largest business newspaper is definitely not an “obscure place”. As I’d written a long time back regarding financial forecasts,
…take your outrageous prediction and outrageous reasons and publish a paper. It should ideally be in a mid-table journal – the top journals will never accept anything this outrageous, and you won’t want too much footage for it also.
In all probability your prediction won’t come true. Remember – it was outrageous. No harm with that. Just burn that journal in your safe (I mean take it out of the safe before you burn it). There is a small chance of your prediction coming true. In all likelihood it wont, but just in case it does, pull that journal out of that safe and call in your journalist friends. You will be the toast of the international press.
So maybe choosing to not take the risk with my forecast was a rational decision after all. Just that it doesn’t appear so in hindsight.
The Bruhat Bengaluru Mahanagara Palike (BBMP) went to polls on Saturday, and votes were counted today. The BJP has retained its majority in the house, winning 100 out of 198 available seats. While this is a downer compared to the 113 seats they had won in the previous elections in 2010, the fact that they have won despite being in opposition in the state is a significant achievement.
Based on data put out by Citizen Matters, here is some rudimentary analysis. The first is a choropleth of where each party has performed. Note that this is likely to be misleading since constituencies with large areas are over-represented, but this can give you a good picture.
Notice that there are a few “bands” where the BJP has performed really well. There is the south-western part of the city that it has literally swept, and it has done well in the south-eastern and northern suburbs, too. And the party hasn’t done too well in the north-west, the traditional “cantonment” area.
We can get a better picture of this by looking at the choropleths by assembly constituency. These shapes might be familiar to regular readers of the blog since I’d done one post on gerrymandering.
This tells you where each party has done well. As was evident from the first figure, the BJP has done rather well in Basavanagudi, Jayanagar and Padmanabhanagar in “traditional South Bangalore” and blanked out in Pulakeshi Nagar in the cantonment. In fact, if you try to correlate these results with that of the last Assembly elections, the correlation is rather strong. Most constituencies have gone the way of the assembly segments they are part of.
Then there is the issue of reservation – there were a lot of murmurs that the Congress party which is in power in the state changed the reservations to suit itself. Yet, there are a few interesting factoids that indicate that the new set of reservations were rather logical.
The next two graphs show the distribution of SC and ST populations respectively as a function of the reservations of the constituencies (population data from http://openbangalore.org).
Notice that the constituencies reserved for SCs and STs are among those that have the highest SC and ST populations respectively. The trick in gerrymandering was in terms of distributions between general and OBC constituencies, and among women.
I could put the performance of different parties by reservation categories, and on whether the reservation of a constituency has changed has any effect on results, but (un)fortunately, there aren’t any trends, and consequently there is little information content. Hence I won’t bother putting them in.
Nevertheless, these have been extremely interesting elections. All the postponement, all the drama and court case, and finally the underdog (based on previous trends) winning. Yet, given the structure of the corporation, there is little hope that much good will come of the city in the coming years. And there is nothing in the election results that can alter this.
Kindly note that this post falls under the category of “political gossip” and not under the category of “policy analysis” that this blog is mostly filled with
So the BJP has got trounced in the by elections that were counted yesterday. People have been quick to call this a referendum on Modi’s government and are asking him to change course (each commentator is calling for a change of course in a different direction – possibly with the vector sum of them being nothing). I got an email this morning asking for reasons of the BJP’s poor performance and this is what I wrote back:
So in that one vote that you have, you need to collectively express a range of emotions – like which party you want to form the government, who you want the prime minister to be, who will take best care of your community in your constituency, who is the best person to represent your constituency in the assembly, which local person you can turn to in times of trouble, etc. (it’s a very long list). So your vote is essentially a weighted average of your emotions in all these aspects.In the elections in May, thanks to the non-existence of a government for a very long time, the weights given to a stable and strong government at the centre and choice of prime minister shot up. Like crazy. And it was clear before the elections that there was only one party and one man who could offer this kind of a government.Since the weight given to this factor in the minds of people was so high, it trumped everything else, and even the proverbial lamppost on a BJP ticket (especially in Uttar Pradesh) managed to get elected! And thus we got a party with full majority. And we got the desired man as PM. And we will most likely have a stable government for the next five years.A bypoll is different – especially when you have a small number of by polls they simply don’t affect who forms the government and who the prime minister should be. Thus, the weight given to those elements of the vector, which were extremely high in May,were set to zero. Thanks to that, people voted based on the other components – like caste, local dominance, community support and all that. In that respect I’m not surprised at all in terms of the result.Also it’s not fair to compare the performance in these bye-elections to the party performance in the respective assembly segments in the lok sabha elections. What we should compare these bypolls to is to the parties that held these seats before they fell vacant. The media has once again succeeded in distorting the narrative to come to hopefully desired conclusions?
For those of you who are not particularly aware of it, for the last year and a half I’ve been writing this column called Election Metrics for Mint. It’s basically a quantitative take on elections, and in my estimate I should’ve written over 50 pieces for them so far.
The last two pieces, however, have been different in the sense that I have now moved beyond covering Indian elections to look at elections abroad. In my last but one post, published last month, i took a look at potential cheating in Afghan elections. (Now I remember linking to that piece from here).
Now, in the latest piece that was published today I look at the forthcoming Scottish referendum, and a recent poll by YouGov in which 47% of respondents said they wanted to vote in favour of independence. I use some binomial jugglery that shows that this translates to a 2.5% chance of a Yes vote, which while insignificant, is an order of magnitude higher than the 0.0004% chance of “Yes” that can be implied from an earlier poll.
I then use the “possible, plausible and probable” framework made famous by Bill Gurley and Aswath Damodaran in their “exchange” in July to show why this poll is significant (it shows that a “Yes” vote is “plausible”, while earlier it was possible but definitely not plausible).
In their excellent book “the dictator’s handbook” Bruno bueno de Mesquita and Alastair smith talk about why dictatorships usually put on a garb of democracy and hold (mostly) sham elections.
According to bueno de Mesquita and smith the reason is not to appear good in front of the international community, as the general discourse goes. Dictators are extremely rational actors, they say, and reputation in the international community didn’t usually give enough benefit to compensate for the cost of the garb of democracy and elections.
Instead, bueno de Mesquita and smith say that the real purpose of the elections is to keep followers in check. If a member of the dictator’s team “misbehaves” for example, getting rid of him is normally a difficult process. Essentially sacking is a hard job for anyone, even for hard nosed dictators. In the context of dictatorships sackings can get controversial and often bloody and is not a particularly pleasant process.
By putting in a garb of democracy, however, there is an easy way to sack an official. Assuming that in a dictatorship most citizens vote according to the fancies of the dictator, all a dictator needs to do to sack an official is to instruct the electorate to vote against the official the next time he is up for reelection. The sacking having been effected by “popular mandate”, the process is easier and likely to be less bloody and troublesome for the dictator.
Now, the question is if we can use this framework to understand the new US-style primary elections that the Indian national congress has been using for candidate selection in some constituencies in the forthcoming elections.
Normally in the congress, like in most other parties in India, candidates for elections are determined top-down, by the party “high command”. The risk with this however is that candidates who did not get a ticket to contest the elections know that for whatever reason the party high command is not in favour of them contesting. This can lead to disillusionment and can lead to defections to rival parties.
In this context a primary election acts as a facade through which the party high command can get its choice of candidates without pissing off those applicants who did not get the ticket. Now the purported message to these unsuccessful applicants is that the next time they should work of getting the support of the party rank and file in their constituency.
In reality however, with the party being high command driven, the rank and file has voted as per the instructions of the high command! The high command thus gets its choice of candidates without losing the support of the unsuccessful candidates.
So why is it that primaries work in the US? For the same reasons that elections work in democracies! In the US parties are truly democratic and organised bottom up. There is no high command there to (credibly) dictate the choices for the rank and file. So the results of the primaries are truly reflective of the opinion of the party rank and file.
In conclusion, given the high command based structure of political parties in India, primaries will not work. Instead they will only end up as instruments in the hands of the party high commands, just like the sham elections on dictatorships.
Firstly, on Tuesday, I got interviewed by this magazine called Information Week. Rather, I had gotten interviewed by them a long time back but the interview appeared on Tuesday. I spoke about the challenges of election forecasting in India and the quality of surveys.
Again on Tuesday, and again on Wednesday, I wrote a pair of articles for Mint analyzing constituencies and parties. On Tuesday, I analyzed constituencies whose representatives have always belonged to ruling parties in the last 4 elections. There are 34 such constituencies. Then on Wednesday I wrote about the influence of states in the Lok Sabha, analyzing the proportion of MPs from each major state that was part of the ruling coalition.
If I had forgotten to mention earlier, I have a deal with Mint that lasts till next October where each month I’m supposed to write 3 articles on election data. You can find all my articles so far here.
Then, today, Pragati published my review of the book Why Nations Fail by Acemoglu and Robinson. More than a book on economics or institutions, it is an awesome history book. Get it.
And in the midst of all this, right here, I wrote a “worky” post about the pros and cons of having a dedicated analytics team.
And if you didn’t notice, this website now has “new clothes”. It was a rather long-pending change and the most important feature of the new layout is that it is “responsive”, and thus looks much better on smartphones. I’ve heard a couple of issues with it already, and do let me know if you have any more issues. And for the first time last night I opened this blog on an iPad and I find that it looks fantabulous, thanks to the OnSwipe plugin I use.
Recently I met my MLA, who is from the BJP, and told him about my analysis extrapolating from the recent urban local body elections, which gave Congress an absolute majority. He countered that the BJP has never been strong at the sub-state level so one shouldn’t read too much into these elections. So I decided to create this tool which uses a slider which you can use to decide how much importance to give to the ULB polls. A value of 0 represents the seat distribution as per the 2008 elections. A value of 1 uses an extrapolation of the ULB elections only (without using information from the 2008 polls). I hope you have fun with this tool.
You might also notice that I now give you the actual seat distribution party-wise.