Context sensitive and context free journalism

My wife thinks that most of my writing for Mint is incredibly boring, and of a significantly inferior quality to what I publish on this blog. Initially I thought it was because I was taking myself too seriously for Mint (a national newspaper and all that), but despite my attempts to “loosen up”, and write my Mint articles “in flow”, the criticism continues.

Now thinking about it, one reason why my writing for Mint might be boring is that in the beginning of each piece I try hard to provide context. I write without assuming that my readers know exactly what I’m writing about, and so I spend some time giving sufficient background so that my readers appreciate my articles (and most of my pieces appear with minimal edits).

This, I realise now, is not how Indian journalism usually works, for in a large majority of pieces it is assumed that the reader has context. Many a time I find myself reading a piece and not being able to make the head or tail of it, and then going back and trying to figure out the context. The point of several articles I’ve read has become clear only in retrospect, when I’ve found something else that this article was referencing (but didn’t link to).

On the other hand, a “newspaper” like The Economist mostly does context-free journalism. Every piece comes with sufficient background for the reader to know what is happening. There may not be links to find out more, but the limited context provided is enough to understand the point of the piece. Maybe that they are a weekly, and cover news from all over the world makes them want to provide context?

In any case, I find context-sensitive journalism (like what most daily newspapers practice) irritating. Or maybe it is that they haven’t really made the transition from print to digital? Print is a medium where the publisher controls what articles are seen together by a reader, and so one article can provide the required context to an adjacent article.

There is no such “togetherness” in digital. This strengthens my belief that journalism is still yet to “get” digital.

Pudina family

Sometimes they say that opposites attract.

But more practically, I think it’s impossible to louvvu someone unless you have lots of similar interests, and that also means lots of similar ambitions. And in that sense my wife and I have shared quite a few ambitions.

First we wanted to become celebrity bloggers. Then (ok the order gets messed up here) there was the MBA. And before all this there is Ganeshana Maduve (which we re-watched perhaps for the 50th time this weekend).

And adding to all this, there’s the desire to write in newspapers. I remember that over a decade ago I wanted to regularly write in newspapers, and about “policy issues”. I didn’t follow up on that ambition, of course, but through lots of twists and turns and happy coincidences meant that I started writing for Mint in 2013, and some of the stuff I’ve written there are about “policy issues”.

And the wife has had similar ambitions as well, though her methods have been vastly different, and much more focussed. She’s always wanted to write a column on relationships. Rather, she first wanted to be a relationship blogger, and then a relationship columnist, and she’s gone about the process with single-minded ambition.

So, first there was the MarriageBrokerAuntie blog (now hosted here). Then it turned into a Facebook page. It even led to a business that she ran during her maternity and post-childbirth periods (imagine running a business while nursing a tiny baby). And now she’s in the papers. Yay!

It so happens that it’s the same paper that I write for. And it also happens that the edition of the paper where it was published (Mint on Sunday) also happened to carry an excerpt from my book two-three weeks back. And that also happened to be about relationships.

So a long long time ago, a couple of days after we’d first met, she had written about “Arranged Louvvu“. I don’t think it’s a coincidence that the first piece she’s written for Mint is about “Love, and other arrangements“.  It’s about dating apps, and how what they lead to is not “real love” and it’s no different from “other arrangements”. That people think arranged marriage is uncool, but dating apps lead to basically arranged relationships. And so on.

Read the whole thing, it’s damn well written. Oh, and it features 1-6-1 calls, Panchatantra and George Akerlof’s “market for lemons”, among other stud fundaes.

Now the only thing left is for Berry to start writing for Mint. They don’t have a children’s issue (where they feature drawings, poems, etc. written by kids) so I guess she’ll have to wait a while. But I’m damn hopeful!

In any case, for now massive pride is happening on account of the wife!

When I missed my moment in the sun

Going through an old piece I’d written for Mint, while conducting research for something I’m planning to write, I realise that I’d come rather close to staking claim as a great election forecaster. As it happened, I just didn’t have the balls to stick my neck out (yes, mixed metaphors and all that) and so I missed the chance to be a hero.

I was writing a piece on election forecasting, and the art of converting vote shares into seat shares, which is tricky business in a first past the post system such as India. I was trying to explain how the number of “corners of contests” can have an impact on what seat share a particular vote share can translate to, and I wrote about Uttar Pradesh.

Quoting from my article:

An opinion poll conducted by CNN-IBN and CSDS whose results were published last week predicted that in Uttar Pradesh, the Bharatiya Janata Party is likely to get 38% of the vote. The survey reported that this will translate to about 41-49 seats for the BJP. What does our model above say?

If you look at the graph for the four-cornered contest closely (figure 4), you will notice that 38% vote share literally falls off the chart. Only once before has a party secured over 30% of the vote in a four-cornered contest (Congress in relatively tiny Haryana in 2004, with 42%) and on that occasion went on to get 90% of the seats (nine out of 10).

Given that this number (38%) falls outside the range we have noticed historically for a four-cornered contest, it makes it unpredictable. What we can say, however, is that if a party can manage to get 38% of the votes in a four-cornered state such as Uttar Pradesh, it will go on to win a lot of seats.

As it turned out, the BJP did win nearly 90% of all seats in the state (71 out of 80 to be precise), stumping most election forecasters. As you can see, I had it all right there, except that I didn’t put it in that many words – I chickened out by saying “a lot of seats”. And so I’m still known as “the guy who writes on election data for Mint” rather than “that great election forecaster”.

Then again, you don’t want to be too visible with the predictions you make, and India’s second largest business newspaper is definitely not an “obscure place”. As I’d written a long time back regarding financial forecasts,

…take your outrageous prediction and outrageous reasons and publish a paper. It should ideally be in a mid-table journal – the top journals will never accept anything this outrageous, and you won’t want too much footage for it also.

In all probability your prediction won’t come true. Remember – it was outrageous. No harm with that. Just burn that journal in your safe (I mean take it out of the safe before you burn it). There is a small chance of your prediction coming true. In all likelihood it wont, but just in case it does, pull that journal out of that safe and call in your journalist friends. You will be the toast of the international press.

So maybe choosing to not take the risk with my forecast was a rational decision after all. Just that it doesn’t appear so in hindsight.

Genesis, the Nile and the Nifty

So I’ve written on the financial markets for Mint. This is not part of my usual mandate – which is to write data/quant stories related to politics and the economy and suchlike, I believe this is very different from the kind of markets pieces Mint normally writes.

For starters, I see that there is very little “quant” stuff in mainstream financial reporting. You have tonnes of writing on “fundamentals” (“Company A got this new deal so expect their stock price to increase” or “Company B has regulatory trouble, so short their stock” level) and tonnes on “technicals” (“the nifty will find resistance at 7891” and “we are seeing a head-and-shoulders market (sponsored by P&G)” type), but none on quant.

In fact, if you were to learn finance by reading the newspapers (admittedly a stupid thing to do), you wouldn’t know of how bankers work, of the models they use, of random walks, and of the Black-Scholes-Merton equation. You would think of finance as a rather boring accounting-related fundamentals or voodoo technical stuff. All the cool quant stuff will be lost to you.

So in my attempt to remedy that, I’ve gone all the way and introduced readers of Mint to Fractional Brownian Motion. No, really. I tell a story from Genesis (I actually looked up and read this chapter from the Old Testament so that I could quote it properly), and then relate it to the flooding of the Nile, and that to the work of hydrologist Harold Edwin Hurst, and how that can help us understand the markets. An extract:

Hurst was to remain in Egypt and be associated with the Nile for 62 years (the Egyptian government retained his services after the country’s independence). Looking through 847 years of Nile overflow data (existence of this data tells us much about the farsightedness of the Mameluke and Ottoman rulers of Egypt), Hurst managed to crack the puzzle. The model he built was one of long-range dependence. It is a model that has far-reaching effects, most importantly in the financial markets.
And another:
Turning this around, analysing the rescaled range of a time series as a function of number of time periods will tell us about its long-range dependence. All we need to do is to find the exponent of N, according to which the rescaled range grows. If this exponent is half (in which case the rescaled range grows with the square root of N), we have a regular random walk (or Brownian motion). If the exponent is greater than half, we have positive long-range time dependence, and the value of the exponent tells us the degree of such dependence. Similarly, if the exponent is less than half, we have negative long-range time dependence, which is known as fraction Brownian motion.
Go read the whole thing! And while you’re at it, pick up Benoit Mandelbrot’s The (Mis)Behaviour of Markets and read that, too. It’s a fantastic book on financial markets, and while it has been written by a mathematician, contains no math. And it’s absolutely fascinating stuff. Oh, I’ve read that book twice. And refer to it quite often.

 

Op-Ed in Mint on Environmentalism, Baptists and Bootleggers

After a very long time (~7 months) I’ve written an Op-Ed in Mint. It got published in the physical paper this morning. I’ve used the “Baptists and Bootleggers” framework propounded by economist Bruce Yandle in 1983 to analyse the hijacking of the green cause in India. An excerpt:

In the context of Indian environmental regulation, bootleggers refers to the vast coalition that seeks to profit from curbing industrial growth and development. This includes but is not limited to industries seeking to stifle competition (by preventing competitors’ plants from being built), political parties that rely on people’s poverty and backwardness in order to come to power, and local politicians with vested interests.

The baptists are environmentalists, conservationists and people who are truly interested in the green cause and ensuring sustainable development. Their motivations are straightforward, in that they do not want any developments that could cause lasting damage to natural resources, and they believe that strong environmental regulations are necessary to guard natural resources and ensure sustainable development.

While I was writing the piece I found that Yandle himself has written about the application of the framework to climate change, Kyoto Protocol, etc. This paper (possibly paywall, I only read the abstract) and this one (I’ve read it, and it’s good) are some suggested readings if you want to know more of the concept.

Election Metrics goes international

For those of you who are not particularly aware of it, for the last year and a half I’ve been writing this column called Election Metrics for Mint. It’s basically a quantitative take on elections, and in my estimate I should’ve written over 50 pieces for them so far.

The last two pieces, however, have been different in the sense that I have now moved beyond covering Indian elections to look at elections abroad. In my last but one post, published last month,  i took a look at potential cheating in Afghan elections. (Now I remember linking to that piece from here).

Now, in the latest piece that was published today I look at the forthcoming Scottish referendum, and a recent poll by YouGov in which 47% of respondents said they wanted to vote in favour of independence. I use some binomial jugglery that shows that this translates to a 2.5% chance of a Yes vote, which while insignificant, is an order of magnitude higher than the 0.0004% chance of “Yes” that can be implied from an earlier poll.

I then use the “possible, plausible and probable” framework made famous by Bill Gurley and Aswath Damodaran in their “exchange” in July to show why this poll is significant (it shows that a “Yes” vote is “plausible”, while earlier it was possible but definitely not plausible).

The Afghan elections were rigged

That’s what I conclude in a piece I’ve written for Mint which got published today.

I analyze the last digit of the vote tally of different contestants in different provinces, and find an unusually large number of numbers that end in zero – the odds of this happening at random are at most 2.25%, I conclude.

You might be aware that I’ve been doing this series on elections for Mint for over a year now. Since the Indian elections are over, and steam is yet to pick up for state elections in Maharashtra, Haryana and Jharkhand I’m dabbling a bit in analysis of international elections. If there’s something potentially interesting that you want me to analyze, do drop a note here.

Me, all over the interwebs this week

Firstly, on Tuesday, I got interviewed by this magazine called Information Week. Rather, I had gotten interviewed by them a long time back but the interview appeared on Tuesday. I spoke about the challenges of election forecasting in India and the quality of surveys.

Again on Tuesday, and again on Wednesday, I wrote a pair of articles for Mint analyzing constituencies and parties. On Tuesday, I analyzed constituencies whose representatives have always belonged to ruling parties in the last 4 elections. There are 34 such constituencies. Then on Wednesday I wrote about the influence of states in the Lok Sabha, analyzing the proportion of MPs from each major state that was part of the ruling coalition.

If I had forgotten to mention earlier, I have a deal with Mint that lasts till next October where each month I’m supposed to write 3 articles on election data. You can find all my articles so far here.

Then, today, Pragati published my review of the book Why Nations Fail by Acemoglu and Robinson. More than a book on economics or institutions, it is an awesome history book. Get it.

And in the midst of all this, right here, I wrote a “worky” post about the pros and cons of having a dedicated analytics team.

And if you didn’t notice, this website now has “new clothes”. It was a rather long-pending change and the most important feature of the new layout is that it is “responsive”, and thus looks much better on smartphones. I’ve heard a couple of issues with it already, and do let me know if you have any more issues. And for the first time last night I opened this blog on an iPad and I find that it looks fantabulous, thanks to the OnSwipe plugin I use.

Time for bragging

So the Karnataka polls are done and dusted. The Congress will form the next government here and hopefully they won’t mess up. This post, however, is not about that. This is to stake claim on some personal bragging rights.

1. Back in March, after the results of the Urban Local Body polls came out, I had predicted a victory for the Congress in the assembly elections.

2. Then, a couple of weeks back, I used the logic that people like to vote for the winner, and this winner-chasing will result in a self-fulfilling prophecy that will lead to a comfortable Congress victory.

These two predictions were on the “Resident Quant” blog that I run for the Takshashila Institution. It was a classic prediction strategy – put out your predictions in a slightly obscure place, so that you can quickly bury it in case it doesn’t turn out to be right, but showcase it in case you are indeed correct! After that, however, things went slightly wrong (or right?). Looking at my election coverage Mint asked me to start writing for them.

As it happened I didn’t venture to make further predictions till the elections, apart from building a DIY model where people could input swings in favour of or against parties, and get a seat projection. Watching the exit polls on Sunday, however, compelled me to plug in the exit poll numbers into my DIY model, and come up with my own prediction. I quickly wrote up a short piece.

3. As it happened, Mint decided to publish my predictions on its front page, and now I had nowhere to hide. I had taken a more extreme position compared to most other pollsters. While they had taken care to include some numbers that didn’t mean an absolute majority in the range the predicted for the Congress (so as to shield themselves in that eventuality), I found my model compelling enough to predict an outright victory for the Congress. “A comfortable majority of at least 125 seats”, I wrote.

I had a fairly stressful day today, as the counting took place. Initial times were good, as the early leads went according to my predictions. Even when the BJP had more leads than the Congress, I knew those were in seats that I had anyway tipped them to win, so I felt smug. Things started going bad, however, when the wins of the independents started coming out. The model I had used was unable to take care of them, so I had completely left them out of my analysis. And now I was staring at the possibility that the Congress may not even hit the magic figure of 113 (for an absolute majority), let alone reach my prediction of 125. I prepared myself to eat the humble pie.

Things started turning then, however. It turned out that counting had begun late in the hyderabad karnataka seats – a region that the Congress virtually swept. As I left my seat to get myself some lunch, the Congress number tipped past 113. And soon it was at 119. And then five minutes again back at 113. And so it continued to see-saw for a while, as I sat at the edge of my office chair which I had transplanted to in front of my television.

And then it ticked up again, and stayed at 119 for a while. And soon it was ticking past 120. All results have now been declared, with the Congress clocking up 121 seats. It falls short of the majority I had predicted, but it is a comfortable majority nevertheless. I know I got the BJP number horribly wrong, but so did most other pollsters, for nobody expected them to get only 20% of the popular vote. I also admit to have missed the surge in Independents and “Others”.

Nevertheless, I think I’ve consistently got the results of the elections broadly right, and so I can stake claim to some bragging rights. Do you think I’m being unreasonable?

Corruption and Communism

In an article arguing why Kolkata is best placed to be India’s “best city” in another 20 years, Aakar Patel (I’ve started looking forward to his columns in Mint Lounge) mentions that there isn’t much corruption in the governments in Bengal (at both the center and city level). I don’t know the reasoning for this, but I wonder if this is primarily responsible for the long run that the Communists had at the helm in that  state.

I had argued in a not-so-recent piece in Pragati that big governments tend to be bad governments . I had argued that big government means more ways in which government employees can seek rents, and hence one way of reducing corruption is by reducing the size of the government. Now, assume that for some magical reason, a certain section of the population is sincere and incorruptible. In that case, big government need not be bad government. In other words, people don’t really resent the presence of government everywhere since they don’t see any rent seeking by the government officers. And since they are not unhappy with the size of the government, they don’t mind voting in every time the communists, who will keep the big government!

So I wonder if it is the incorruptibility of the Bengali (for whatever reason; I’m drawing this inference from Patel’s article) that has led to the long communist rule there. Incidentally, the one time the government was seen to be corrupt (in discretionary land allotments in favour of the Tatas, Salim Group, etc.) it wasn’t voted back to power!