Just Plot It

One of my favourite work stories is from this job I did a long time ago. The task given to me was demand forecasting, and the variable I needed to forecast was so “micro” (this intersection that intersection the other) that forecasting was an absolute nightmare.

A side effect of this has been that I find it impossible to believe that it’s possible to forecast anything at all. Several (reasonably successful) forecasting assignments later, I still dread it when the client tells me that the project in question involves forecasting.

Another side effect is that the utter failure of standard textbook methods in that monster forecasting exercise all those years ago means that I find it impossible to believe that textbook methods work with “real life data”. Textbooks and college assignments are filled with problems that when “twisted” in a particular way easily unravel, like a well-tied tie knot. Industry data and problems are never as clean, and elegance doesn’t always work.

Anyway, coming back to the problem at hand, I had struggled for several months with this monster forecasting problem. Most of this time, I had been using one programming language that everyone else in the company used. The code was simultaneously being applied to lots of different sub-problems, so through the months of struggle I had never bothered to really “look at” the data.

I must have told this story before, when I spoke about why “data scientists” should learn MS Excel. For what I did next was to load the data onto a spreadsheet and start looking at it. And “looking at it” involved graphing it. And the solution, or the lack of it, lay right before my eyes. The data was so damn random that it was a wonder that anything had been forecast at all.

It was also a wonder that the people who had built the larger model (into which my forecasting piece was to plug in) had assumed that this data would be forecast-able at all (I mentioned this to the people who had built the model, and we’ll leave that story for another occasion).

In any case, looking at the data, by putting it in a visualisation, completely changed my perspective on how the problem needed to be tackled. And this has been a learning I haven’t let go of since – the first thing I do when presented with data is to graph it out, and visually inspect it. Any statistics (and any forecasting for sure) comes after that.

Yet, I find that a lot of people simply fail to appreciate the benefits of graphing. That it is not intuitive to do with most programming languages doesn’t help. Incredibly, even Python, a favoured tool of a lot of “data scientists”, doesn’t make graphing easy. Last year when I was forced to use it, I found that it was virtually impossible to create a PDF with lots of graphs – something that I do as a matter of routine when working on R (I subsequently figured out a (rather inelegant) hack the next time I was forced to use Python).

Maybe when you work on data that doesn’t have meaningful variables – such as images, for example – graphing doesn’t help (since a variable on its own has little information). But when the data remotely has some meaning – sales or production or clicks or words, graphing can be of immense help, and can give you massive insight on how to develop your model!

So go ahead, and plot it. And I won’t mind if you fail to thank me later!

Ticking all the boxes

Last month my Kindle gave up. It refused to take charge, only heating up the  charging cable (and possibly destroying an Android charger) in the process. This wasn’t the first time this was happening.

In 2012, my first Kindle had given up a few months after I started using it, with its home button refusing to work. Amazon had sent me a new one then (I’d been amazed at the no-questions-asked customer-centric replacement process). My second Kindle (the replacement) developed problems in 2016, which I made worse by trying to pry it open with a knife. After I had sufficiently damaged it, there was no way I could ask Amazon to do anything about it.

Over the last year, I’ve discovered that I read much faster on my Kindle than in print – possibly because it allows me to read in the dark, it’s easy to hold, I can read without distractions (unlike phone/iPad) and it’s easy on the eye. I possibly take half the time to read on a Kindle what I take to read in print. Moreover, I find the note-taking and highlighting feature invaluable (I never made a habit of taking notes on physical books).

So when the kindle stopped working I started wondering if I might have to go back to print books (there was no way I would invest in a new Kindle). Customer care confirmed that my Kindle was out of warranty, and after putting me on hold for a long time, gave me two options. I could either take a voucher that would give me 15% off on a new Kindle, or the customer care executive could “talk to the software engineers” to see if they could send me a replacement (but there was no guarantee).

Since I had no plans of buying a new Kindle, I decided to take a chance. The customer care executive told me he would get back to me “within 24 hours”. It took barely an hour for him to call me back, and a replacement was in my hands in 2 days.

It got me wondering what “software engineers” had to do with the decision to give me a replacement (refurbished) Kindle. Shortly I realised that Amazon possibly has an algorithm to determine whether to give a replacement Kindle for those that have gone kaput out of warranty. I started  trying to guess what such an algorithm might look like.

The interesting thing is that among all the factors that I could list out based on which Amazon might make a decision to send me a new Kindle, there was not one that would suggest that I shouldn’t be given a replacement. In no particular order:

  • I have been an Amazon Prime customer for three years now
  • I buy a lot of books on the Kindle store. I suspect I’ve purchased books worth more than the cost of the Kindle in the last year.
  • I read heavily on the Kindle
  • I don’t read Kindle books on other apps (phone / iPad / computer)
  • I haven’t bought too many print books from Amazon. Most of the print books I’ve bought have been gifts (I’ve got them wrapped)
  • My Goodreads activity suggests that I don’t read much outside of what I’ve bought from the Kindle store

In hindsight, I guess I made the correct decision of letting the “software engineers” determine whether I qualify for a new Kindle. I guess Amazon figured that had they not sent me a new Kindle, there was a significant amount of low-marginal-cost sales that they were going to lose!

I duly rewarded them with two book purchases on the Kindle store in the course of the following week!

The Cow-Postman Paradigm

I thought I had written about this already sometime ago, but I can’t seem to find it. Anyways.

When we were in class 6, we were taught to write essays in Hindi. “Taught to write” is a loose phrase there, for we were just given two essays and asked to mug them up. One was on “gaay” (cow) and the other was on “daakiya” (postman). The teacher had reliably informed us that while the exam would have a question requiring us to write an essay, the topic would be either “gaay” or “daakiya”. And it was a given that mugging up the given essay and spitting it out would get full credit.

To my credit (I used to be topper types, you remember?), I mugged up both. This girl who was at my van stop didn’t. She only mugged up “gaay” since that seemed to be the more likely to appear in the exam. Another girl, two years our senior, who had been through this routine, consoled her saying that if the question is for “daakiya”, she can still write about “gaay”, and she would get at least 2 out of 5 marks.

As it happened, the exam required us to write a question on daakiya. The girl in my van stop wrote an essay on gaay. And got 0 for it.

This brings me to a fairly common practice, at least in India, of coming up with a set of answers and spouting them out irrespective of what the question is. And I call this the “cow-postman paradigm” based on the above anecdote.

A popular example of the cow-postman paradigm can be seen at beauty contests, where the beauty queens spout inanities in order to show off their “nobility” and show themselves as being “worthy ambassadors” of the Miss Whatever project. The most famous example of this probably is Priyanka Chopra, who answered “Mother Teresa” when asked who is the “living woman she most admires” (Mother Teresa had died two years before Priyanka Chopra became Miss World).

Politicians and PR agents are also masters of the Cow-Postman paradigm. Irrespective of what interviewers ask, they simply spout their prepared lines  in the hope that it will sometimes answer the question. In case the interviewer decides to be snarky, this can be made fun of. In most cases, the “leaders” get away with it.

It is possibly these instances of “getting away with it” (or even “benefitting from it”) that results in propagating the Cow-Postman paradigm. Maybe it is a worthy effort of journalists, possibly at listicle-based publications, to make note of and make a list of such famous cow-postman instances. That’s the only way we can cure it!

 

When Jayalalithaa Ruined My Birthday

As the Babri Masjid was being brought down, I celebrated.

I had come up with this line a few years ago, and said that whenever I write my autobiography, I’m going to begin it this way. And while I’m not as certain nowadays that I’ll write an autobiography, in case I write one I’ll still use this line to open it.

This line could also be used in a logic class, the kind of lectures I delivered fairly frequently between 2012 and 2016, illustrating logical fallacies. For this one might induce the correlation-is-causation fallacy in your head, and you might think that if I celebrated while the Babri Masjid was being brought down, I must be a Muslim-hating bigot. So here is what will be the second line in my autobiography, whenever I write it:

It was my tenth birthday, and there was a party at home.

There is something special about your birthday falling on Sundays. The first time that happened, in 1987, was also the first time that my parents organised a birthday party for me. I’m too young to know how many people came, but there were a lot of people filling our house that evening. We had professional catering and I got so many gifts that I got to using some of them (such as Enid Blyton story books) several years later.

Maybe I read some of the books around the time my birthday fell on a Sunday once again, which happened in 1992. That also happened to be the next time I had a party at home, and this one was different, with less than ten guests, with all of them being my classmates in school.

My mother had done the cooking that day. We played cricket and hide-and-seek, and some other party games (which I don’t remember now). And then later that evening, news on television told us that the Babri Masjid had been brought down that day and riots had started.

 

The only thing that registered in my head then was that there would be no school the next day, and I didn’t know when I would distribute the chocolates I had bought for the customary school distribution.

The long term impact, though, was that my birthday got inextricably linked to the Babri Masjid demolition.

So over the years, when people have searched for an anchor to remember my birthday, they’ve inevitably used news of the anniversary of the Babri Masjid demolition. This morning, for example, I got a message that said “Happy birthday. Babri Masjid article came up somewhere 🙂 “. Another friend messaged me to remind me of what I’ve written to being this post.

A couple of years back, a friend messaged me later in December apologising for missing my birthday, adding that he had missed it because there wasn’t much news about the Babri Masjid anniversary. This must have been in 2016, which was among my worst birthdays because beyond close family, hardly anyone wished me that day.

And I blame former Tamil Nadu chief minister Jayalalithaa for that, for after a rather prolonged illness, she had passed away the previous night. And that meant that the news waves in India on the 6th of December 2016 were filled with news of Jayalalithaa’s demise, with any Babri Masjid anniversary stuff being pushed to the backburner.

The situation got rectified last year with it being the 25th anniversary of the Babri Masjid demolition, so the number of people who wished me went back to “normal levels”. And perhaps with elections being round the corner again, and without an important death to distract the news, I’m guessing that Babri Masjid has made enough news today for enough people to remember my birthday!

I must also take this opportunity to thank certain entities who unfailingly wish me on every birthday.

Oh, and I discovered this morning that today is 6/12/18. And my wife helpfully added that I turned 36 today.

Now I feel really old!

What Ails Liverpool

So Liverpool FC has had a mixed season so far. They’re second in the Premier League with 36 points from 14 games (only points dropped being draws against ManCity, Chelsea and Arsenal), but are on the verge of going out of the Champions League, having lost all three away games.

Yesterday’s win over Everton was damn lucky, down to a 96th minute freak goal scored by Divock Origi (I’d forgotten he’s still at the club). Last weekend’s 3-0 against Watford wasn’t as comfortable as the scoreline suggested, the scoreline having been opened only midway through the second half. The 2-0 against Fulham before that was similarly a close-fought game.

Of concern to most Liverpool fans has been the form of the starting front three – Mo Salah, Roberto Firmino and Sadio Mane. The trio has missed a host of chances this season, and the team has looked incredibly ineffective in the away losses in the Champions League (the only shot on target in the 2-1 loss against PSG being the penalty that was scored by Milner).

There are positives, of course. The defence has been tightened considerably compared to last season. Liverpool aren’t leaking goals the way they did last season. There have been quite a few clean sheets so far this season. So far there has been no repeat of last season’s situation where they went 4-1 up against ManCity, only to quickly let in two goals and then set up a tense finish.

So my theory is this – each of the front three of Liverpool has an incredibly low strike rate. I don’t know if the xG stat captures this, but the number of chances required by each of Mane, Salah and Firmino before they can convert is rather low. If the average striker converts one in two chances, all of these guys convert one in four (these numbers are pulled out of thin air. I haven’t looked at the statistics).

And even during the “glory days” of last season when Liverpool was scoring like crazy, this low strike rate remained. Instead, what helped then was a massive increase in the number of chances created. The one game I watched live (against Spurs at Wembley), what struck me was the number of chances Salah kept missing. But as the chances kept getting created, he ultimately scored one (Liverpool lost 4-1).

What I suspect is that as Klopp decided to tighten things up at the back this season, the number of chances being created has dropped. And with the low strike rate of each of the front three, this lower number of chances translates into much lower number of goals being scored. If we want last season’s scoring rate, we might also have to accept last season’s concession rate (though this season’s goalie is much much better).

There ain’t no such thing as a free lunch.

Showing off

So like good Indian parents we’ve started showing off the daughter in front of guests. And today she showed us that she’s equal to the task.

A couple of weeks back, after seeing the photo of a physicist friend’s son with the book Quantum Physics for babies, I decided to get a copy. Like with all new things the daughter gets, she “read” the book dutifully for the rest of the day it arrived. She learnt to recognised the balls in the book, but wasn’t patient enough for me to teach her about atoms.

The next day the book got put away into her shelf, never to appear again, until today that is. Some friends were visiting and we were all having lunch. As I was feeding the daughter she suddenly decided to run off towards her bookshelf, and with great difficulty pulled out a book – this one. As you might expect, our guests were mighty impressed.

Then they started looking at her bookshelf and were surprised to find a “children’s illustrated atlas” there. We told them that the daughter can identify countries as well. Soon enough, she had pulled out the atlas from the shelf (she calls it the “Australia book”) and started pointing out continents an d countries in that.

To me the high point was the fact that she was looking at the maps upside down (or northside-down – the book was on the table facing the guests), and still identified all the countries and continents she knows correctly. And once again, I must point out that she hadn’t seen the atlas for at least two or three weeks now.

Promise is showing, but we need to be careful and make sure we don’t turn her into a performing monkey.

PS: Those of you who follow me on Instagram can look at this video of Berry identifying countries.

PS2: Berry can identify continents on a world map, but got damn disoriented the other day when I was showing her a map that didn’t contain Antarctica.

Magnus Carlsen’s Endowment

Game 12 of the ongoing Chess World Championship match between Magnus Carlsen and Fabiano Caruana ended in an unexpected draw after only 31 moves, when Carlsen, in a clearly better position and clearly ahead on time, made an unexpected draw offer.

The match will now go into a series of tie-breaks, played with ever-shortening time controls, as the world looks for a winner. Given the players’ historical record, Carlsen is the favourite for the rapid playoffs. And he knows it, since starting in game 11, he seemed to play towards taking it into the playoffs.

Yesterday’s Game 12 was a strange one. It started off with a sharp Sicilian Pelikan like games 8 and 10, and then between moves 15 and 20, players repeated the position twice. Now, the rules of chess state that if the same position appears three times on the board, the game is declared a draw. And there was this move where Caruana had the chance to repeat a position for the third time, thus drawing the game.

He spent nearly half an hour on the move, and at the end of it, he decided to deviate. In other words, no quick draw. My suspicion is that this unnerved Carlsen, who decided to then take a draw at the earliest available opportunity available to him (the rules of the match state that a draw cannot be agreed before move 30. Carlsen made his offer on move 31).

In behavioural economics, Endowment Effect refers to the bias where you place a higher value on something you own than on something you don’t own. This has several implications, all of which can lead to potentially irrational behaviour. The best example is “throwing good money after bad” – if you have made an investment that has lost money, rather than cutting your losses, you double down on the investment in the hope that you’ll recoup your losses.

Another implication is that even when it is rational to sell something you own, you hold on because of the irrationally high value you place on it. The endowment effect also has an impact in pricing and negotiations – you don’t mind that “convenience charge” that the travel aggregator adds on just before you enter your credit card details, for you have already mentally “bought” the ticket, and this convenience charge is only a minor inconvenience. Once you are convinced that you need to do a business deal, you don’t mind if the price moves away from you in small marginal steps – once you’ve made the decision that you have to do the deal, these moves away are only minor, and well within the higher value you’ve placed on the deal.

So where does this fit in to Carlsen’s draw offer yesterday? It was clear from the outset that Carlsen was playing for a draw. When the position was repeated twice, it raised Carlsen’s hope that the game would be a draw, and he assumed that he was getting the draw he wanted. When Caruana refused to repeat position, and did so after a really long think, Carlsen suddenly realised that he wasn’t getting the draw he thought he was getting.

It was as if the draw was Carlsen’s and it had now been taken away from him, so now he needed to somehow get it. Carlsen played well after that, and Caruana played badly, and the engines clearly showed that Carlsen had an advantage when the game crossed move 30.

However, having “accepted” a draw earlier in the game (by repeating moves twice), Carlsen wanted to lock in the draw, rather than play on in an inferior mental state and risk a loss (which would also result in the loss of the Championship). And hence, despite the significantly superior position, he made the draw offer, which Caruana was only happy to accept (given his worse situation).