Big Data and Fast Frugal Trees

In his excellent podcast episode with EconTalk’s Russ Roberts, psychologist Gerd Gigerenzer introduces the concept of “fast and frugal trees“. When someone needs to make decisions quickly, Gigerenzer says, they don’t take into account a large number of factors, but instead rely on a small set of thumb rules.

The podcast itself is based on Gigerenzer’s 2009 book Gut Feelings. Based on how awesome the podcast was, I read the book, but found that it didn’t offer too much more than what the podcast itself had to offer.

Coming back to fast and frugal trees..

In recent times, ever since “big data” became a “thing” in the early 2010s, it is popular for companies to tout the complexity of their decision algorithms, and machine learning systems. An easy way for companies to display this complexity is to talk about the number of variables they take into account while making a decision.

For example, you can have “fin-tech” lenders who claim to use “thousands of data points” on their prospective customers’ histories to determine whether to give out a loan. A similar number of data points is used to evaluate resumes and determine if a candidate should be called for an interview.

With cheap data storage and compute power, it has become rather fashionable to “use all the data available” and build complex machine learning models (which aren’t that complex to build) for decisions that were earlier made by humans. The problem with this is that this can sometimes result in over-fitting (system learning something that it shouldn’t be learning) which can lead to disastrous predictive power.

In his podcast, Gigerenzer talks about fast and frugal trees, and says that humans in general don’t use too many data points to make their decisions. Instead, for each decision, they build a quick “fast and frugal tree” and make their decision based on their gut feelings about a small number of data points. What data points to use is determined primarily based on their experience (not cow-like experience), and can vary by person and situation.

The advantage of fast and frugal trees is that the model is simple, and so has little scope for overfitting. Moreover, as the name describes, the decision process is rather “fast”, and you don’t have to collect all possible data points before you make a decision. The problem with productionising the fast and frugal tree, however, is that each user’s decision making process is different, and about how we can learn that decision making process to make the most optimal decisions at a personalised level.

How you can learn someone’s decision-making process (when you’ve assumed it’s a fast and frugal tree) is not trivial, but if you can figure it out, then you can build significantly superior recommender systems.

If you’re Netflix, for example, you might figure that someone makes their movie choices based only on age of movie and its IMDB score. So their screen is customised to show just these two parameters. Someone else might be making their decisions based on who the lead actors are, and they need to be shown that information along with the recommendations.

Another book I read recently was Todd Rose’s The End of Average. The book makes the powerful point that nobody really is average, especially when you’re looking a large number of dimensions, so designing for average means you’re designing for nobody.

I imagine that is one reason why a lot of recommender systems (Netflix or Amazon or Tinder) fail is that they model for the average, building one massive machine learning system, rather than learning each person’s fast and frugal tree.

The latter isn’t easy, but if it can be done, it can result in a significantly superior user experience!


Over the last 4-5 days I kinda immersed myself in finishing Rory Sutherland’s excellent book Alchemy.

It all started with a podcast, with Sutherland being the guest on Russ Roberts’ EconTalk last week. I’d barely listened to half the podcast when I knew that I wanted more of Sutherland, and so immediately bought the book on Kindle. The same evening, I finished my previous book and started reading this.

Sometimes I get a bit concerned that I’m agreeing with an author too much. What made this book “interesting” is that Sutherland is an ad-man and a marketer, and keeps talking down on data and economics, and plays up intuition and “feeling”. In other words, at least as far as professional career and leanings go, he is possibly as far from me as it gets. Yet, I found myself silently nodding in agreement as I went through the book.

If I have to summarise the book in one line I would say, “most decisions are made intuitively or based on feeling. Data and logic are mainly used to rationalise decisions rather than making them”.

And if you think about it, it’s mostly true. For example, you don’t use physics to calculate how much to press down on your car accelerator while driving – you do it essentially by trial and error and using your intuition to gauge the feedback. Similarly, a ball player doesn’t need to know any kinematics or projectile motion to know how to throw or hit or catch a ball.

The other thing that Sutherland repeatedly alludes to is that we tend to try and optimise things that are easy to measure or optimise. Financials are a good example of that. This decade, with the “big data revolution” being followed by the rise of “data science”, the amount of data available to make decisions has been endless, meaning that more and more decisions are being made using data.

The trouble, of course, is availability bias, or what I call as the “keys-under-lamppost bias”. We tend to optimise and make decisions on things that are easily measurable (this set of course is now much larger than it was a decade ago), and now that we know we are making use of more objective stuff, we have irrational confidence in our decisions.

Sutherland talks about barbell strategies, ergodicity, why big data leads to bullshit, why it is important to look for solutions beyond the scope of the immediate domain and the Dunning-Kruger effect. He makes statements such as “I would rather run a business with no mathematicians than with second-rate mathematicians“, which exactly mirrors my opinion of the “data science industry”.

There is absolutely no doubt why I liked the book.

Thinking again, while I said that professionally Sutherland seems as far from me as possible, it’s possibly not so true. While I do use a fair bit of data and economic analysis as part of my consulting work, I find that I make most of my decisions finally on intuition. Data is there to guide me, but the decision-making is always an intuitive process.

In late 2017, when I briefly worked in an ill-fated job in “data science”, I’d made a document about the benefits of combining data analysis with human insight. And if I think about my work, my least favourite work is where I’ve done work with data to help clients make “logical decision” (as Sutherland puts it).

The work I’ve enjoyed the most has been where I’ve used the data and presented it in ways in which my clients and I have noticed patterns, rationalised them and then taken a (intuitive) leap of faith into what the right course of action may be.

And this also means that over time I’ve been moving away from work that involves building models (the output is too “precise” to interest me), and take on more “strategic” stuff where there is a fair amount of intuition riding on top of the data.

Back to the book, I’m so impressed with it that in case I was still living in London, I would have pestered Sutherland to meet me, and then tried to convince him to let me work for him. Even if at the top level it seems like his work and mine are diametrically opposite..

I leave you with my highlights and notes from the book, and this tweet.

Here’s my book, in case you are interested.


Serials and movies

Yesterday I finished reading Gita Krishnankutty’s English translation of MT Vasudevan Nair’s Randamoozham. It’s the story of the Mahabharata told from Bhima’s perspective.

This wasn’t the first time that I was reading a translation of this magnificent book. A few years ago, journalist Prem Panicker had created a series on his blog where he would put up translations of bits of this book daily. I remember quite liking that, and a lot of people raving about it.

Prem’s version of the book was far longer than the version that I finished yesterday (Gita Krishnankutty’s version is 380 pages long, which comes to around 70000 words or less. Prem’s is 120,000 words long). It was also far more passionate. Rather than directly translating the novel, Prem took liberties in adding his own inputs.

It’s been over a decade since I read Prem’s version, but from what I remember, the parts of the story where Bhima mourns Ghatotkacha’s death, for example, are far more well sketched out in that version. It is similar with the parts which show Bhima’s frustration with Yudhishthira’s leadership.

Thinking about it, though, one reason why Prem was able to go into such detail was that he presented his book in a serialised format. Every day he would put out the translation of a few pages’ worth of a book, and the translation would come out to be the length of a long form article (the kind of articles that Prem became a specialist in writing during his time at Rediff).

When you’re reading it in book form, in which you read the whole thing together, reading in such detail may not work so well since that might make the book unnecessarily thick, and people might put NED midway. Give the inputs in small doses, however, and people will be happy to consume the greater detail. In that sense, Prem’s and Gita Krishnankutty’s translations are both excellent, and both very well suited for the formats they came out in.

It is a similar story with movies and serials. Movies have a 2-2.5 hour length because that’s how much typically people can consume at a time without putting NED. Serials, on the other hand, because they are consumed bit by bit at a time, can go much longer in aggregate (sometimes unnecessarily long).

Netflix releasing all episodes of a series at the same time, however, is changing this dynamic. Sacred Games apart, I’ve been unable to get through any Netflix fiction series because of their sheer length. Because binge-watching has become a thing (thanks to Netflix putting out an entire season at once), the entire season comes to resemble a movie. So a season with 8 one-hour episodes effectively becomes a 8-hour movie. And unless it’s extremely well made, or has sufficient stuff going on through the 8 hours, it becomes incredibly hard to sit through!


Dreamers and Dignity

If I’d picked up Snigdha Poonam’s Dreamers before I had read Chris Arnade’s Dignity, I might have liked it better. As it happened, having read Dignity, I found Dreamers to be unnecessarily judgmental and prescriptive, and was unable to read it beyond the first two chapters. It is now there on my goodreads page, as a book that I “finished” and gave one star.

Dignity is a book I highly recommend. Chris Arnade, a former investment banker with a PhD in astrophysics wanders around and hangs around in what he calls as “back row America”, and chronicles people’s lives there. The entire book is simply a set of chronicles, garnished with beautiful photos he has taken of his interviewees.

While Arnade makes no secrets of his own political leaning, he doesn’t let that affect his book. Rather, he keeps his own politics to the minimum and lets his interviews do the talking, literally. There are no policy prescriptions in the book, and the reader is simply presented a set of lives and asked to draw her own conclusions. And that means that even if you don’t agree with the politics of the author (I certainly don’t), the book is an incredibly compelling read.

I picked up Snigdha Poonam’s Dreamers about a month or so after I’d finished Dignity. The premise is sort of similar – except that given that India has recently had far higher growth than the US, the “back row Indians” can be classified as “dreamers” who are seeking a better life. And in this book, Poonam chronicles the stories of some of these dreamers, and what they are doing to get themselves a better life.

Poonam is clear about her politics as well (“my family has always voted for the Congress Party”), but what makes her book different from Arnade’s is that she lets her politics take over her narrative. While telling the story of Moin Khan, who runs a spoken English class in Ranchi, she doesn’t hesitate to make snide remarks about either the teacher or any of his students.

Rather than letting her characters talk, Poonam talks on their behalf and overlays her politics to pretty much everything she is talking about. “This is how you are expected to get ahead in Modi’s India” is a refrain through the book.

And even leaving the politics aside, what made me uncomfortable with Dreamers is that the author seems to talk down to the interviewees. The tone throughout the parts of the book that I read is one of moral superiority and smugness of being part of “front row India”.

Maybe if I had read Dreamers before I read Dignity, I would have appreciated it for what it is, and for the stories that it told. I might have discarded the politics and the tone and just enjoyed the stories (I see the book has got 4 stars on Goodreads from a lot of my friends).

Having read Dignity, however, I perhaps had this image in my head of how these stories can be told well. And that meant that I was simply unable to look beyond the overt politics and smug tone in Dreamers. And that meant I abandoned it midway, and gave it a low rating.

Margaret Atwood doesn’t escape my fate

My book released exactly two years ago (if you haven’t read it yet, you can buy it here). Rather, it was supposed to release two years ago, on 6th of September 2017. As it happened, people who had pre-ordered the book got deliveries a few days early. Amazon had messed up with the release date.

I remember getting in touch with Amazon Customer Care. They didn’t seem to care. I spoke to friends and relatives who worked there, and they suggested a “Jeff B escalation” (an email sent to Jeff Bezos – apparently he reads them). There was no response to that either. And so my book came out in a trickle, being sent to people as they ordered them, rather than with a bang.

I’m possibly feeling a sense of schadenfreude that it’s not just first-time authors like me who got screwed over like this by Amazon in terms of early release of the book. I am in illustrious company – Canadian author Margaret Atwood suffered the same fate this week.

Amazon, the biggest book vendor in the United States, recently started shipping preorders of Margaret Atwood’s book Testaments. The problem, notably, is that Atwood’s book is not supposed to launch until Tuesday, September 10. Amazon is violating the embargo that all sellers of the book have agreed to. And its indie bookselling rivals are pissed.

In my case, Amazon had exclusive sales on the book – thanks to using a small first-time publisher, we didn’t have the network to go wider and get the book into more stores. In that sense, apart from me, there was possibly nobody pissed off at the early release of the book.

Then again, this early release of pre-ordered books was an endemic problem to Amazon, and a high-profile leak such as this one was bound to happen some time or the other. Hopefully this will lead to the retailer to put enough measures in place to prevent this kind of thing from happening again (mainstream publishers have strong relationships with bookshops, so they are likely to put pressure on Amazon).

In any case, I’m glad to have such good company!

PS: If you haven’t listened to Atwood’s conversation with Tyler Cowen, you should do so soon. It’s fantastic (and I say this as someone who hasn’t read any of her works)

Gamification and finite and infinite games

Ok here I’m integrating a few concepts that I learnt via Venkatesh Guru Rao. The first is that of Finite and Infinite games, a classic if hard to read book written by philosopher James Carse (which I initially discovered thanks to his Breaking Smart Season 1 compilation). The second is of “playflow”, which again I discovered through a recent edition of his newsletter.

A lot of companies try to “gamify” the experiences for their employees in order to make work more fun, and to possibly make them more efficient.

For example, sales organisations offer complicated incentives (one of my historically favourite work assignments has been to help a large client optimise these incentives). These incentives are offered at multiple “slabs”, and used to drive multiple objectives (customer acquisition, retention, cross-sell, etc.). And by offering employees incentives for achieving some combination of these objectives, the experience is being “gamified”. It’s like the employee is gaining points by achieving each of these objectives, and the points together lead to some “reward”.

This is just one example. There are several other ways in which organisations try to gamify the experience for their employees. All of them involve some sort of award of “points” for things that people do, and then a combination of points leading to some “reward”.

The problem with gamification is that the games organisations design are usually finite games. “Sell 10 more widgets in the next month”. “Limit your emails to a maximum of 200 words in the next fifteen days”. “Visit at least one client each day”. And so on.

Running an organisation, however, is an infinite game. At the basic level, the objective of an organisation is to remain a going concern, and keep on running. Growth and dividends and shareholder returns are secondary to that – if the organisation is not a going concern, none of that matters.

And there is the contradiction – the organisation is fundamentally playing an infinite game. The employees, thanks to the gamified experience, are playing finite games. And they aren’t always compatible.

Of course, there are situations where finite games can be designed in a way that their objectives align with the objectives of the overarching infinite game. This, however, is not always possible. Hence, gamification is not always a good strategy for organisations.

Organisations have figured out the solution to this, of course. There is a simple way to make employees play the same infinite game as the organisation – by offering employees equity in the company. Except that employees have the option of converting that to a finite game by selling the said equity.

Whoever said incentive alignment is an easy task..


Marginalised communities and success

Yesterday I was listening to this podcast where Tyler Cowen interviews Neal Stephenson, who is perhaps the only Science Fiction author whose books I’ve read. Cowen talks about the characters in Stephenson’s The Baroque Cycle, a masterful 3000-page work which I polished off in a month in 2014.

The key part of the conversation for me is this:

COWEN: Given your focus on the Puritans and the Baroque Cycle, do you think Christianity was a fundamental driver of the Industrial Revolution and the Scientific Revolution, and that’s why it occurred in northwestern Europe? Or not?

STEPHENSON: One of the things that comes up in the books you’re talking about is the existence of a certain kind of out-communities that were weirdly overrepresented among people who created new economic systems, opened up new trade routes, and so on.

I’m talking about Huguenots, who were the Protestants in France who suffered a lot of oppression. I’m talking about the Puritans in England, who were not part of the established church and so also came in for a lot of oppression. Armenians, Jews, Parsis, various other minority communities that, precisely because of their outsider minority status, were forced to form long-range networks and go about things in an unconventional, innovative way.

So when we think about communities such as Jews or Parsis, and think about their outsized contribution to business or culture, it is this point that Stephenson makes that we should keep in mind. Because Jews and Parsis and Armenians were outsiders, they were “forced to form long-range networks”.

In most cases, for most people of these communities, these long-range networks and unconventional way of doing things didn’t pay off, and they ended up being worse off compared to comparable people from the majority communities in wherever they lived.

However, in the few cases where these long-range networks and innovative ways of doing things succeeded, they succeeded spectacularly. And these incidents are cases that we have in mind when we think about the spectacular success or outsized contributions of these communities.

Another way to think of this is – denied “normal life”, people from marginalised communities were forced to take on much more risk in life. The expected value of this risk might have been negative, but this higher risk meant that these communities had a much better “upper tail” than the majority communities that suppressed and oppressed them.

Given that in terms of long-term contributions and impact and public visibility it is only the tails of the distribution that matter (mediocrity doesn’t make news), we think of these communities as having been extraordinary, and wonder if they have “better genes” and so on.

It’s a simple case of risk, and oppression. This, of course, is no justification for oppressing swathes of people and forcing them to take more risks than necessary. People need to decide on their own risk preferences.

Gruffaloes and Finite Games

One story that my daughter knows well, rather too well, is the story of the Gruffalo. This is a story of a mouse told in two parts.

In the first part, the mouse fools a fox, an owl and a snake from eating him by convincing them that he’s having lunch, tea and dinner respectively with a supposedly imaginary creature named “Gruffalo”. And when they each ask him what the Gruffalo is like, he makes up stuff fantastically (terrible teeth in terrible jaws, turned out paws, etc.).

Except that midway through the story there is a kahaani mein twist, and the mouse actually encounters the gruffalo. In the second part of the story, the mouse tells the gruffalo that he is going to have lunch, tea and dinner with the fox, owl and snake, and prevents the gruffalo from eating him. And the mouse lives another day.

It is evidently a nice story, and the rhyme means that the daughter had mugged up the entire story enough when she was barely two years old that she could “read” it when shown the book (she can’t read a word yet). However, I don’t like it because I don’t like the plot.

One of the most influential books I’ve read is James Carse’s Finite and Infinite Games. Finite Games are artificial games where we play to “win”. There is a defined finish, and there is a set of tasks that we need to achieve that constitutes “victory”. Most real-life games are on the other hand are “infinite games” where the objective is to simply ensure that the game simply goes on.

From the point of stories, the best stories are ones which represent finite games, where there is a clear objective, and the story ends in “victory” or “lack of victory” (in the case of a tragedy). The Good, The Bad and the Ugly has the finite aim of finding the treasure buried in the graveyard. Ganeshana Maduve has the finite aim of YG Rao marrying “Shruti”. Gangs of Wasseypur has the finite aim of the Khan family taking revenge on Ramadhir Singh. Odyssey has the finite aim of Odysseus returning home to Penelope. And so forth.

Putting it another way, finite games make for nice stories, since stories are themselves finite, with a beginning and an end. A story that represents an infinite game is necessarily left incomplete, and you don’t know what happens just outside the slice of action that the story covers. So infinite games, which is how life is lived, make for lousy stories.

And the gruffalo story is an infinite game, since the “game” that the mouse is playing in the story is survival – by definition an infinite game. There is no “victory” by being alive at the end of the day the story covers – like there is no she-mouse to marry, or a baby mouse to see for the first time, or a party to go to. It is just another day in the life of the mouse, and the events of the day are unlikely to be that much more spectacular than the days not covered by the story.

That is what makes the gruffalo story so unsatisfying. Yes, the mouse played off the fox, owl and snake against the gruffalo to ensure his survival, but what about the next day? Would he have to invent another creature to ensure his survival? Would the predators buy the same story another time?

I don’t know, and so the story rings hollow. But the rhyme is good, and so my daughter loves the story!

Telling stories with data

I’m about 20% through with The Verdict by Prannoy Roy and Dorab Sopariwala. It’s a fascinating book, except for one annoyance – it is full of tables that serve no purpose but to break the flow of text.

I must mention that I’m reading the book on the Kindle, which means that the tables can pose a major annoyance. Text breaks off midway through one page, and the next couple of pages involve a table or two, with several lines of text explaining what’s in the table. And then the text continues. It makes for a rather disruptive reading experience. And some of the tables have just one data point – making one wonder why it has been inserted there at all.

This is not the first book that I’ve noticed that makes this mistake. Some of the sports analytics books I’ve read in recent times, such as The Numbers Game also make the same error (I read that in print, and still had the same disruption). Bhagwati and Panagariya’s Why Growth Matters is similarly unreadable. Tables abruptly inserted into the middle of text, leading to the reader losing flow in the reading.

Telling a data story in book length is a completely different challenge to telling one in article length. And telling a story with data is a complete art form. When you’re putting a table there, you need to be able to explain why that table is important to the story – rather than putting it there just because it seems more rigorous.

Also the exact placement of the table (something that can’t be controlled well in Kindle, but is easy to fix in either HTML or print) matters –  the table should be relevant to the piece of text immediately preceding and succeeding it, in a way that it doesn’t disrupt the reader’s flow. More importantly, the table should be able to add value at that particular point – perhaps building on something that has been described in the previous paragraph.

Book length makes it harder because people don’t normally expect tables and figures to disturb their reading flow when reading something of book length. Also, the book format means that it is not always possible to insert a table at a precise point (even in print, where pagination is an issue).

So how do you tell a book length story with data? Firstly, be very stingy about the data that you want to show – anything that doesn’t immediately add value should be banished to the appendix. Even the rigour, which academics might be particular about, can be pushed to the end notes (not footnotes, since those can be disruptive to flow as well, turning pages into half pages).

Then, once you know that showing a particular table or graph is inevitable to telling the story, put it either in the beginning or the end of a chapter. This way, it doesn’t break the reader’s flow. Then, refer to individual numbers in the middle of the text without having to put the entire table in there. Unless each and every data point in the table is important, banish it to the endnotes.

One other common mistake (I did it in my piece in Forbes published yesterday) is to put a big table and not talk about it. It only seeks to confuse the reader, who starts looking for explanations for everything in the table in later parts.

I guess authors and analysts tend to get possessive. If you have worked hard to produce insights from data, you seek to share as much of it as possible. And this can mean simply dumping data all the data in the piece without a regard for what the reader will do with it.

I’m making a note to myself to not repeat this mistake in future.

Tigers and Bullwhips

Over three years ago, well before our daughter was born, my wife’s cousin had told us that she likes to watch her daughter’s TV shows because they contained “morals”, which were often useful to her at work. While we never took to the “moral” TV show she mentioned (Daniel Tiger – it is bloody boring), I have begun to notice that there are important management lessons in other popular children’s stories.

So I hereby begin this blog series on what I call the “Kiddie MBA” – basically business lessons from kids’s stories. And we will start with that all-time classic, The Tiger Who Came To Tea, by Judith Kerr. 

The basic premise of this story that remains a classic fifty years after being published is what operations managers call the “bullwhip effect“. Sometimes a business, possibly in trading, can be subject to a sudden demand, which the business will not be able to fulfil given its current inventories.

As a result of this sudden one-time spurt in demand, the business increases its future forecasts of demand, and starts keeping more inventory. This business’s supplier sees this increased demand and increases its own forecasts upward, and increases its own inventory. Thus, this one-time demand “shock” percolates up the supply chain, giving the illusion of higher demand and with each layer in the chain keeping higher and higher inventory.

And then one day the retailer will realise that this demand shock is not replicable and moves forecasts downwards, and this triggers a downward edge in the forecasts up the value chain, and demand at the source comes crashing down.

Being a children’s book, The Tiger Who Came To Tea eschews the complexity of the supply chain and instead keeps the story at one level – at the level of the household of the protagonist Sophie (not to be confused with Sophie the Giraffe).

The premise of the story is the demand shock for supplies in Sophie’s home – a tiger comes home for tea and eats up everything that’s at home, drinks up all that’s there to be drunk (including “all the water in the tap”) and leaves, leaving nothing for Sophie and her family.

Assuming that the tiger will return the next day, Sophie’s family stocks up heavily, including “lots of tiger food”. And the tiger never arrives.

My guess is that the rest of the supply chain is left as an exercise to the reader – how the retailer who sold Sophie the tiger food will react to the suddenly higher demand for food (and for tiger food), how this retailer’s supplier will react, whether the tiger visits some other household for tea the next day (making this demand “regular” at the retailer’s level), and so forth.

Perhaps this is what makes this such as great book, and an all-time classic!