Profit and politics

Earlier today I came across this article about data scientists on LinkedIn that I agreed with so much that I started wondering if it was simply a case of confirmation bias.

A few sentences (possibly taken out of context) from there that I agree with:

  • Many large companies have fallen into the trap that you need a PhD to do data science, you don’t.
  • There are some smart people who know a lot about a very narrow field, but data science is a very broad discipline. When these PhD’s are put in charge, they quickly find they are out of their league.
  • Often companies put a strong technical person in charge when they really need a strong business person in charge.
  •  I always found the academic world more political than the corporate world and when your drive is profits and customer satisfaction, that academic mindset is more of a liability than an asset.

Back to the topic, which is the last of these sentences. This is something I’ve intended to write for 5-6 years now, since the time I started off as an independent management consultant.

During the early days I took on assignments from both for-profit and not-for-profit organisations, and soon it was very clear that I enjoyed working with for-profit organisations a lot more. It wasn’t about money – I was fairly careful in my negotiations to never underprice myself. It was more to do with processes, and interactions.

The thing in for-profit companies is that objectives are clear. While not everyone in the company has an incentive to increase the bottom-line, it is not hard to understand what they want based on what they do.

For example, in most cases a sales manager optimises for maximum sales. Financial controllers want to keep a check on costs. And so on. So as part of a consulting assignment, it’s rather easy to know who wants what, and how you should pitch your solution to different people in order to get buy-in.

With a not-for-profit it’s not that clear. While each person may have their own metrics and objectives, because the company is not for profit, these objectives and metrics need not be everything they’re optimising for.

Moreover, in the not for profit world, the lack of money or profit as an objective means you cannot differentiate yourself with efficiency or quantity. Take the example of an organisation which, for whatever reason, gets to advice a ministry on a particular subject, and does so without a fee or only for a nominal fee.

How can a competitor who possibly has a better solution to the same problem “displace” the original organisation? In the business world, this can be done by showing superior metrics and efficiency and offering to do the job at a lower cost and stuff like that. In the not-for-profit setup, you can’t differentiate on things like cost or efficiency, so the only thing you can do is to somehow provide your services in parallel and hope that the client gets it.

And then there is access. If you’re a not-for-profit consultant who has a juicy project, it is in your interest to become a gatekeeper and prevent other potential consultants from getting the same kind of access you have – for you never know if someone else who might get access through you might end up elbowing you out.

Seven

It’s a little over seven years ago that we got married. It was a traditional Hindu ceremony. It was so traditional that we began at around 11am and finished with a ceremonial lunch only around 5:30 pm. And tradition meant that the priests hurried through the mantras, not bothering to explain what they were supposed to say (it’s another matter that had they bothered to stop and explain, we would’ve been getting married for another two days).

It was later that I got to know that some of those mantras were rather insightful, though archaic and backward if you go by modern sensibilities. Like this one Mantra the search for which led me to a website titled “6 noble virtues of an ideal wife“. As the website explains, it is from “Neeti Saara”, written by a Telugu poet Baddena in 13th century (yes, some of the wedding rituals are only 800 years old).

I won’t go into explaining these “noble virtues” here, but as I look back at our seven years of marriage, I realise that Pinky has been a brilliant wife. And she has done so while either studying or keeping a (mostly demanding) full time job for most of the duration of our marriage.

I’m reminded of the time when we lived in Rajajinagar, when I was working as an independent management consultant. I would work from home, and having disposed of our cook a few months earlier, I had the task of making my own lunch.

Pinky would have none of it. She would wake up at 5:30 and painstakingly make tall stacks of chapatis (I was going through one of those “I don’t want to eat rice” phases at that time) before she ran to catch her factory bus, so that I could have a good meal. And the dutiful husband I was, I’d finish the full day’s quota in the afternoon itself which meant she would be forced to cook again once she was back after a tiring day and 40-kilometre commute.

I’m also reminded of the time earlier this year when we’d moved to London and I hadn’t yet figured out what I was going to do here. Pinky not only supported our family financially, but also ferry Berry all the way into town each day so that I could figure out life, possibly find a job and finish my book. And on most days it would be Pinky who would cook dinner after another long commute (usually with a screaming baby).

During my consulting life, every time I had to strike a deal or go through a tough negotiation, I would turn to Pinky. Each time she would unfailingly help me sort out my demons and give me sage advice. On several occasions she saved me from pricing too low. When she would see that I was getting into a bad deal, she would firmly pull me back.

She was insanely supportive when in late 2011 I decided to quit my job and lead a portfolio life. She would find her own cute ways of supporting me in every endeavour, like buying me a new fancy notebook when I was going to meet my first prospective big client, or trying to find me a fancy water bottle prior to my bike trip across Rajasthan.

When I would wake up at 4am to catch the first flight out to see outstation clients, she would wake up along with me, make coffee for me and polish my shoes. On two such occasions she wasn’t around. I almost missed my flight on one of those occasions, and missed my flight on the other!

From time to time she plans fun activities for us to do together, like the time she took me to the A-Paul-ogy art gallery, or when she took me for a Japanese meal to Matsuri and totally bossed the menu (before I knew it she’d ordered a host of dishes which made for a wonderful and complete meal!).

In some senses, I feel I haven’t held up my side of the bargain at times. One thing she absolutely enjoys is for us to watch movies together, but we almost never do that since I don’t generally have the patience to watch movies. She would love to spend a Friday evening cuddling on the couch watching something together, but I prefer to be on my computer instead. She loves being surprised, but my ineptitude means that on most occasions I’ve tried it’s been a shock rather than a surprise.

And then I have my occasional bouts of extreme anger, and she’s borne the brunt of it on several occasions. Usually it involves shouting (I’m an absolute shouter and love to went out my frustrations; she’s the quiet types) and I have to try hard to not get violent at times (on some occasions I don’t succeed in restraining myself). On some occasions it starts with something seemingly silly. On others, I fuck up like crazy. Either ways, it occasionally gets ugly – something I’m definitely not proud about.

I know I have driven her insane. I know that my negativity and NED has rubbed off on her. It is normal for a married couple to influence and change each other, and I know I’ve changed her in ways she absolutely hates.

Yet she’s always been the forgiving sorts. She’s stood by me thick and thin through the seven years of our marriage. Thanks to her adventures during her MBA, I’ve managed to increase my country count (and also got to move to a new country!).

And she’s been a wonderful mother to Berry. One old friend who met Pinky for the first time last year later told me, “man, she is so sorted!”. She remains cool, and seemingly without taking much stress, has managed to turn Berry into a bright and naughty toddler – I’ve mostly been a freerider! Oh, and do you know that she writes an absolutely delightful letter to Berry each month?

It’s mostly been a wonderful seven years with Pinky so far! I know we’ve had the odd low moment, and I want to take this opportunity to apologise to her for that. But in my mind, these have been far far outweighed by all the wonderful times we’ve been through, and all the fun we’ve had together! I hope to get another seventy such wonderful years with this wonderful woman!

And for after that, there’s this:

PS: The more perceptive of the readers here will know that this blog (its predecessor, rather) played a not insignificant part in us meeting. One of the posts that drew her attention, and which got us talking was this.

AlphaZero defeats Stockfish: Quick thoughts

The big news of the day, as far as I’m concerned, is the victory of Google Deepmind’s AlphaZero over Stockfish, currently the highest rated chess engine. This comes barely months after Deepmind’s AlphaGo Zero had bested the earlier avatar of AlphaGo in the game of Go.

Like its Go version, the AlphaZero chess playing machine learnt using reinforcement learning (I remember doing a term paper on the concept back in 2003 but have mostly forgotten). Basically it wasn’t given any “training data”, but the machine trained itself on continuously playing with itself, with feedback given in each stage of learning helping it learn better.

After only about four hours of “training” (basically playing against itself and discovering moves), AlphaZero managed to record this victory in a 100-game match, winning 28 and losing none (the rest of the games were drawn).

There’s a sample game here on the Chess.com website and while this might be a biased sample (it’s likely that the AlphaZero engineers included the most spectacular games in their paper, from which this is taken), the way AlphaZero plays is vastly different from the way engines such as Stockfish have been playing.

I’m not that much of a chess expert (I “retired” from my playing career back in 1994), but the striking things for me from this game were

  • the move 7. d5 against the Queen’s Indian
  • The piece sacrifice a few moves later that was hard to see
  • AlphaZero’s consistent attempts until late in the game to avoid trading queens
  • The move Qh1 somewhere in the middle of the game

In a way (and being consistent with some of the themes of this blog), AlphaZero can be described as a “stud” chess machine, having taught itself to play based on feedback from games it’s already played (the way reinforcement learning broadly works is that actions that led to “good rewards” are incentivised in the next iteration, while those that led to “poor rewards” are penalised. The challenge in this case is to set up chess in a way that is conducive for a reinforcement learning system).

Engines such as StockFish, on the other hand, are absolute “fighters”. They get their “power” by brute force, by going down nearly all possible paths in the game several moves down. This is supplemented by analysis of millions of existing games of various levels which the engine “learns” from – among other things, it learns how to prune and prioritise the paths it searches on. StockFish is also fed a database of chess openings which it remembers and tries to play.

What is interesting is that AlphaZero has “discovered” some popular chess openings through the course of is self-learning. It is interesting to note that some popular openings such as the King’s Indian or French find little favour with this engine, while others such as the Queen’s Gambit or the Queen’s Indian find favour. This is a very interesting development in terms of opening theory itself.

Frequency of openings over time employed by AlphaZero in its “learning” phase. Image sourced from AlphaZero research paper.

In any case, my immediate concern from this development is how it will affect human chess. Over the last decade or two, engines such as stockfish have played a profound role in the development of chess, with current top players such as Magnus Carlsen or Sergey Karjakin having trained extensively with these engines.

The way top grandmasters play has seen a steady change in these years as they have ingested the ideas from engines such as StockFish. The game has become far more quiet and positional, as players seek to gain small advantages which steadily improves over the course of (long) games. This is consistent with the way the engines that players learn from play.

Based on the evidence of the one game I’ve seen of AlphaZero, it plays very differently from the existing engines. Based on this, it will be interesting to see how human players who train with AlphaZero based engines (or their clones) will change their game.

Maybe chess will turn back to being a bit more tactical than it’s been in the last decade? It’s hard to say right now!

Bond Market Liquidity and Selection Bias

I’ve long been a fan of Matt Levine’s excellent Money Stuff newsletter. I’ve mentioned this newsletter here several times in the past, and on one such occasion, I got a link back.

One of my favourite sections in Levine’s newsletter is called “people are worried about bond market liquidity”. One reason I got interested in it was that I was writing a book on Liquidity (speaking of which, there’s a formal launch function in Bangalore on the 15th). More importantly, it was rather entertainingly written, and informative as well.

I appreciated the section so much that I ended up calling one of the sections of one of the chapters of my book “people are worried about bond market liquidity”. 

In any case, the Levine has outdone himself several times over in his latest instalment of worries about bond market liquidity. This one is from Friday’s newsletter. I strongly encourage you to read fully the section on people being worried about bond market liquidity.

To summarise, the basic idea is that while people are generally worried about bond market liquidity, a lot of studies about such liquidity by academics and regulators have concluded that bond market liquidity is just fine. This is based on the finding that the bid-ask spread (gap between prices at which a dealer is willing to buy or sell a security) still remains tight, and so liquidity is just fine.

But the problem is that, as Levine beautifully describes the idea, there is a strong case of selection bias. While the bid-ask spread has indeed narrowed, what this data point misses out is that many trades that could have otherwise happened are not happening, and so the data comes from a very biased sample.

Levine does a much better job of describing this than me, but there are two ways in which a banker can facilitate bond trading – by either taking possession of the bonds (in other words, being a “market maker” (PS: I have a chapter on this in my book) ), or by simply helping find a counterparty to the trade, thus acting like a broker (I have a chapter on brokers as well in my book).

A new paper by economists at the Federal Reserve Board confirms that the general finding that bond market liquidity is okay is affected by selection bias. The authors find that spreads are tighter (and sometimes negative) when bankers are playing the role of brokers than when they are playing the role of market makers.

In the very first chapter of my book (dealing with football transfer markets), I had mentioned that the bid-ask spread of a market is a good indicator of market liquidity. That the higher the bid-ask spread, the less liquid a market.

Later on in the book, I’d also mentioned that the money that an intermediary can make is again a function of how inherent the market is.

This story about bond market liquidity puts both these assertions into question. Bond markets see tight bid-ask spreads and bankers make little or no money (as the paper linked to above says, spreads are frequently negative). Based on my book, both of these should indicate that the market is quite liquid.

However, it turns out that both the bid-ask spread and fees made by intermediaries are biased estimates, since they don’t take into account the trades that were not done.

With bankers cutting down on market making activity (see Levine’s post or the paper for more details), there is many a time when a customer will not be able to trade at all since the bankers are unable to find them a counterparty (in the pre Volcker Rule days, bankers would’ve simply stepped in themselves and taken the other side of the trade). In such cases, the effective bid-ask spread is infinity, since the market has disappeared.

Technically this needs to be included while calculating the overall bid-ask spread. How this can actually be achieve is yet another question!

Algorithms and the Turing Test

One massive concern about the rise of artificial intelligence and machine learning is the perpetuation of human biases. This could be racism (the story, possibly apocryphal, of a black person being tagged as a gorilla) or sexism (see tweet below) or any other forms of discrimination (objective looking data that actually represents certain divisions).

In other words, mainstream concern about artificial intelligence is that it is too human, and such systems should somehow be “cured” of their human biases in order to be fair.

My concern, though, is the opposite. That many of the artificial intelligence and machine learning systems are not “human enough”. In other words, that most present day artificial intelligence and machine learning systems would not pass the Turing Test.

To remind you of the test, here is an extract from Wikipedia:

The Turing test, developed by Alan Turing in 1950, is a test of a machine’s ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. Turing proposed that a human evaluator would judge natural language conversationsbetween a human and a machine designed to generate human-like responses. The evaluator would be aware that one of the two partners in conversation is a machine, and all participants would be separated from one another. The conversation would be limited to a text-only channel such as a computer keyboard and screen so the result would not depend on the machine’s ability to render words as speech.[2] If the evaluator cannot reliably tell the machine from the human, the machine is said to have passed the test. The test does not check the ability to give correct answers to questions, only how closely answers resemble those a human would give.

The test was introduced by Turing in his paper, “Computing Machinery and Intelligence“, while working at the University of Manchester (Turing, 1950; p. 460).

Think of any recommender system, for example. With some effort, it is easy for a reasonably intelligent human to realise that the recommendations are being made by a machine. Even the most carefully designed recommender systems give away the fact that their intelligence is artificial once in a while.

To take a familiar example, people talk about the joy of discovering books in bookshops, and about the quality of recommendations given by an expert bookseller who gets his customers. Now, Amazon perhaps collects more data about its customers than any such bookseller, and uses them to recommend books. However, even a little scrolling reveals that the recommendations are rather mechanical and predictable.

It’s similar with my recommendations on Netflix – after a point you know the mechanics behind them.

In some sense this predictability is because the designers possibly think it’s a good thing – Netflix, for example, tells you why it has recommended a particular video. The designers of these algorithms possibly think that explaining their decisions might given their human customers more reason to trust them.

(As an aside, it is common for people to rant against the “opaque” algorithms that drive systems as diverse as Facebook’s News Feed and Uber’s Surge Pricing. So perhaps some algorithm designers do see reason in wanting to explain themselves).

The way I see it, though, by attempting to explain themselves these algorithms are giving themselves away, and willingly failing the Turing test. Whenever recommendations sound purely mechanical, there is reason for people to start trusting them less. And when equally mechanical reasons are given for these mechanical recommendations, the desire to trust the recommendations falls further.

If I were to design a recommendation system, I’d introduce some irrationality, some hard-to-determine randomness to try make the customer believe that there is actually a person behind them. I believe it is a necessary condition for recommendations to become truly personalised!

Python and Hindi

So I’ve recently discovered that using Python to analyse data is, to me, like talking in Hindi. Let me explain.

Back in 2008-9 I lived in Delhi, where the only language spoken was Hindi. Now, while I’ve learnt Hindi formally in school (I got 90 out of 100 in my 10th boards!), and watched plenty of Hindi movies, I’ve never been particularly fluent in the language.

The basic problem is that I don’t know the language well enough to think in it. So when I’m talking Hindi, I usually think in Kannada and then translate my thoughts. This means my speech is slow – even Atal Behari Vajpayee can speak Hindi faster than me.

More importantly, thinking in Kannada and translating means that I can get several idioms wrong (can’t think of particular examples now). And I end up using the language in ways that native speakers don’t (again can’t think of examples here).

I recently realised it’s the same with programming languages. For some 7 years now I’ve mostly used R for data analysis, and have grown super comfortable with it. However, at work nowadays I’m required to use Python for my analysis, to ensure consistency with the rest of the firm.

While I’ve grown reasonably comfortable with using Python over the last few months, I realise that I have the same Hindi problem. I simply can’t think in Python. Any analysis I need to do, I think about it in R terms, and then mentally translate the code before performing it in Python.

This results in several inefficiencies. Firstly, the two languages are constructed differently and optimised for different things. When I think in one language and mentally translate the code to the other, I’m exploiting the efficiencies of the thinking language rather than the efficiencies of the coding language.

Then, the translation process itself can be ugly. What might be one line of code in R can sometimes take 15 lines in Python (and vice versa). So I end up writing insanely verbose code that is hard to read.

Such code also looks ugly – a “native user” of the language finds it rather funnily written, and will find it hard to read.

A decade ago, after a year of struggling in Delhi, I packed my bags and moved back to Bangalore, where I could both think and speak in Kannada. Wonder what this implies in a programming context!

Generalist and specialist managers

A really long time ago, I’d written this blog post about “comparative advantage” versus “competitive advantage” employees. A competitive advantage employee is better at a particular kind of task or skill compared to the rest of the team, and he is valued for that kind of skill.

A comparative advantage employee, on the other hand, is “dominated” by at least one other person in the team – in the sense that someone else is better than this person at everything required for the job. In that sense, the value that the comparative advantage employee adds is by taking load off his colleagues, and allowing them to do more (and focus on the more productive parts of their jobs).

Thinking about it now, I realise that a similar classification exists from the manager’s perspective as well. And this is broadly correlated with whether the manager manages a “generalist” or a “specialist” team.

A specialist manager manages a team all of whose members work on and excels at one specialist task. This task could come from any part of the organisation – it could be sales or a particular kind of operations, or some financial activity or whatever. The defining feature of this kind of task is that it is usually repetitive and needs to be done in high volumes. Such tasks also offer high “returns to experience”.

The average employee of a specialist team is usually a “comparative advantage” employee. In most cases, such an employee is likely to be “dominated” by the manager, and the value he adds is by taking the load off the manager and allowing him to do more. Over the course of time, he becomes good enough at the job to become a manager himself, and the cycle continues – he will manage a team of people who are mostly inferior to him in the job.

Due to managers dominating direct reports, such teams end up being largely hierarchical, and there can be a tendency for the manager to micro-manage – if you are better at the task than the person actually doing it, you can do worse than giving specific instructions.

Generalist managers, on the other hand, manage teams that involve at least a few competitive advantage employees. What this implies is that there is a set of people who are better than the manager at certain parts of the business. The manager’s role in such a team is more of a facilitator, in terms of bringing the team together and coordinating in a way that they can maximise the team’s effectiveness.

Generalist managers seldom micromanage, since usually their team members know better (literally). They are also usually open-minded, since extracting full value from the team means recognising each member’s strengths (and consequently their own weaknesses). They learn the art of asking questions and verifying insights and work of the team in a cheap manner (remember from complexity theory that the complexity of verifying a solution can be much lower than the complexity of finding a solution).

Regular readers of the blog might have anticipated this paragraph – the trouble comes when a generalist manager has to manage a specialist team or the other way round.

A generalist manager managing a specialist team may not offer as much as he can to the team based on his experience. He might be too hands-off and team members used to more handholding and direction might feel lost. And so on.

A specialist manager managing a generalist team can be more damaging – not appreciating that some members might know more about some parts of the business might limit the performance of the team (since what the team can do is limited by what the manager knows). Also too much micromanagement on employees who know better about some parts of the business than the manager can result in disillusionment and ultimately backfire on the manager.

I wonder if this has something to do with the Peter Principle!