More on statistics and machine learning

I’m thinking of a client problem right now, and I thought that something that we need to predict can be modelled as a function of a few other things that we will know.

Initially I was thinking about it from the machine learning perspective, and my thought process went “this can be modelled as a function of X, Y and Z. Once this is modelled, then we can use X, Y and Z to predict this going forward”.

And then a minute later I context switched into the statistical way of thinking. And now my thinking went “I think this can be modelled as a function of X, Y and Z. Let me build a quick model to see if the goodness of fit, and whether a signal actually exists”.

Now this might reflect my own biases, and my own processes for learning to do statistics and machine learning, but one important difference I find is that in statistics you are concerned about the goodness of fit, and whether there is a “signal” at all.

While in machine learning as well we look at what the predictive ability is (area under ROC curve and all that), there is a bit of delay in the process between the time we model and the time we look for the goodness of fit. What this means is that sometimes we can get a bit too certain about the models that we want to build without thinking if in the first place they make sense and there’s a signal in that.

For example, in the machine learning world, the concept of R Square is not defined for regression –  the only thing that matters is how well you can predict out of sample. So while you’re building the regression (machine learning) model, you don’t have immediate feedback on what to include and what to exclude and whether there is a signal.

I must remind you that machine learning methods are typically used when we are dealing with really high dimensional data, and where the signal usually exists in the interplay between explanatory variables rather than in a single explanatory variable. Statistics, on the other hand, is used more for low dimensional problems where each variable has reasonable predictive power by itself.

It is possibly a quirk of how the two disciplines are practiced that statistics people are inherently more sceptical about the existence of signal, and machine learning guys are more certain that their model makes sense.

What do you think?

Studs and fighters: Origin

As far as this blog is concerned, the concept of studs and fighters began sometime in 2007, when I wrote the canonical blog post on the topic. Since then the topic has been much used and abused.

Recently, though, I remembered when I had first come across the concept of studs and fighters. This goes way back to 1999, and has its origins in a conversation with two people who I consider as among the studdest people I’ve ever met (they’re both now professors at highly reputed universities).

We were on a day-long train journey, and were discussing people we had spent a considerable amount of time with over the previous one month. It was a general gossip session, the sort that was common to train journeys in the days before smartphones made people insular.

While discussing about one guy we had met, one of us (it wasn’t me for sure. It was one of the other two but I now can’t recall which of them it was) said “well, he isn’t particularly clever, but he is a very hard worker for sure”.

And so over time this distinction got institutionalised, first in my head and then in the heads of all my readers. There were two ways to be good at something – by either being clever or by being a very hard worker.

Thinking about it now, it seems rather inevitable that the concept that would become studs and fighters came about in the middle of a conversation among studs.

News

I wake up early on weekdays nowadays, so go the first two hours of the day without really knowing what is happening in the world. As you might know, I’m on a social media break, so that source of news is cut off. And it’s only around 7 am by when a copy of the Business Standard gets delivered to my door.

Until last month, a copy of the Deccan Herald would arrive at home as well, but I stopped it after I found it to be largely useless. A lot of stories in that newspaper were written as they might have been 20 or 30 years ago. There was little distinction between reporting and analysis and opinion. A lot of news couldn’t be simply consumed without the accompanying (and sometimes patronising) opinion.

The Business Standard, which I started reading in 2005, is still a very good paper. The editorials continue to be first-rate (though their quality had dipped in the 2011-14 period). The analysis pieces and columns cover a variety of topics that simply don’t make it to social media (since they aren’t really “sensational”). And the newspaper is “crisp” and quickly tells you what’s going on in India.

For two years, when I lived in London, I lived without a daily newspaper, and it was a struggle. Online newspapers have simply not been able to provide the same kind of product as offline newspapers. And the reason is that online newspapers are “flat” – all the contextualising and prioritising that a dead-tree paper can do is completely absent in the online version.

In a dead-tree newspaper, you know how important a piece of news is based on the page it appears, the size of the headline, the size of the column and so on. Based on where it appears, you know if it is news or analysis or opinion. In case it is opinion, you can easily see who has written it before you “click through” (start reading it). You can easily how big a piece is (and how much of your time it will take) before deciding to invest time in it.

All this is absent from an online newspaper. Check out, for example, the homepage of the Business Standard, that I so fulsomely praised earlier in this post.

It is impossible to know what’s the important stuff here. If I have only five minutes to read, I don’t know what to focus on. I don’t know which of this is opinion and which is news. Before I click through, I don’t know how big a piece is or who has written it or if it has been syndicated.

Unless the link has come from a qualified source (such as Twitter) I don’t know much about it, and so don’t know how to consume it. This might explain to you why a lot of online news sources are losing revenues to the likes of Google or Facebook – the latter do the important job of putting the news in context!

Finally, I’m glad I now consume news only once a day (from the physical paper). Sometimes, what is news intra-day would have ceased to be news by nightfall. So when you consume news at a reasonable interval (such as daily), what comes to you is “qualified” real stuff. A piece of news should have been important enough for a day to make it to the next day’s newspapers. And once a day is also a reasonable interval to get to know of what is happening in the world.

Data, football and astrology

Jonathan Wilson has an amusing article on data and football, and how many data-oriented managers in football have also been incredibly superstitious.

This is in response to BT Sport’s (one of the UK broadcasters of the Premier League) announcement of it’s “Unscripted” promotion where “some of the world’s foremost experts in both sports and artificial intelligence to produce a groundbreaking prophecy of the forthcoming season”.

Wilson writes:

I was reminded also of the 1982 film adaptation of Agatha Christie’s 1939 novel Murder is Easy in which a computer scientist played by Bill Bixby enters the details of the case into a programme he has coded to give the name of the murderer. As it turns out, the programmer knows this is nonsense and is merely trying to gauge the reaction of the heroine, played by Lesley-Anne Down, when her name flashes on the screen.

But this, of course, is not what data-based analysis is for. Its predictive element deals in probability not prophecy. It is not possessed of some oracular genius. (That said, it is an intriguing metaphysical question: what if you had all the data, not just ability and fitness, but every detail of players’ diet, relationships and mental state, the angle of blades of grass on the pitch, an assessment of how the breathing of fans affected air flow in the stadium … would the game’s course then be inevitable?)

This reminded me of my own piece that I wrote last year about how data science “is simply the new astrology“.

Gaming and social media

Now that I’m off social media I’m wondering if it’s a good idea to replace it with gaming.

Basically I need a fix. I have this need to be distracted/”accelerated” all the time. And I need something that I can go to, get distracted for a little bit and then come back to work.

Sometimes a text message conversation in the middle of work does the trick, but not all conversations are good for this (some people are just too quick in replying and end up stressing me). Also there’s the issue of finding someone to talk to when you need to.

Over the last decade or so I’ve turned to social media. Twitter with its stream of tweets provides good stimulation. The problem with that is that it results in a lot of mental bandwidth going wasted due to massive context switching that happens. Yes it provides distraction but is also tiring.

So I’m wondering if I should try out gaming. I know it is addictive, which is both good and bad (addictions are good when you are trying to take your mind off something). And gaming sure provides the “kick” though I’m not sure if it will be as taxing on the mind as social media. The key, however, is to find a game that can be played in short bursts whenever I need that “kick”.

What do you think? Is there a game that I can use for my distractions, or do you think starting gaming might unleash a different beast that might be hard to tame?

Television and interior design

One of the most under-rated developments in the world of architecture and interior design has been the rise of the flat-screen television. Its earlier avatar, the Cathode Ray Tube version, was big and bulky, and needed special arrangements to keep. One solution was to keep it in corners. Another was to have purpose-built deep “TV cabinets” into which these big screens would go.

In the house that I grew up in, there was a purpose-built corner to keep our televisions. Later on in life, we got a television cabinet to put in that place, that housed the television, music system, VCR and a host of other things.

For the last decade, which has largely coincided with the time when flat-screen LCD/LED TVs have replaced their CRT variations, I’ve seen various tenants struggle to find a good spot for the TVs. For the corner is too inelegant for the flat screen television – it needs to be placed flat against the middle of a large wall.

When the flat screen TV replaced the CRT TV, out went the bulky “TV cabinets” and in came the “console” – a short table on which you kept the TV, and below which you kept the accompanying accessories such as the “set top box” and DVD player. We had even got a purpose-built TV console with a drawer to store DVDs in.

Four years later, we’d dispensed with our DVD player (at a time when my wife’s job involved selling DVDs and CDs, we had no device at home that could play any of these storage devices!). And now we have “cut the cord”. After we returned to India earlier this year, we decided to not get cable TV, relying on streaming through our Fire stick instead.

And this heralds the next phase in which television drives interior design.

In the early days of flat screen TVs, it became common for people to “wall mount” them. This was usually a space-saving device, though people still needed a sort of console to store input devices such as set top boxes and DVD players.

Now, with the cable having been cut and DVD player not that common, wall mounting doesn’t make sense at all. For with WiFi-based streaming devices, the TV is now truly mobile.

In the last couple of months, the TV has nominally resided in our living room, but we’ve frequently taken it to whichever room we wanted to watch it in. All that we need to move the TV is a table to keep it on, and a pair of plug points to plug in the TV and the fire stick.

In our latest home reorganisation we’ve even dispensed with a permanent home for the TV in the living room, thus radically altering its design and creating more space (the default location of the TV now is in the study). The TV console doesn’t make any sense, and has been temporarily converted into a shoe rack. And the TV moves from room to room (it’s not that heavy, either), depending on where we want to watch it.

When the CRT TV gave way to the flat screen, architects responded by creating spaces where TVs could be put in the middle of a long wall, either mounted on the wall or kept on a console. That the TV’s position in the house changed meant that the overall architecture of houses changed as well.

Now it will be interesting to see what large-scale architectural changes get driven by cord-cutting and the realisation that the TV is essentially a mobile device.

Coordinated and uncoordinated potlucks

Some potluck meals are coordinated. One or more coordinators assume leadership and instruct each attending member what precisely to bring. It’s somewhat like central planning in that sense – the coordinators make assumptions on what each person wants and how much they will eat and what goes well with what, and make plans accordingly.

Uncoordinated potlucks can be more interesting. Here, people don’t talk about what to bring, and simply bring what they think the group might be interested. This can result in widely varying outcomes – some great meals, occasionally a lot of wasted food, and some weird mixes of starters, main courses and desserts.

We had one such uncoordinated potluck at my daughter’s school picnic last week. All children were accompanied by their parents and were asked to bring “snacks”. Nothing was specified apart from the fact that we should bring it in steel containers, and that we should get homemade stuff.

Now, for a bit of background. For slightly older kids (my daughter doesn’t qualify yet) the school has a rotating roster for lunch, where each kid brings in lunch for the entire class on each day. So parents are used to sending lunch for all the children, and children are used to eating a variety of foods. A friend who sent his daughter to the same school tells me that it can become a bit too competitive sometimes, with families seeking to outdo one another with the fanciness of the foods they send.

In that sense, I guess the families of these older kids had some information on what normally came for lunch and what got eaten and so on – a piece of information we didn’t have. The big difference between this picnic potluck and school lunch (though I’m not sure if other parents knew of this distinction) was that this was “anonymous”.

All of us kept our steel boxes and vessels on a large table set up for the purpose, so when people served themselves there was little clue of which food had come from whose house. In that sense there was no point showing off (though we tried, taking hummus with carrot and cucumber sticks). And it resulted in what I thought was a fascinating set of food, though I guess some of it couldn’t really be classified as “snack”.

The fastest to disappear was a boxful of chitranna (lemon rice). I thought it went rather well with roasted and salted peanuts that someone else had bought. There were some takers for our hummus as well, though our cut apples didn’t “do that well”. I saw a boxful of un-taken idlis towards the end of the snack session. Someone had brought boiled sweet corn on the cob. And there were many varieties of cakes that families had (presumably baked and) brought.

What I found interesting was that despite their being zero coordination between the families, they had together served up what was a pretty fascinating snack, with lots of variety. “Starters”, “Mains”, “Desserts” and “Sides” were all well represented, even if the balance wasn’t precisely right.

The number of families involved here (upwards of 30) meant that perfect coordination would’ve been nigh impossible, and I’m not sure if a command-and-control style coordinated potluck would have worked in any case (that would have also run the risk of a family bunking the picnic last moment, and an important piece of the puzzle missing).

The uncoordinated potluck meant that there were no such imbalances, and families, left to themselves and without any feedback, had managed to serve themselves a pretty good “snack”!

More power to decentralised systems!

Gamification and finite and infinite games

Ok here I’m integrating a few concepts that I learnt via Venkatesh Guru Rao. The first is that of Finite and Infinite games, a classic if hard to read book written by philosopher James Carse (which I initially discovered thanks to his Breaking Smart Season 1 compilation). The second is of “playflow”, which again I discovered through a recent edition of his newsletter.

A lot of companies try to “gamify” the experiences for their employees in order to make work more fun, and to possibly make them more efficient.

For example, sales organisations offer complicated incentives (one of my historically favourite work assignments has been to help a large client optimise these incentives). These incentives are offered at multiple “slabs”, and used to drive multiple objectives (customer acquisition, retention, cross-sell, etc.). And by offering employees incentives for achieving some combination of these objectives, the experience is being “gamified”. It’s like the employee is gaining points by achieving each of these objectives, and the points together lead to some “reward”.

This is just one example. There are several other ways in which organisations try to gamify the experience for their employees. All of them involve some sort of award of “points” for things that people do, and then a combination of points leading to some “reward”.

The problem with gamification is that the games organisations design are usually finite games. “Sell 10 more widgets in the next month”. “Limit your emails to a maximum of 200 words in the next fifteen days”. “Visit at least one client each day”. And so on.

Running an organisation, however, is an infinite game. At the basic level, the objective of an organisation is to remain a going concern, and keep on running. Growth and dividends and shareholder returns are secondary to that – if the organisation is not a going concern, none of that matters.

And there is the contradiction – the organisation is fundamentally playing an infinite game. The employees, thanks to the gamified experience, are playing finite games. And they aren’t always compatible.

Of course, there are situations where finite games can be designed in a way that their objectives align with the objectives of the overarching infinite game. This, however, is not always possible. Hence, gamification is not always a good strategy for organisations.

Organisations have figured out the solution to this, of course. There is a simple way to make employees play the same infinite game as the organisation – by offering employees equity in the company. Except that employees have the option of converting that to a finite game by selling the said equity.

Whoever said incentive alignment is an easy task..

 

Marginalised communities and success

Yesterday I was listening to this podcast where Tyler Cowen interviews Neal Stephenson, who is perhaps the only Science Fiction author whose books I’ve read. Cowen talks about the characters in Stephenson’s The Baroque Cycle, a masterful 3000-page work which I polished off in a month in 2014.

The key part of the conversation for me is this:

COWEN: Given your focus on the Puritans and the Baroque Cycle, do you think Christianity was a fundamental driver of the Industrial Revolution and the Scientific Revolution, and that’s why it occurred in northwestern Europe? Or not?

STEPHENSON: One of the things that comes up in the books you’re talking about is the existence of a certain kind of out-communities that were weirdly overrepresented among people who created new economic systems, opened up new trade routes, and so on.

I’m talking about Huguenots, who were the Protestants in France who suffered a lot of oppression. I’m talking about the Puritans in England, who were not part of the established church and so also came in for a lot of oppression. Armenians, Jews, Parsis, various other minority communities that, precisely because of their outsider minority status, were forced to form long-range networks and go about things in an unconventional, innovative way.

So when we think about communities such as Jews or Parsis, and think about their outsized contribution to business or culture, it is this point that Stephenson makes that we should keep in mind. Because Jews and Parsis and Armenians were outsiders, they were “forced to form long-range networks”.

In most cases, for most people of these communities, these long-range networks and unconventional way of doing things didn’t pay off, and they ended up being worse off compared to comparable people from the majority communities in wherever they lived.

However, in the few cases where these long-range networks and innovative ways of doing things succeeded, they succeeded spectacularly. And these incidents are cases that we have in mind when we think about the spectacular success or outsized contributions of these communities.

Another way to think of this is – denied “normal life”, people from marginalised communities were forced to take on much more risk in life. The expected value of this risk might have been negative, but this higher risk meant that these communities had a much better “upper tail” than the majority communities that suppressed and oppressed them.

Given that in terms of long-term contributions and impact and public visibility it is only the tails of the distribution that matter (mediocrity doesn’t make news), we think of these communities as having been extraordinary, and wonder if they have “better genes” and so on.

It’s a simple case of risk, and oppression. This, of course, is no justification for oppressing swathes of people and forcing them to take more risks than necessary. People need to decide on their own risk preferences.

10X Studs and Fighters

Tech twitter, for the last week, has been inundated with unending debate on this tweetstorm by a VC about “10X engineers”. The tweetstorm was engineered by Shekhar Kirani, a Partner at Accel Partners.

I have friends and twitter-followees on both sides of the debate. There isn’t much to describe more about the “paksh” side of the debate. Read Shekhar’s tweetstorm I’ve put above, and you’ll know all there is to this side.

The vipaksh side argues that this normalises “toxicity” and “bad behaviour” among engineers (about “10X engineers”‘s hatred for meetings, and their not adhering to processes etc.). Someone I follow went to the extent to say that this kind of behaviour among engineers is a sign of privilege and lack of empathy.

This is just the gist of the argument. You can just do a search of “10X engineer”, ignore the jokes (most of them are pretty bad) and read people’s actual arguments for and against “10X engineers”.

Regular readers of this blog might be familiar with the “studs and fighters” framework, which I used so often in the 2007-9 period that several people threatened to stop reading me unless I stopped using the framework. I put it on a temporary hiatus and then revived it a couple of years back because I decided it’s too useful a framework to ignore.

One of the fundamental features of the studs and fighters framework is that studs and fighters respectively think that everyone else is like themselves. And this can create problems at the organisational level. I’d spoken about this in the introductory post on the framework.

To me this debate about 10X engineers and whether they are good or bad reminds me of the conflict between studs and fighters. Studs want to work their way. They are really good at what they’re competent at, and absolutely suck at pretty much everything else. So they try to avoid things they’re bad at, can sometimes be individualistic and prefer to work alone, and hope that how good they are at the things they’re good at will compensate for all that they suck elsewhere.

Fighters, on the other hand, are process driven, methodical, patient and sticklers for rules. They believe that output is proportional to input, and that it is impossible for anyone to have a 10X impact, even 1/10th of the time (:P). They believe that everyone needs to “come together as a group and go through a process”.

I can go on but won’t.

So should your organisation employ 10X engineers or not? Do you tolerate the odd “10X engineer” who may not follow company policy and all that in return for their superior contributions? There is no easy answer to this but overall I think companies together will follow a “mixed strategy”.

Some companies will be encouraging of 10X behaviour, and you will see 10X people gravitating towards such companies. Others will dissuade such behaviour and the 10X people there, not seeing any upside, will leave to join the 10X companies (again I’ve written about how you can have “stud organisations” and “fighter organisations”.

Note that it’s difficult to run an organisation with solely 10X people (they’re bad at managing stuff), so organisations that engage 10X people will also employ “fighters” who are cognisant that 10X people exist and know how they should be managed. In fact, being a fighter while recognising and being able to manage 10X behaviour is, I think, an important skill.

As for myself, I don’t like one part of Shekhar Kirani’s definition – that he restricts it to “engineers”. I think the sort of behaviour he describes is present in other fields and skills as well. Some people see the point in that. Others don’t.

Life is a mixed strategy.