Information gain from relationship attempts

Every failed relationship (or attempt at a relationship) has plenty to teach you – in terms of things you got right, or wrong. Things that would make you cringe later on, and others that would make you wonder why the relationship failed. Each failed relationship (or attempt) helps you recalibrate yourself as a person – in terms of what kind of people to go after, and what kind of strategies to adopt during the process. Thus, a relationship is important not only from the direct joy it provides you, but also in terms of learnings for future relationships.

The standard model about “finding your level” in terms of determining your expectations from a potential partners involves trial and error. You “sample” by hitting on someone who you think might be a good fit. If it goes well, story ends. Else, you “learn” from this experience and hit on someone else.

How good a learner you are determines how many attempts you’ll take to find someone “your level” who is a “good fit” and end up in a great relationship. Yet, the kind of attempts you make puts a natural cap on the amount of information you extract from the attempt.

For example, there might be a potential counterparty with whom you have an extremely low (close to nothing) chance of getting into a relationship. Conventional wisdom says that you shouldn’t attempt hitting on her (to avoid pronoun confusion, let’s assuming that everyone you can hit on is feminine. Adjust accordingly if your preferences vary), for the odds are stacked against.

While this is good enough reason not to attempt that relationship (though sometimes the downside might be low enough for you to take a punt), the other problem is that you don’t learn anything from it. The extremely low prior probability of succeeding would mean that there is no information from this that can help tune your system. So you’re wasting your time in more than one way.

It works the other way also. Let’s say there’s someone who really looks up to you and wants to be in a relationship with you. You know that all it takes for you to get into a relationship with her is to express interest. If you know the relationship will add value to you, go ahead. However, it is absolutely useless in terms of your “find your level” – the extremely high prior probability means it won’t add sufficient value to the process.

So while they say that someone who’s been through failed relationships (or attempts at relationships) is experienced and has a more refined set of expectations, the sheer number matters less than the quality. It is the amount of information you’ve been able to extract from each such relationship (or attempt). A one-sided (where one of you is clearly “out of the league” of the other, doesn’t matter who is who) relationship doesn’t add much value.

Happy Valentine’s Day!

Number thirteen

The number of rats I’ve killed in my life has remained constant at twelve for way too long. One of my biggest sources of embarrassment in recent times has been in 2011 when a rat inhabited our home for a full fortnight without me catching it. I’d even chased it around the house one day but it had proved elusive. Considering that that was my first attempt at killing a rat in Pinky’s presence, after having bragged much to her about my previous twelve kills, it was a major disappointment. It was as if my manhood was in question.
I made up for that today by killing number thirteen. This one was rather timid and easy kill. I’d closed the door between the drawing room and back room of my in-laws’ house but the stupid rodent just kept pushing against the door trying to open it. At that moment my mother in law alerted me (I was on a phone call in another room) and in I came and whacked the rat a few times with a broom. It quickly keeled over and I put it in a polythene bag and threw it out.


Of course there was considerable groundwork I’d done before this. I’d procured from the mother-in-law a broom and a long stick. I’d closed all the doors and thus restricted the rat to one closed space. I’d tapped around the entire house with the stick trying to scare the rat. And then I moved away to talk on the phone. Groundwork thus laid I proceeded to whack the rodent.

Pinky may not have seen the kill herself but I was on the phone with her as I whacked the rat. Phone in one hand held to the ear. Broom in another whacking the rat dead. Quite I sight I made I imagine!

And while Pinky did not see this kill directly I did one better – Pinky’s mother was there through the entire process (she did her bit by supplying the whacking implements and sacrificing a sweet to lay bait for the rat and most importantly alerting me when the rodent made its stupid move). I feel so glad. I feel like my life has been resurrected. I can finally rest, having made up fit that spectacular failure of 2011.

The rat is dead. Long live the rat!

Should you have an analytics team?

In an earlier post a couple of weeks back, I had talked about the importance of business people knowing numbers and numbers people knowing business, and had put in a small advertisement for my consulting services by mentioning that I know both business and numbers and work at their cusp. In this post, I take that further and analyze if it makes sense to have a dedicated analytics team.

Following the data boom, most companies have decided (rightly) that they need to do something to take advantage of all the data that they have and have created dedicated analytics teams. These teams, normally staffed with people from a quantitative or statistical background, with perhaps a few MBAs, is in charge of taking care of all the data the company has along with doing some rudimentary analysis. The question is if having such dedicated teams is effective or if it is better to have numbers-enabled people across the firm.

Having an analytics team makes sense from the point of view of economies of scale. People who are conversant with numbers are hard to come by, and when you find some, it makes sense to put them together and get them to work exclusively on numerical problems. That also ensures collaboration and knowledge sharing and that can have positive externalities.

Then, there is the data aspect. Anyone doing business analytics within a firm needs access to data from all over the firm, and if the firm doesn’t have a centralized data warehouse which houses all its data, one task of each analytics person would be to get together the data that they need for their analysis. Here again, the economies of scale of having an integrated analytics team work. The job of putting together data from multiple parts of the firm is not solved multiple times, and thus the analysts can spend more time on analyzing rather than collecting data.

So far so good. However, writing a while back I had explained that investment banks’ policies of having exclusive quant teams have doomed them to long-term failure. My contention there (including an insider view) was that an exclusive quant team whose only job is to model and which doesn’t have a view of the market can quickly get insular, and can lead to groupthink. People are more likely to solve for problems as defined by their models rather than problems posed by the market. This, I had mentioned can soon lead to a disconnect between the bank’s models and the markets, and ultimately lead to trading losses.

Extending that argument, it works the same way with non-banking firms as well. When you put together a group of numbers people and call them the analytics group, and only give them the job of building models rather than looking at actual business issues, they are likely to get similarly insular and opaque. While initially they might do well, soon they start getting disconnected from the actual business the firm is doing, and soon fall in love with their models. Soon, like the quants at big investment banks, they too will start solving for their models rather than for the actual business, and that prevents the rest of the firm from getting the best out of them.

Then there is the jargon. You say “I fitted a multinomial logistic regression and it gave me a p-value of 0.05 so this model is correct”, the business manager without much clue of numbers can be bulldozed into submission. By talking a language which most of the firm understands you are obscuring yourself, which leads to two responses from the rest. Either they deem the analytics team to be incapable (since they fail to talk the language of business, in which case the purpose of existence of the analytics team may be lost), or they assume the analytics team to be fundamentally superior (thanks to the obscurity in the language), in which case there is the risk of incorrect and possibly inappropriate models being adopted.

I can think of several solutions for this – but irrespective of what solution you ultimately adopt –  whether you go completely centralized or completely distributed or a hybrid like above – the key step in getting the best out of your analytics is to have your senior and senior-middle management team conversant with numbers. By that I don’t mean that they all go for a course in statistics. What I mean is that your middle and senior management should know how to solve problems using numbers. When they see data, they should have the ability to ask the right kind of questions. Irrespective of how the analytics team is placed, as long as you ask them the right kind of questions, you are likely to benefit from their work (assuming basic levels of competence of course). This way, they can remain conversant with the analytics people, and a middle ground can be established so that insights from numbers can actually flow into business.

So here is the plug for this post – shortly I’ll be launching short (1-day) workshops for middle and senior level managers in analytics. Keep watching this space 🙂


Bayesian Recognition

We don’t meet often, but every time we talk, she reminds me that I had failed to recognize her the first time we had met after graduating together from school. Yes, I could claim in my defence that I was seeing her for the first time in over six years. While that might be a valid excuse for most people, it doesn’t apply to me, since I normally claim to have superior long-term memory. If I’ve seen you somewhere before, I ought to recognize you. The only times I don’t I’m pretending, since I don’t want to embarrass you (and myself) by recognizing you while you don’t recognize me (see this incident for an example of this).

The reason for my failure that cold Bangalore evening in December 2006 was that my Bayesian system had failed me. Let me explain, in the process giving you an insight into my Bayesian system which I use to recognize you when I meet you.

About a month or two back, I was at a friend’s wedding, which is where I hit upon this term “Bayesian recognition” to explain this phenomenon  (which I’ve been practicing for ages). Now, this friend whose wedding I was attending was one year my junior at two different schools. As you might expect at an event where you and the host share more than one social network, there were a lot of familiar faces. Some people I knew fairly well, and could easily recognize. But the others had to go through a “Bayesian search”.

So when I saw someone who was one of three people I know – let’s say X, Y and Z. In order to determine which of these this person is, I would ask myself two questions – firstly, what were the prior odds that the person I saw could be each of X, Y or Z. Secondly, what were the odds of each of X, Y and Z being there at that event. Note that the latter is important. For example, if someone at the event looks like you and I know (for example) that you are currently in another country, despite the strong resemblance I can discount the possibility that that person is you, and go ahead with my search.

Note that this differs from “frequentist recognition”, where I only look at the person’s face and try and understand who he/she most resembles, without any thought to the odds that that person is there. Frequentist recognition can lead to a large number of false positives, and after a few rounds of embarrassment, you start giving up on recognizing, and many a possible reunion thus gets missed. Bayesian recognition, on the other hand, restricts your field of search (to the people who you give good odds of being there), prevents you from being distracted and increases your chances of making a good recognition.

So why did Bayesian recognition fail me when I met this former classmate back in 2006? The problem was her company. She had come for this Deep Purple concert with another friend of mine, who was my classmate in another school (and who I had been in touch with, and so easily recognized). I had no clue that these two were friends (it turned out they didn’t know each other that well – they had come there with a common friend). So when this girl (the one I didn’t recognize) popped up with “Hey SK! Do you remember me?” I assumed that she was someone I knew from the same school as the other girl I was meeting, and that wrongly restricted my search space. And so my mind was trying to map her to my friends from school 1, while she happened to be a friend from school 2. And my search returned a blank, and my legendary long-term memory skills were embarrassed.

I must mention here, though, that this is possibly the only time that my Bayesian recognition model has acted up, and refused to recognize someone I know. There have been 2-3 false positives, but this has been the only negative. And when you consider the sample size to be all the people I have recognized in different places, this is small indeed.

Oh, and after failing to recognize her then, I’ve kept in touch with this friend.

Handling Jesus

A few months back, perhaps during the football world cup, I had talked about the role of Jesus Navas in the Spanish attack. He would mostly be brought on as a “plan B”, mostly when the Spanish tiki-taka failed to break down the opposition defence.

And by hogging the right touchline, he would single-handedly offer a new line of attack, without taking too much away from the existing tiki-taka attack down the middle. Though quite under-rated, I think he had valuable contributions in the Spanish victory.

So I was thinking about the conditions that are essential for the success of Jesus Navas. And the primary condition, I thought, was the support of his team-mates. For example, when Xavi passed the ball right to Navas, he recognized fully well that there was little chance Navas would give it back to him. Xavi would recognize that Navas would play his own game, and all he had to do would be to perhaps send Sergio Ramos to support and get players in the box waiting for the cross.

It is to the credit of Xavi and the other members of Spanish “Plan A attack” that they recognized this and allowed Navas to play his own game whenever he came on. If they hadn’t, Navas would surely have never been as effective. In fact, he would have been a complete misfit and failure.

You might want to draw your own analogies from this but what I want to say is that when you have a guy in your team who does things differently, who is there to “provide a different angle to the attack”, you need to create conditions to facilitate his work. At the very least, you need to ensure that all members of the team recognize that this guy is different, and what they need to do to enable his success.

Talking about diversity and diversity policies is all fine, but to get the best out of the diversity policy, you need to create conditions to extract the best out of the “diversity hire”, in whatever context you choose to view this.

Meeting Sickness

Ok here is another reason I can think of as to why I didn’t do well in my consulting career. This is based on something I’ve been observing at office over the last week or two. I suffer from what I call as “meeting sickness”. The inability to work immediately after a meeting.

Rough empirical analysis tells me that for every meeting of N minutes that I sit through, I need another N minuts of downtime following it before I can get back to work. I don’t know why this happens to me. I don’t know if I’m having to spend too much willpower inside the meeting. Or if it is just that at meetings i get into high-intensity mode and that drains me out.

Whatever it is, in a typical consulting environment, you are expected to attend lots of meetings. If you work for a company that believes in the philosophy that all work is to be done at the client’s location (such as AT Kearney) then you have meetings throughout the day. it is only in between meetings that you get time to work, and usually the way the projects have been sold means that you can’t afford any downtime.

So that explains it. The other big reasons I’ve come up with for my failure in consulting environment are it requires a high degree of willpower which I dont’ have; and that it is an essentially fighter job. Maybe these are inter-related. Need to think on these lines and come up with something.

And if you didn’t like this post, my apologies. I had an extra-long meeting at work this evening from 4 to 7 (and had sat through three other meetings since morning). I fled immediately after, but I’m yet to recover.