Tag: decisions

Decision making and explainability

This is NOT a post about AI. It is, instead, about real intelligence.

My hypothesis is – the more you need to explain your decisions to people, the worse your decision-making gets.

Basically, instinct gets thrown out of the window.

Most of you who have worked in a company would have seen a few attempts at least of the company trying to be “more data driven”. Instead of making decisions on executives’ whims and will, they decide to set up a process with objective criteria. The decision is evaluated on each of these criteria and weights drawn up (if the weights are not known and you have a large number of known past decisions, this is just logistic regression). And then a sumproduct is computed, based on which the decision is made.

Now, I might be biased by the samples of this I’ve seen in real life (both in companies I’ve worked for and where I’ve been a consultant), but this kind of decision making usually results in the most atrocious decisions. And it is not even a problem with the criteria that are chosen or the weights each is assigned (so optimising this will get you nowhere). The problem is with the process.

As much as we would like to believe that the world is objective (and we are objective), we as humans are inherently instinctive and intuitive individuals (noticed that anupraas alankaar?). If we weren’t we wouldn’t have evolved as much as we have, since a very large part of the decisions we need to make need to be made quickly (running from a lion when you see one, for example, or braking when the car in front of you also brakes suddenly).

Quick decisions can never be made based on first principles – to be good at that, you need to have internalised the domain and the heuristics sufficiently, so that you know what to do.

I have this theory on why I didn’t do well in traditional strategy consulting (it was the first career I explored, and I left my job in three months) – it demanded way too much structure, and I had faked my way in. For all the interview cases, I would intuitively come up with a solution and then retrofit a “framework”. N-1 of the companies I applied to had possibly seen through this. One didn’t and took me in, and I left very soon.

What I’m trying to say is – when you try to explain your decisions, you are trying to be analytical about something you have instinctively come to the conclusion about, and with the analysis being “a way to convince the other person that I didn’t use my intuition”.

So when a bunch of people come up with their own retrofits on how they make the decision, the “process” that you come up with is basically a bunch of junk. And when you try to follow the process the next time, you end up with a random result.

The other issue with explaining decisions is that you try to come up with explanations that sound plausible and inoffensive. For example, you might interview someone (in person) and decide you don’t want to work with them because they have bad breath (perfectly valid, in my opinion, if you need to work closely with them – no pun intended). However, if you have to document your reason for rejection, this sounds too rude. So you say something rubbish like “he is overqualified for the role”.

At other times, you clearly don’t like the person you have spoken to but are unable to put your rejection reason in a polite manner, so you just reverse your decision and fail to reject the person. If everyone else also thinks the same as you (didn’t like but couldn’t find a polite enough reason to give, so failed to reject), through the “Monte Carlo process”, this person you clearly didn’t like ends up getting hired.

Yet another time, you might decide to write an algorithm for your decision (ok I promised to not talk about AI here, but anyways). You look at all the past decisions everyone has made in this context (and the reasons for those), and based on that, you build an algorithm. But then, if all these decisions have been made intuitively and the people’s documented decisions only retrofits, you are basing your algorithm on rubbish data. And you will end up with a rubbish algorithm (or a “data driven process”).

Actually – this even applies to artificial intelligence, but that is for another day.

 

Big Data and Fast Frugal Trees

In his excellent podcast episode with EconTalk’s Russ Roberts, psychologist Gerd Gigerenzer introduces the concept of “fast and frugal trees“. When someone needs to make decisions quickly, Gigerenzer says, they don’t take into account a large number of factors, but instead rely on a small set of thumb rules.

The podcast itself is based on Gigerenzer’s 2009 book Gut Feelings. Based on how awesome the podcast was, I read the book, but found that it didn’t offer too much more than what the podcast itself had to offer.

Coming back to fast and frugal trees..

In recent times, ever since “big data” became a “thing” in the early 2010s, it is popular for companies to tout the complexity of their decision algorithms, and machine learning systems. An easy way for companies to display this complexity is to talk about the number of variables they take into account while making a decision.

For example, you can have “fin-tech” lenders who claim to use “thousands of data points” on their prospective customers’ histories to determine whether to give out a loan. A similar number of data points is used to evaluate resumes and determine if a candidate should be called for an interview.

With cheap data storage and compute power, it has become rather fashionable to “use all the data available” and build complex machine learning models (which aren’t that complex to build) for decisions that were earlier made by humans. The problem with this is that this can sometimes result in over-fitting (system learning something that it shouldn’t be learning) which can lead to disastrous predictive power.

In his podcast, Gigerenzer talks about fast and frugal trees, and says that humans in general don’t use too many data points to make their decisions. Instead, for each decision, they build a quick “fast and frugal tree” and make their decision based on their gut feelings about a small number of data points. What data points to use is determined primarily based on their experience (not cow-like experience), and can vary by person and situation.

The advantage of fast and frugal trees is that the model is simple, and so has little scope for overfitting. Moreover, as the name describes, the decision process is rather “fast”, and you don’t have to collect all possible data points before you make a decision. The problem with productionising the fast and frugal tree, however, is that each user’s decision making process is different, and about how we can learn that decision making process to make the most optimal decisions at a personalised level.

How you can learn someone’s decision-making process (when you’ve assumed it’s a fast and frugal tree) is not trivial, but if you can figure it out, then you can build significantly superior recommender systems.

If you’re Netflix, for example, you might figure that someone makes their movie choices based only on age of movie and its IMDB score. So their screen is customised to show just these two parameters. Someone else might be making their decisions based on who the lead actors are, and they need to be shown that information along with the recommendations.

Another book I read recently was Todd Rose’s The End of Average. The book makes the powerful point that nobody really is average, especially when you’re looking a large number of dimensions, so designing for average means you’re designing for nobody.

I imagine that is one reason why a lot of recommender systems (Netflix or Amazon or Tinder) fail is that they model for the average, building one massive machine learning system, rather than learning each person’s fast and frugal tree.

The latter isn’t easy, but if it can be done, it can result in a significantly superior user experience!

The Old Shoe Theory of Relationships

When our daughter was young, some friends saw uncanny resemblances between her and me, and remarked that “Karthik could have married an old shoe and still produced a child that looks like this”, essentially remarking that at least as far as looks were concerned, the wife hadn’t contributed much (Bambi eyes apart).

Over time, the daughter has shown certain other traits that make her seem rather similar to me. For example, she has the practice of sticking her tongue out when performing tasks that require some degree of concentration. She laughs like me. Screeches like me. And makes a “burl-burl” noise with her fingers and lips like I do (admittedly the last one is taught). I’ve already written a fuller list of ways in which the daughter is similar to me.

If you are single and looking to get into a long-term gene propagating relationship, you inevitably ask yourself the question of whether someone is “the one” for you. We have discussed this topic multiple times on this blog.

For example, we have discussed that as far as men are concerned, one thing they look for in potential partners is “consistent fuckability“. We have also discussed that whether someone is “the one” is not a symmetric question, and when you ask yourself the question, you either get “no” or “maybe” as an answer, implying that you need to use Monte Carlo algorithms. Being married to the Marriage Broker Auntie, I’m pretty sure I’ve discussed this topic on this blog several other times.

This is a tubelight post – at least two years too late (the “old shoe” comment came that long ago), but this is yet another framework you can use to determine if you want someone as your long-term gene-propagating partner. Basically you replace yourself by an old shoe.

In other words, assume that the genes that you will propagate along with this person will result in kids who look like them, talk like them, act like them, and rather than a “next best thing”, might just be a superior version of them. Ask yourself if you are okay with having a child who is like this, and who you will be proud of.

This is another Monte Carlo type question, but if the answer in this case is no (you may not be particularly proud of a progeny who is exactly like the person under consideration – for whatever reason), you don’t want to risk propagating genes with this person. In case the answer is yes – that you are willing to parent a child who is exactly like this counterparty, then you can seriously consider this long term relationship.

Again, this applies if and only if you’re looking for a gene propagating relationship. If that isn’t an issue (no pun intended), then you don’t need to worry about old shoes of any kind.

The Upside of ADHD

Attention Deficit Hyperactivity Disorder (ADHD), a condition that I have, isn’t all bad. In fact, it was a recognition of the qualities of this “disorder” that led me to stop my medication for it. I figured that I prefer the with-ADHD me to the without-ADHD me. I found the latter too boring, not creative enough, and unable to connect seemingly unrelated things – something that I’ve always taken a lot of pride in.

Yet another positive of ADHD, I realized yesterday, is that it allows you to lead a “markovian” life. It allows you to easily get rid of historical baggage while taking decisions, and makes you look forward by taking a decision based on the present. Yes, it can sometimes be a bit troublesome, as it prevents you from following long-drawn-out plans, but mostly it’s a good thing.

It makes you disregard that you’d taken a decision for some reason in the past. It makes it easy for you to disconnect from your earlier decisions, and look forward. It doesn’t allow you to be swayed by emotions – on account of some decision you’d taken in the past, and instead makes you rely on rational reasons.

There’s this documentary called “ADD and loving it”. Maybe I should watch that. And maybe I’ll have something to add to that.

The Importance of Discipline

I’ve never been a fan of discipline. I think it is a major constraint and hinders creativity, and puts too many walls within which you need to live your life. Despite constant exhortations by my father, I never wanted to join the army. Hell, I tried my best (successfully) in order to even avoid NCC when I was at IIT. I pride myself on being some sort of a free spirit who isn’t held back by any arbitrary rules that I create for myself to live my life by.

A really nice article that I read today, however, makes me think twice about this stand. So this article is about “decision fatigue” and is not very dissimilar to what I’d read a long time back (again in the NYT) about the Law of Conservation of Willpower. So this article talks about how every time you need to make a decision it consumes some part of your mental energy. Irrespective of the size of the decision that is to be made, there is some willpower that is lost, and that causes you to be suboptimal in your decision making as the day progresses.

The article really struck a chord with me, and I realize I’m also heavily prone to decision fatigue. Sometimes the smallest decisions take away so much energy from me that I simply put NED. And yeah, on a related note, I’ve got the wife upset innumerable times solely because of my indecisiveness, a part of which can be attributed to decision fatigue. I even remember not going to a wedding reception some three years back because I couldn’t decide which shirt to wear! And no, I’m not making this up.

So on that note, here’s where I think discipline has a part to play in life. By putting certain constraints on your life, you are reducing the number of decisions that you have to make. And that implies your willpower and mental energy will be reserved for those things where it’s really important that you decide carefully. By making a schedule for yourself, you are outsourcing to you-the-planner all the trivial decisions of your life. Yes, you might feel constrained at times. But it saves you so much energy by way of saving you from several trivial decisions.

Of course, feeling constrained can also affect your mental energy in a negative way, and prevent you from giving your best. Nevertheless, this decision fatigue thingy implies that discipline may not be all that bad. Or maybe I need to think about it some more.

Head, heart, phallus and arranged louvvu

In response to my arranged scissors series, my stalker has started her own thing called the “karabath series“. In the first (and so far, only) post in that series, she talks about this concept that she describes as “arranged louvvu”. It is a bit convoluted but the essence is that in “arranged louvvu” you don’t blindly get into it. Instead, you put on a rational approach to decide if the counterparty is best for you and if the cost of giving  up on all the other women in the world is covered by the joy this woman brings you, and then make sure that the counterparty satisfies all CMP constraints and only then, in a phased manner, do you fall in louvvu with the person.

It is a nice concept but unfortunately I think the way the stalker has explained it is extremely convoluted, and I think using the head-heart-phallus framework (Kunal Sawardekar, you can abuse me for this also) I can explain this a lot better. So basically the idea is that the phallus is the quickest to react, followed by the head and the heart takes the longest to react.

The way most louvvu happens, the way they show it in the movies is that phallus reacts first (it’s a purely biological reaction, so it’s quick and trustworthy). And then, quickly enough, the heart gets involved. And the thing with heart involvement is tthat it is an extremely illiquid investment – it is really tough to liquidate the “heart investment” without booking significant losses. And since the heart entered the scene before the head also gave its verdict, when the head finally comes into the picture, it finds the whole thing irrational, and thus it goes “love is blind”.

The typical arranged scissors process, however, doesn’t leave you with enough time for the heart to enter the process. And since relatives can’t help you with the phallus process (and since that’s anyway instantaneous) it’s the head that gets involved. It’s the head, the rational head that takes all the decisions there. The heart enters only later, in most cases after marriage.

So the funda is that the stalker is confusing “head involvement” for “arranged process”. What she calls as “arranged louvvu” is a case where one holds back the heart (yeah, it takes effort to do that after phallus has said yes) in order to allow for the head to take a rational decision, and then go ahead with “heart investment” only after head has said yes. This way, the head always has buy-in on any relationship that the heart has gotten into, so irrationality gets minimized to a large extent. And some of the problems of “pure louvvu” are hedged off.

Yeah, it takes effort. It takes effort to hold back the heart once the phallus has given go-ahead. But getting the right amount of head involved at the right time helps in preventing much disaster. It’s the logical way to go about things. “Arranged louvvu” is only a fancy name people give it. And now I don’t really know if the head-heart-phallus framework has made it any easier for you to understand this.