The high cost of “relaxing” activities

So I have a problem. I can’t seem to enjoy movies any more. I’ve written about this before. My basic problem is that I end up double-guessing the plots of most movies that I watched (how many storylines are there anyways? According to Kurt Vonnegut, there are six story arcs).

So as I watch movies, I know exactly what is going to happen. And just continuing to watch the movie waiting for that to happen is simply a waste of time – it adds no information content to me.

The result is that I’m extremely selective about the kinds of movies I watch. Some genres, such as Westerns, work because even if the stories may be predictable, the execution and the manner of execution are not, and that makes for interesting watching.

Then, of course, there are directors who have built up a reputation of being “offbeat”, where you can expect that their movies don’t follow expected story arcs – their movies have enough information content to make them worth watching.

And most “classic” movies (take any of the IMDB Top 250, for example) have stories that are told in an extremely compelling fashion – sometimes you might know what happens, but the way things are built up implies that you don’t want to miss watching it happening.

Now, all this is fine, and something I’ve written about before. The point of this post is that while I feel this way about movies, my wife doesn’t feel the same way. She watches pretty much anything, even if the stories are utterly predictable.

For example, she’s watched at least a 100 Telugu movies (though, admittedly, during a particularly jobless stretch in her MBA when she was watching loads of movies, even she got bored of the predictability of Telugu movies and switched to Tamil instead!). She likes to watch endless reruns of 90s Kannada movies that now appear rather lame (to me). She especially loves chick flicks, which I think have excess redundancy built into them for a very specific reason.

I don’t have a problem with any of this! In fact, I’m damn happy that she has a single-player hobby that enables her to keep herself busy when she’s bored. The only little problem I have is that she believes it is romantic to watch movies together. She might sell video for Amazon for a living, but she surely is a fan of “netflix and chill” (more the literal meaning than the euphemistic one).

And that is a problem for me, since I find the vast majority of movies boring and predictable, and she thinks the kind of movies I like are “too serious” and “not suitable for watching together” – an assessment I don’t disagree with (though I did make her watch For a Few Dollars More with me a couple of months back).

I’d prefer to spend our time together not spent in talking doing other activities – reading, for example (reading offers significantly higher throughput than movies, and that, I think, is a result of formats of several lengths being prevalent – newspaper articles, longform articles, books, etc.). I’ve offered to watch movies with her on the condition that I read something at the same time – an offer that has been soundly rejected (and I understand her reasons for that).

And so we reach a deadlock, and it repeats every time when we have time and want to chill. She wants to watch movies together. I initially agree, and then back out when presented with a choice of movies to watch. Sometimes I put myself through it, thoroughly not enjoying the process. Other times, much to her disappointment, we end up not watching.

Clearly there are no winners in this game!

 

 

Curation, editing and predictability

One of my favourite lunchtime hobbies over the last one year has been watching chess videos. My favourite publishers in this regard are GM Daniel King and Mato Jelic. King is a far superior analyst and goes into more depth while analysing games, though Jelic has a far larger repertoire (King usually only analyses games the day they were played).

In some ways I might be biased towards Jelic because his analysis and focus are largely in line with my strengths back during my days as a competitive chess player. Deep opening analysis, attacking games, the occasional tactical flourish and so on. He has a particular fondness for the games of Mikhail Tal, showering praises on his (Tal’s) sometimes erratic and seemingly purposeless sacrifices.

Once you watch a few videos of Jelic, though, you realise that there is a formula to his commentary. At some point in the game, he announces that the game is in a “critical position” and asks the viewer to pause the video and guess the next move. And a few seconds of pause later, he proceeds to show the move and move on with the game.

While this is an interesting exercise the first few times around, after a few times I started seeing a pattern – Jelic has a penchant for attacking positions, and the moves following his “critical positions” are more often than not sacrifices. And once I figured this bit out, I started explicitly looking for sacrifices or tactical combination every time he asked me to pause, and that has made the exercise a lot less fun.

I’d mentioned on this blog a few weeks back about my problem with watching movies – in that I’m constantly trying to second-guess the rest of the movie based on the information provided thus far. And when a movie gets too predictable, it tends to lose my attention. And thinking about it, I think sometimes it’s about curation or editing that makes things too predictable.

To take an example, my wife and I have been watching Masterchef Australia this year (no spoilers, please!), and I remarked to her the other day that episodes have been too predictable – at the end of every contest, it seems rather easy to predict who might win or go down, and so there has been little element of surprise in the show.

My wife remarked that this was not due to the nature of the competition itself (which she said is as good as earlier editions), but due to the poor editing of the show – during each competition, there is a disproportional amount of time dedicated to showing the spectacularly good and spectacularly bad performances.

Consequently, just this information – on who the show’s editors have chosen to focus on for the particular episode – conveys a sufficient amount of information on each person’s performance, without even seeing what they’ve made! A more equitable distribution of footage across competitors, on the other hand, would do a better job of keeping the viewers guessing!

It is similar in the case of Jelic’s videos. There is a pattern to the game situation where he pauses, which biases the viewer in terms of guessing what the next move will be. In order to make the experience superior for his viewers, Jelic should mix it up a bit, occasionally showing slow Carlsen-like positions, and stopping games at positional “critical positions”, for example. That can make the pauses more interesting, and improve viewer experience!

What are other situations where bad editing effectively gives away the plot, and diminishes the experience?

The toin coss method

My mother had an interesting way to deal with dilemmas for which she had no solution – she would just toss a coin. She had only one rule for the “game” – that once she had decided to toss the coin, she would accept the “coin’s decision” and not think further about it.

This enabled her to get over many instances of decision fatigue – you have a dilemma only when you have two comparable choices, and won’t do too much worse by picking either.

So there’s this dilemma that’s hit me since this morning and facing trouble in making the decision (one of the choices has unquantifiable benefits so an objective cost-benefit analysis is not possible), I thought I should go back to my mother’s old method. And conveniently I see a coin lying on the table a metre away from me.

Thinking about it, tossing it and accepting its decision is acceptable only if I’m equally inclined to the two possibilities (assuming it’s a fair coin). Let’s say that I want to pick choice A three out of four times (“mixed strategies” can be rational in game theory), then I should toss the coin twice and pick A if either of the tosses returns a head. And so forth.

Considering how much decision fatigue I face (there have been times when I’ve actually turned around a dozen times after having taken only one step in each direction, not able to make up my mind), I should perhaps adopt this method. This makes me think that decision fatigue is also hereditary – and it was because she faced so much decision fatigue that my mother had to invent the coin toss method.

The title of this post is a tribute to an old colleague who would unfailingly say “toin coss” every time he intended to say “coin toss”, and tossing coins was an analogy he would make fairly often.

Good things do happen to those who wait

So once again I’ve taken myself off Twitter and Facebook. After a three-month sabbatical which ended a month back, I was back on these two social networks in a “limited basis” – I had not installed the apps on my phone and would use them exclusively from my computer. But as days went by, I realised I was getting addicted once again, and losing plenty of time just checking if someone had replied to any of the wisecracks I had put on some of those. So I’ve taken myself off once again, this time for at least one month.

This post is about the last of my wisecracks on facebook before I left it. A facebook friend had put an update that said “good things do happen to those who wait”. I was in a particularly snarky mood, and decided to call out the fallacy and left the comment below.

Good things

In hindsight I’m not sure if it was a great decision – perhaps something good had happened to the poor guy after a really long time, and he had decided to celebrate it by means of putting this cryptic message. And I, in my finite wisdom, had decided to prick his balloon by spouting gyaan. Just before I logged out of facebook this morning, though, I checked and found that he had liked my comment, though I don’t know what to make of it.

Earlier this year I had met an old friend for dinner, and as we finished and were walking back to the mall parking lot, he asked for my views on religion. I took a while to answer, for I hadn’t given thought to the topic for a while. And then it hit me, and I told him, “once I started appreciating that correlation doesn’t imply causation, it’s very hard for me to believe in religion”. Thinking about it now, a lot of other common practices, which go beyond religion, are tied to mistaking correlation for causation.

Take, for example, the subject of the post. “Good things happen to those who wait”, they say. It is basically intended as encouragement for people who don’t succeed in the first few attempts. What it doesn’t take care of it that the failures in the first few attempts might be “random”, or that even success when it does happen is the result of a random process.

Say, for example, you are trying to get a head upon the toss of a coin. You expect half a chance of a head the first time. It disappoints. You assume the second time the chances should be better, since it didn’t work out the first time (you don’t realise the events are independent), and are disappointed again. A few more tails and disappointment turns to disillusionment, and you start wondering if the coin is fair at all. Finally, when you get a head, you think it is divine retribution for having waited, and say that “good things happen to those who wait”.

In your happiness that you finally got a head, what you assume is that repeated failure on the first few counts actually push up your chance of getting your head, and that led to your success on the Nth attempt. What you fail to take into account is that there was an equal chance (assuming a fair coin) of getting a tail on the Nth attempt also (which you would have brushed off, since you were used to it).

In my comment above I’ve said “selection bias” but I’m not sure if that’s the right terminology – essentially when things go the way you want them to, you take notice and ascribe credit, but when things don’t go the way you want you don’t notice.

How many times have you heard people going through a happy experience saying they’re going through it “by God’s grace?”. How many times have you heard people curse God for not listening to their prayers when they’re going through a bad patch? Hardly? Instead, how many times have you heard people tell you that God is “testing them” when they’re going through a bad patch?

It’s the same concept of letting your priors (you see God as a good guy who will never harm you) affect the way you see a certain event. So in my friend’s case above, after a few “tails” he had convinced himself that “good things do happen to those who wait” and was waiting for a few more coin tosses until he finally sprang a head and announced it to the world!

Now I remember: I think it’s called confirmation bias.

Why being on time is a wonderful thing

This post is NOT about Indigo airlines, though I do fly them fairly frequently (approximately once a month). It is about the general culture of timeliness, and how it can help all of us save time and money.

If you and I decide to meet at say, 1 pm tomorrow, what time are you likely to turn up? There are two factors to consider here – you don’t want to be too late since that will create a bad impression in my mind, and you wouldn’t want that. You don’t want to turn up too early, either, for you don’t want to end up waiting for me. So when you plan your travel to the place we are meeting, you will first estimate what time I’m likely to show up and then plan to turn up such that you’ll maximize the probability of turning up between the time I’m expected to show up and five minutes earlier.

Notice how this can change depending upon the culture of timeliness. If you and I know each other, and I know that you are a punctual person and vice versa, we will both make an attempt to time our travel so that we maximize our probability of being there before 1 pm (the appointed time). What if I think that you are perennially late? The problem here is that I need to not only shift the “mean” of when I want to get to the place, but the variance also changes!

Notice that in case I know you are habitually late, I’m unlikely to know precisely when you’re going to arrive. Say I estimate based on our past record that you might turn up any time between ten and twenty minutes after the appointed time. How will I now plan to arrive so that I arrive between five and zero minutes of the time when I expect you to arrive? My travel time to get to the place already creates one level of uncertainty and to that I need to add another level of uncertainty in terms of when you are expected to arrive! Thus, these two sources of variation end up adding up and I will either be late (in case I’m okay wtih that) or end up spending more time just waiting for you!

Essentially, because I know that I cannot precisely determine when you are likely to get there, I assume a variance of when you are likely to get there, and that variance will add to the variance of my travel time and thus I’ll have to give myself a larger buffer so that I need to be on time while not waiting for too long!

This is similar to what people in quantitative finance call “market price of risk”. Let me illustrate that again using travel time as an example. In case 1, travel time from my office to yours has a mean of 40 minutes and a variance of 10 minutes (let us assume it is normally distributed). In case 2, travel time from my office to yours has a mean of 40 minutes (same as above) but a variance of only 5 minutes. Let us assume I want to be on time for the meeting at least 97.5% of the time. What time should I leave in each case?

In the first case, the one sided 97.5% confidence interval for my travel time is 40 + 2 * 10 = 60 minutes, or I expect to take no more than 60 minutes 97.5% of the time. In the second case, however, it is only 50 (40  + 2 * 5) minutes! In the first case, if I want to ensure a 97.5% chance of being on time for our 1 pm meeting, I’ll need to leave my office at 12 noon, while in the second case I can leave a full ten minutes later!

You need to notice here is that in both cases, the average travel time is the same. The only thing that has changed is the variance. In the first case, because the variance of the travel time is larger, I need to leave earlier! Leaving ten minutes earlier is essentially the price I have to pay because of the larger variance!

Similarly, when there is a variance in my estimate of when you will arrive for the meeting, it adds to the variance of my travel time, and the total variance I need to consider for when I need to leave goes up! In other words, simply because there is a variance in when you will arrive for the meeting,  i will have to leave earlier to compensate for your variance!

What if we had a culture of being on time? Then, I would know that with a very high probability you would be there on time for the meeting, and that would reduce my overall variance, and make it easier for me to also be on time for the meeting!

Essentially, a culture of being on time can save time for both of us – simply because it eliminates the variability of when we will end up arriving for the meeting, and this saved time is reason enough to build a culture of punctuality.

Yet you have people who schedule back-to-back meetings that invariably cascade and ruin their reputations of being on time, and thus inconvenience themselves and their counterparties!

Rare observations and observed distributions

Over the last four years, one of my most frequent commutes in Bangalore has been between Jayanagar and Rajajinagar – I travel between these two places once a week on an average. There are several routes one can take to get to Rajajinagar from Jayanagar, and one of them happens to be from the inside of Chamrajpet. However, I can count the number of times I’ve taken that route in the last four years on the fingers of one hand. This is because the first time I took that route I got stuck in a massive traffic jam.

Welcome to the world of real distributions and observed distributions. The basic concept is that if you observe a particular event rarely, the distribution you observe can be very different from the actual distribution. Take for example, the above example of driving through inner Chamrajpet. Let us say that the average time to drive through that particular road on a Saturday evening is 10 minutes. Let us say that 99% of the time on a Saturday evening, you take less than 15 minutes to drive through that road. In the remaining 1% of the time, you can take as much as an hour to drive through the road.

Now, if you are a regular commuter who drives through this road every Saturday evening, you will be aware of the distribution. You will be aware that 99% of the time you will take at most 15 minutes to get past, and base your routing decision based on that. When it takes an hour to drive past, you know that it is a rare event and discount it from your future calculations. If, however, you are an irregular commuter like me and happened to drive through that road on that one day when it took an hour you get past, you will assume that that is the average time it takes to get past! You are likely to mistake the rare event as the usual, and that can lead to suboptimal decisions in the future.

In his book The Black Swan, Nassim Nicholas Taleb talks about the inability of people to model for rare events. He says that the problem is that people underestimate the probability of rare events and fail to account for it in their models, leading to blow ups when they do occur. While I agree that is a problem, I contend that the opposite problem can also be not ignored. Sometimes you fail to recognize that what has happened to you is a rare event and thus end up with a wrong model.

Let me illustrate both problems with the same example. Think of a game where 99 times out of 100 you win a rupee. The rest of the time (i.e. 1%) you lose fifty rupees. Regular players of the game, who have “sampled” this enough will know the full distribution, and will take that into account when deciding on whether to play the game. Non-regular players, however, don’t have complete information.

Let us say there are a hundred cards. 99 of them have a +1 written on it, and the 100th has a -50. Let us suppose you pick ten cards. Ninety percent of the time, all ten cards you pick will be a “+1”, and you will conclude that all cards are “+1”. You will model for the game to give you a rupee each time you play. The other 10% of the time, however, you will draw nine +1s and one -50. You will then assume that the expected value of playing the game is Rs. -4 .1( (9 * 1  + 1 * (-50))/10 ). Notice that both times you are wrong in your inference!

So while it is important that you recognize black swans, it is also important that you don’t overestimate their probability. Always remember that if you are a rare observer, the distribution you observe may not reflect the real distribution.

Religion and Probability

If only people were better at mathematics in general and probability in particular, we may not have had religion

Last month I was showing my mother-in-law the video of the meteor that fell in Russia causing much havoc, and soon the conversation drifted to why the meteor fell where it did. “It is simple mathematics that the meteor fell in Russia”, I declared, trying to show off my knowledge of geography and probability, arguing that Russia’s large landmass made it the most probable country for the meteor to fall in. My mother-in-law, however, wasn’t convinced. “It’s all god’s choice”, she said.

Recently I realized the fallacy in my argument. While it was probabilistically most likely that the meteor would fall in Russia than in any other country, there was no good scientific reason to explain why it fell at the exact place it did. It could have just as likely fallen in any other place. It was just a matter of chance that it fell where it did.

Falling meteors are not the only events in life that happen with a certain degree of randomness. There are way too many things that are beyond our control which happen when they happen and the way they happen for no good reason. And the kicker is that it all just doesn’t average out. Think about the meteor itself for example. A meteor falling is such a rare event that it is unlikely to happen (at least with this kind of impact) again in most people’s lifetimes. This can be quite confounding for most people.

Every time I’ve studied probability (be it in school or engineering college or business school), I’ve noticed that most people have much trouble understanding it. I might be generalizing based on my cohort but I don’t think it would be too much of a stretch to say that probability is not the easiest of subjects to grasp for most people. Which is a real tragedy given the amount of randomness that is a fixture in everyone’s lives.

Because of the randomness inherent in everyone’s lives, and because most of these random events don’t really average out in people’s lifetimes, people find the need to call upon an external entity to explain these events. And once the existence of one such entity is established, it is only natural to attribute every random event to the actions of this entity.

And then there is the oldest mistake in statistics – assuming that if two events happen simultaneously or one after another, one of the events is the cause for the other. (I’m writing this post while watching football) Back in 2008-09, the last time Liverpool FC presented a good challenge for the English Premier League, I noticed a pattern over a month where Liverpool won all the games that I happened to watch live (on TV) and either drew or lost the others. Being rather superstitious, I immediately came to the conclusion that my watching a game actually led to a Liverpool victory. And every time that didn’t happen (that 2-2 draw at Hull comes to mind) I would try to rationalize that by attributing it to a factor I had hitherto left out of “my model” (like I was seated on the wrong chair or that my phone was ringing when a goal went in or something).

So you have a number of events which happen the way they happen randomly, and for no particular reason. Then, you have pairs of events that for random reasons happen in conjunction with one another, and the human mind that doesn’t like un-explainable events quickly draws a conclusion that one led to the other. And then when the pattern breaks, the model gets extended in random directions.

Randomness leads you to believe in an external entity who is possibly choreographing the world. When enough of you believe in one such entity, you come up with a name for the entity, for example “God”. Then people come up with their own ways of appeasing this “God”, in the hope that it will lead to “God” choreographing events in their favour. Certain ways of appeasement happen simultaneously with events favourable to the people who appeased. These ways of appeasement are then recognized as legitimate methods to appease “God”. And everyone starts following them.

Of course, the experiment is not repeatable – for the results were purely random. So people carry out activities to appease “God” and yet experience events that are unfavourable to them. This is where model extension kicks in. Over time, certain ways of model extension have proved to be more convincing than others, the most common one (at least in India) being ‘”God” is doing this to me because he/she wants to test me”. Sometimes these model extensions also fail to convince. However, the person has so much faith in the model (it has after all been handed over to him/her by his/her ancestors, and a wrong model could definitely not have propagated?) that he/she is not willing to question the model, and tries instead to further extend it in another random direction.

In different parts of the world, different methods of appeasement to “God” happened in conjunction with events favourable to the appeasers, and so this led to different religions. Some people whose appeasements were correlated with favourable events had greater political power (or negotiation skills) than others, so the methods of appeasement favoured by the former grew dominant in that particular society. Over time, mostly due to political and military superiority, some of these methods of appeasement grew disproportionately, and others lost their way. And we had what are now known as “major religions”. I don’t need to continue this story.

So going back, it all once again boils down to the median man’s poor understanding of concepts of probability and randomness, and the desire to explain all possible events. Had human understanding of probability and randomness been superior, it is possible that religion didn’t exist at all!