Where Uncertainty is the killer: Jakarta Traffic Edition

So I’m currently in Jakarta. I got here on Friday evening, though we decamped to Yogyakarta for the weekend, and saw Prambanan and Borobudur. The wife is doing her mid-MBA internship at a company here, and since it had been a while since I’d met her, I came to visit her.

And since it had been 73 whole days since the last time we’d met, she decided to surprise me by receiving me at the airport. Except that she waited three and a half hours at the airport for me. An hour and quarter of that can be blamed on my flight from Kuala Lumpur to Jakarta being late. The rest of the time she spent waiting can be attributed to Jakarta’s traffic. No, really.

Yesterday evening, as soon as we got back from Yogyakarta, we went to visit a friend. Since this is Jakarta, notorious for its traffic, we landed up at his house straight from the airport. To everyone’s surprise, we took just forty minutes to get there, landing up much earlier than expected in the process.

So I’ve described two situations above which involved getting to one’s destination much ahead of schedule, and attributed both of them to Jakarta’s notorious traffic. And I’m serious about that. I might be extrapolating based on two data points (taking into the prior that Jakarta’s traffic is notorious), but I think I have the diagnosis.

The problem with Jakarta’s traffic is its volatility. Slow-moving and “bad” traffic can be okay if it can be predictable. For example, if it takes between an hour and half to hour and three-quarters most of the time to get to a place, one can easily plan for the uncertainty without the risk of having to wait it out for too long. Jakarta’s problem is that its traffic is extremely volatile, and the amount of time taken to go from one place to the other has a massive variance.

Which leads to massive planning problems. So on Friday evening, the wife’s colleague told her to leave for the airport at 7 pm to receive me (I was scheduled to land at 10:45 pm). The driver said they were being too conservative, and suggested they leave for the airport at 8, expecting to reach by 10:30. As it happened, she reached the airport at 8:45, even before my flight was scheduled to take off from KL! And she had to endure a long wait anyways. And then my flight got further delayed.

That the variance of traffic can be so high means that people stop planning for the worst case (or 95% confidence case), since that results in a lot of time being wasted at the destination (like for my wife on Friday). And so they plan for a more optimistic case (say average case), and they end up being late. And blame the traffic. And the traffic becomes notorious!

So the culprit is not the absolute amount of time it takes (which is anyway high, since Jakarta is a massive sprawling city), but the uncertainty, which plays havoc with people’s planning and messes with their minds. Yet another case of randomness being the culprit!

And with Jakarta being such a massive city and personal automobile (two or four wheeled) being the transport of choice, the traffic network here is rather “complex” (complex as in complex systems), and that automatically leads to wild variability. Not sure what (apart from massive rapid public transport investment) can be done to ease this.

Sigma and normal distributions

I’m in my way to the Bangalore airport now, north of hebbal flyover. It’s raining like crazy again today – the second time in a week it’s raining so bad.

I instinctively thought “today is an N sigma day in terms of rain in Bangalore” (where N is a large number). Then I immediately realized that such a statement would make sense only if rainfall in Bangalore were to follow a normal distribution!

When people normally say something is an N sigma event what they’re really trying to convey is that it is a very improbable event and the N is a measure of this improbability. The relationship between N and the improbability implied is given by the shape of the normal curve.

However when a quantity follow a distribution other than normal the relationship between the mean and standard deviation (sigma) and the implied probability breaks down and the number of sigmas will mean something totally different in terms of the implied improbability.

It is good practice, thus, to stop talking in terms of sigma and talk in terms of of odds. It’s better to say “a one in forty event” rather than saying “two sigma event” (I’m assuming a one tailed normal distribution here).

The broader point is that the normal distribution is too ingrained in people’s minds which leads then to assume all quantities follow a normal distribution – which is dangerous and needs to be discouraged strongly.

In this direction any small measure – like talking odds rather than in terms of sigma – will go a long way!

Accuracy of GDP Numbers

Earlier today on Twitter, RahulRG pointed out a research report by Credit Suisse analysts Neelkanth Mishra and Ravi Shankar which talks about India’s massive informal economy. The report says that by nature the informal economy cannot be measured, because of which our estimates of GDP may not be accurate. The analysts point out that every time we move to a new series of GDP (we last did so in 2004, and are likely to do so again shortly), there is an upward revision in the GDP for the preceding series, which they attribute to underestimation of the contribution of the informal sector.

While these numbers are likely to get fixed when we move to a new series, what I’m concerned about is what this uncertainty in GDP estimation means with respect to the GDP growth rate, since that is the one number that analysts of all hues track when trying to understand how the country is doing. For example, if you google around you will see analysts arguing about whether India’s GDP growth in the next quarter will be 4.7% or 4.8%. Before we settle to argue on such minutae, I argue, we first need to understand the possible uncertainty in GDP estimates.

In order to estimate the impact of uncertainty of the GDP calculation on uncertainty in GDP growth, I did what I know best – a simulation. For different levels of accuracy, I calculated the range that the actual GDP growth can take. The results are presented in the following table. The first column in the table refers to the accuracy of the GDP estimate at the 95% confidence level. That is, if the first column shows 1%, it means that if the GDP is estimated to be 100, the “true” value of the GDP will be between 99 and 101 95% of the time.

Error True GDP Growth Rate
5% 6% 7% 8%
0.05% 4.93-5.07 5.93-6.07 6.92-7.08 7.92-8.08
0.1% 4.85-5.15 5.85-6.15 6.85-7.15 7.85-8.15
0.2% 4.7-5.3 5.7-6.3 6.7-7.3 7.69-8.31
0.5% 4.26-5.74 5.26-6.75 6.25-7.76 7.24-8.77
1% 3.54-6.49 4.51-7.52 5.5-8.52 6.48-9.53
2% 2.09-8.03 3.03-9.02 4-10.05 4.98-11.13

Notice that even if the measurement of the actual GDP is accurate up to 0.05% (or 5 basis points), we can estimate the growth in GDP only up to an accuracy of 15 basis points! So arguing whether the GDP growth will be 4.7% or 4.8% is, in my opinion, moot! Unless our statisticians can say that the accuracy in measurement of the GDP is within 5 basis points that is!

PS: Also read Neelkanth Mishra’s excellent op-ed in the Indian Express on India’s informal economy.

Jobs and courtship

Jobs, unlike romantic relationships, don’t come with a courtship period. You basically go for a bunch of interviews and at the end of it both parties (you and the employer) have to decide whether it is going to be a good fit. Neither party has complete information – you don’t know what a typical day at the job is like, and your employer doesn’t know much about your working style. And so both of you are taking a risk. And there is a significant probability that you are actually a misfit and the “relationship” can go bad.

For the company it doesn’t matter so much if the odd job goes bad. They’ll usually have their recruitment algorithm such that the probability of a misfit employee is so low it won’t affect their attrition numbers. From the point of view of the employees, though, it can get tough. Every misfit you go through has to be explained at the next interview. You have a lot of misfits, and you’re deemed to be an unfaithful guy (like being called a “much-married man”). And makes it so tough for you to get another job that you are more likely to stumble into one where you’re a misfit once again!

Unfortunately, it is not practical for companies to hire interns. I mean, it is a successful recruitment strategy at the college-students level but not too many people are willing to get into the uncertainty of a non-going-concern job in the middle of their careers. This risk-aversion means that a lot of people have no option but to soldier on despite being gross misfits.

And then there are those that keep “divorcing” in an attempt to fit in, until they are deemed unemployable.

PS: In this regard, recruitments are like arranged marriage. You make a decision based on a handful of interviews in simulated conditions without actually getting to know each other. And speaking of arranged marriage, I reprise this post of mine from six years ago.

Relationships and the Prisoner’s Dilemma Part Deux

Those of you who either follow me on twitter or are my friends on GTalk will know that my earlier post on relationships and the prisoner’s dilemma got linked to from Cheap Talk, the only good Game Theory blog that I’m aware of. After I wrote that post, I had written to Jeffrey Ely and Sandeep Baliga of Cheap Talk, and Jeff decided to respond to my post.

It was an extremely proud moment for me and I spent about half a day just basking in the glory of having been linked from a blog that I follow and like. What made me prouder was the last line in Jeff’s post where he mentioned that my blog post had been part of his dinner conversation. I’m humbled.

So coming to the point of this post. Jeff, in his post, writes:

Some dimensions are easier to contract on.  It’s easy to commit to go out only on Tuesday nights.  However, text messages are impossible to count and the distortions due to overcompensation on these slippery-slope dimensions may turn out even worse than the original state of affairs.

I argue that it is precisely this kind of agreements that leads to too much engagement. The key, I argue, is to keep things loosely coupled and uncertain; and this, I say, doesn’t apply to only romantic relationships. I argue in favour of principles, as opposed to rules. Wherever the human mind is concerned, it is always better to leave room for uncertainty. Short term volatility decreases the chances of long-term shocks.

So if you contract to date only on Tuesday nights, and on a certain Thursday both of you get a sudden craving for each other. In a rule-based system, you’d have to wait till Tuesday to meet, and that would mean that you’d typically spend the next five days in high engagement, since you wouldn’t want to let go given the craving. There is also the chance that when you finally meet, there has been so much build-up that it leaves you unsettled.

The way to go about this is to not make rules and just make do with some simple principles regarding the engagement, and more importantly to keep things flexible. If you have a “I won’t call you when you’re at work” rule, and there is something you really need to say, this leads to wasted mind space since you’ll be holding this thought in the head till the other person is out of office, and thus give less for other things you need to do in that time.

You might ask me what principles one can use. I don’t know, and there are no rules governing principles. It is entirely to do with the parties involved and what they can agree upon. A simple principle might be “if I don’t reply to your text message it doesn’t mean I don’t love you”. You get the drift, I suppose. And the volatility, too. (ok I’m sorry about that one)

The mechanism design problem for scaling down that Jeff talks about is indeed interesting. His solution makes sense but it assumes the presence of a Trusted Third Party. Even if one were to find one such, and that person understands Binary Search techniques, it might take too much effort to find the level of interaction. I wonder if the solution to scaling down also is the Bilateral Nudge (will talk about this in another post).

Yede thumbi haaduvenu format is unfair

A month or so back, I had blogged about yede thumbi haaduvenu, a talent hunt show for young singers on ETV Kannada. I was full of praise for the event. About the format. About the way SPB comperes it. About the judging. Organization. And all that. I think I had written that post towards the end of last season. The new season has just begun. And I have a crib. It is not a minor one.

The format has changed. Last time around, it was a “normal knockout”, with round of 16, quarters, semis, final, etc. Each round would have four contestants of which two would progress to the next round and two would get eliminated. It was a nice and clean system – considering that any non-knockout format for a TV show isn’t a good idea.

Now, they have some sort of a serial knockout. Each episode has four kids, of which two get knocked out. The two who survive compete the following week, with two new people. Two out of these four qualify further. And so on.

This might have been an excellent format – if only the players were robots. If only the players didn’t have that human element called “form”. The format as it is right now is heavily biased in favour of kids who join the program in later rounds. Maybe they have been seeded there based on qualification placings. Nevertheless, it is wrong, and puts the kids who join early at too much of a disadvantage.

Kids who join early need to be at their top form for a larger number of episodes than those that join later. Sustaining an above-average performance over a larger stretch of time takes much more effort. You will also need to keep in mind that the pressure to perform in such events is huge. For the kids who join later, however, all it takes is for them to get lucky and produce terrific form for  a handful of episodes and they are through.

I suppose the producers of this event simply didnt’ realize that there is something called uncertainty. They would’ve looked at the format and said “this seems simpler for spectators and anyways the best will have to beat everyone else so this is ok”. I’m sure it the people who came up with this format are a bunch of fools who have no clue about either mathematics or about human tendency. I go back to one of my recent posts and call for the so-called “creative” or “qualitative” industry to cash in on the ibanking bust and take in some quants.

I’m reminded of one of the world chess championship (FIDE) cycles in the late 90s. They had a strenuous knockout tournament for a month to decide the challenger. And the winner of this tournament (Anand) then played the reigning champion Karpov who had been directly “seeded into the finals”. Anand got walloped by Karpov. And he had said something like “this is not fair. I have run the full marathon and in the last 100 meters this guy joins the race. what sort of a contest is this”

The current format of yede thumbi haaduvenu is no different. Now, if only the producers were to have some sense.