Category: randomness

Religion and Probability

If only people were better at mathematics in general and probability in particular, we may not have had religion

Last month I was showing my mother-in-law the video of the meteor that fell in Russia causing much havoc, and soon the conversation drifted to why the meteor fell where it did. “It is simple mathematics that the meteor fell in Russia”, I declared, trying to show off my knowledge of geography and probability, arguing that Russia’s large landmass made it the most probable country for the meteor to fall in. My mother-in-law, however, wasn’t convinced. “It’s all god’s choice”, she said.

Recently I realized the fallacy in my argument. While it was probabilistically most likely that the meteor would fall in Russia than in any other country, there was no good scientific reason to explain why it fell at the exact place it did. It could have just as likely fallen in any other place. It was just a matter of chance that it fell where it did.

Falling meteors are not the only events in life that happen with a certain degree of randomness. There are way too many things that are beyond our control which happen when they happen and the way they happen for no good reason. And the kicker is that it all just doesn’t average out. Think about the meteor itself for example. A meteor falling is such a rare event that it is unlikely to happen (at least with this kind of impact) again in most people’s lifetimes. This can be quite confounding for most people.

Every time I’ve studied probability (be it in school or engineering college or business school), I’ve noticed that most people have much trouble understanding it. I might be generalizing based on my cohort but I don’t think it would be too much of a stretch to say that probability is not the easiest of subjects to grasp for most people. Which is a real tragedy given the amount of randomness that is a fixture in everyone’s lives.

Because of the randomness inherent in everyone’s lives, and because most of these random events don’t really average out in people’s lifetimes, people find the need to call upon an external entity to explain these events. And once the existence of one such entity is established, it is only natural to attribute every random event to the actions of this entity.

And then there is the oldest mistake in statistics – assuming that if two events happen simultaneously or one after another, one of the events is the cause for the other. (I’m writing this post while watching football) Back in 2008-09, the last time Liverpool FC presented a good challenge for the English Premier League, I noticed a pattern over a month where Liverpool won all the games that I happened to watch live (on TV) and either drew or lost the others. Being rather superstitious, I immediately came to the conclusion that my watching a game actually led to a Liverpool victory. And every time that didn’t happen (that 2-2 draw at Hull comes to mind) I would try to rationalize that by attributing it to a factor I had hitherto left out of “my model” (like I was seated on the wrong chair or that my phone was ringing when a goal went in or something).

So you have a number of events which happen the way they happen randomly, and for no particular reason. Then, you have pairs of events that for random reasons happen in conjunction with one another, and the human mind that doesn’t like un-explainable events quickly draws a conclusion that one led to the other. And then when the pattern breaks, the model gets extended in random directions.

Randomness leads you to believe in an external entity who is possibly choreographing the world. When enough of you believe in one such entity, you come up with a name for the entity, for example “God”. Then people come up with their own ways of appeasing this “God”, in the hope that it will lead to “God” choreographing events in their favour. Certain ways of appeasement happen simultaneously with events favourable to the people who appeased. These ways of appeasement are then recognized as legitimate methods to appease “God”. And everyone starts following them.

Of course, the experiment is not repeatable – for the results were purely random. So people carry out activities to appease “God” and yet experience events that are unfavourable to them. This is where model extension kicks in. Over time, certain ways of model extension have proved to be more convincing than others, the most common one (at least in India) being ‘”God” is doing this to me because he/she wants to test me”. Sometimes these model extensions also fail to convince. However, the person has so much faith in the model (it has after all been handed over to him/her by his/her ancestors, and a wrong model could definitely not have propagated?) that he/she is not willing to question the model, and tries instead to further extend it in another random direction.

In different parts of the world, different methods of appeasement to “God” happened in conjunction with events favourable to the appeasers, and so this led to different religions. Some people whose appeasements were correlated with favourable events had greater political power (or negotiation skills) than others, so the methods of appeasement favoured by the former grew dominant in that particular society. Over time, mostly due to political and military superiority, some of these methods of appeasement grew disproportionately, and others lost their way. And we had what are now known as “major religions”. I don’t need to continue this story.

So going back, it all once again boils down to the median man’s poor understanding of concepts of probability and randomness, and the desire to explain all possible events. Had human understanding of probability and randomness been superior, it is possible that religion didn’t exist at all!

Staggered surprises

When you have a number of things to surprise someone with, you can either flood them with that, or present it to them in a staggered manner. And based on recent experience with both forms, on both ends of the divide, I get the feeling that staggered surprises are superior and more effective than flooded surprises.

A year and half back, for my then girlfriend’s (now wife) birthday, I had got a bunch of things. There were clothes, food, a collage and even this laptop I’m writing this post on. And as soon as I entered the girlfriend’s house that day, I started producing these one by one. Before she could react to any of the gifts, I had produced another, and there was a flood. In hindsight, I thought the value of some of the things I’d got her were lost because I didn’t give her enough time to appreciate them while she was still surprised.

She played it differently at my birthday yesterday. Again, there was a bunch of things she had lined up. So at midnight yesterday, she says happy birthday and hands me a kurta. I try it out, and as soon as I’ve finished appreciating it (took a while) she makes me take it off, and gives me another. This way, over the course of the next ten minutes, she gives me five kurtas. And then a leather bag. And a box of tea. And some fancy paper to scribble on.

Giving gifts in a trickle, I think, works because of the expectations it sets. When Pinky produced the first Kurta, the natural thought in my head was, “oh she’s got me a kurta for my birthday”. I had expected one kurta. And when she slowly produced the next, I was surprised. You don’t generally expect someone to get you five kurtas, so each one she produced was met with a fair bit of surprise. The trickle had set my expectations low, and so the degree of surprise was high.

Pinky wasn’t done yet. She had solicited “happy birthday videos” from a number of my friends, from various stages of my life. Due to a personal tragedy (her grandfather passed away on Saturday) she hadn’t had time to put them together in a montage, but that helped her stagger-surprise me again. She first played videos from relatives, and after I had thought that was all to it, she played videos from friends. One by one. Not pushing expectations too high, and continually surprising me.

It was to play out similarly at the surprise party she had organized for me last night (after all the gifts and video messages, the last thing I had expected was a party). I had been told we’d be going out for dinner, when two of my oldest friends (I’ve known them for 25 years now) arrived. “Maybe she’s called my oldest friends to join us for dinner”, I thought. After a while they were followed by a friend from college who lives in the US now. I was truly shocked. He and his wife had dropped in while on their way to a wedding, I was told. I had no idea a party was on.

And then some quizzing friends appeared. And then some most recent colleagues (remember I don’t have any “current” colleagues). And Pinky, who had disappeared a while back, materialized with a cake. Soon enormous quantities of food appeared. I was already drinking by then and it was surreal. The best birthday ever, for sure. No, really! I don’t know if I would have been as happy had the surprises not been staggered.

PS: Ashwin and Vyshnavi responded to Pinky’s call for “happy birthday videos” with this one. It’s total kickass.

The Problem With a Common Engineering Entrance Test

… is correlation and concentration.

Like everything else, a student’s performance in a test can be divided into two – the predictive component (which can be explained based on preparation levels, general intelligence, ability to handle pressure, etc.) and the random component (which includes and is not limited to illness on the day of the exam, reaching the venue late leading to unsettlement, pure luck (or the lack of it) and so on).

Now, when you have a number of exams, what you expect is for a student’s “random component” to even out across these exams. If he outperforms his “predictive component” in one exam, you would expect that he would underperform in another exam. It’s like the “predictive component” of his performance is the expected “value” of his performance.

Thus, when you have a large number of entrance exams, it gives students the opportunity for their random components to even out, and take luck out to some extent from their college admission process. When you collapse all entrance exams into one, however, a student who happens to get a large negative “random component” on that given day is denied a second chance. Thus, the college admissions process will become much more of a crapshoot than it is now.

The other thing about uniform admission standards is why should every college have the same requirements for the students it wants to recruit? Having a common exam forces this upon colleges, unless they are allowed to change their weights allocated to different sections differently. If this doesn’t happen, it’ll only end up bringing all of the country’s education system to a uniform mediocrity.

In search of uncertainty

Back when I was in school, I was a math stud. At least people around me thought so. I knew I wanted to pursue a career in science, and that in part led me to taking science in class XI, and subsequently writing JEE which led to the path I ultimately took. Around the same time (when I was in high school), I started playing chess competitively. I was quite good at it, and I knew that with more effort I could make it big in the game. But then, that never happened, and given that I would fall sick after every tournament, I retired.

It was in 2002, I think, that I was introduced to contract bridge, and I took an instant liking for it. All the strategising and brainwork of chess would be involved once again, and I knew I’d get pretty good at this game, too. But there was one fundamental difference which made bridge so much more exciting – the starting position was randomized (I’m not making a case for Fischer Chess here, mind you). The randomization of starting positions meant that you could play an innumerable number of “hands” with the same set of people without ever getting bored. I simply loved it.

It was around that time that I started losing interest in math and other hard sciences. They had gotten to the point where they were too obscure, and boring, I thought, and that to make an impact in them, I wanted to move towards something less precise, and hard. That was probably what led me to do an MBA. And during the course of my MBA I discovered my interest in economics and social sciences, but am yet to do anything significant on that front, though, apart from the odd blog here or there.

I think what drove me from what I had thought is my topic of interest to what I think now it is is the nature of open problems. In hard sciences, where a lot of things are “known” it’s getting really hard to do anything of substance unless you get really deep in, into the territories of obscurity. In “fuzzy sciences”, on the other hand, nothing too much is “known”, and there will always be scope for doing good interesting work without it getting too obscure.

Similarly, finance, I thought, being a people-driven subject (the price of a stock is what a large set of people think its price is, there are no better models) will have lots of uncertainty, and scope to make assumptions, and thus scope to do good work without getting too obscure. But what I find is that given the influx of hard science grads in Wall Street over the last three decades, most of the large organizations are filled with people who simply choose to ignore the uncertainty and “interestingness” and instead try and solve deterministic problems based on models that they think completely represents the market.

And this has resulted in you having to do stuff that is really obscure and deep (like in the hard sciences) even in a non-deterministic field such as finance, simply because it’s populated by people from hard science background, and it takes way too much in order to go against the grain.

PS: Nice article by Tim Harford on why we can’t have any Da Vincis today. Broadly related to this, mostly on scientific research.

Standard Error in Survey Statistics

Over the last week or more, one of the topics of discussion in the pink papers has been the employment statistics that were recently published by the NSSO. Mint, which first carried the story, has now started a whole series on it, titled “The Great Jobs Debate” where people from both sides of the fence have been using the paper to argue their case as to why the data makes or doesn’t make sense.

The story started when Mint Editor and Columnist Anil Padmanabhan (who, along with Aditya Sinha (now at DNA) and Aditi Phadnis (of Business Standard), ranks among my favourite political commentators in India) pointed out that the number of jobs created during the first UPA government (2004-09) was about 1 million, which is far less than the number of jobs created during the preceding NDA government (~ 60 million). And this has led to hue and cry from all sections. Arguments include leftists who say that jobless growth is because of too much reforms, rightists saying we aren’t creating jobs because we haven’t had enough reform, and some other people saying there’s something wrong in the data. Chief Statistician TCA Anant, in his column published in the paper, tried to use some obscurities in the sub-levels of the survey to point out why the data makes sense.

In today’s column, Niranjan Rajadhyaksha points out that the way employment is counted in India is very different from the way it is in developed countries. In the latter, employers give statistics of their payroll to the statistics collection agency periodically. However, due to the presence of the large unorganized sector, this is not possible in India so we resort to “surveys”, for which the NSSO is the primary organization.

In a survey, to estimate a quantity across a large sample, we simply take a much smaller sample, which is small enough for us to rigorously measure this quantity. Then, we try and extrapolate the results to the large sample. The key thing in survey is “standard error”, which is a measure of error that the “observed statistic” is different from the “true statistic”. What intrigues me is that there is absolutely no mention of the standard error in any of the communication about this NSSO survey (again I’m relying on the papers here, haven’t seen the primary data).

Typically, when we measure something by means of a survey, the “true value” is usually expressed in terms of the “95% confidence range”. What we say is “with 95% probability, the true value of XXXX lies between Xmin and Xmax”. An alternate way of representation is “we think the value of XXXX is centred at Xmid with a standard error of Xse”. So in order to communicate numbers computed from a survey, it is necessary to give out two numbers. So what is the NSSO doing by reporting just one number (most likely the mid)?

Samples used by NSSO are usually very small. At least, they are very small compared to the overall population, which makes the standard error to be very large. Could it be that the standard error is not reported because it’s so large that the mean doesn’t make sense? And if the standard error is so large, why should we even use this data as a basis to formulate policy?

So here’s my verdict: the “estimated mean” of the employment as of 2009 is not very different from the “estimated mean” of the employment as of 2004. However, given that the sample sizes are small, the standard error will be large. So it is very possible that the true mean of employment as of 2009 is actually much higher than the true mean of 2004 (by the same argument, it could be the other way round, which points at something more grave). So I conclude that given the data we have here (assuming standard errors aren’t available), we have insufficient data to conclude anything about the job creation during the UPA1 government, and its policy implications.

Models

This is my first ever handwritten post. Wrote this using a Natraj 621 pencil in a notebook while involved in an otherwise painful activity for which I thankfully didn’t have to pay much attention to. I’m now typing it out verbatim from what I’d written. There might be inaccuracies because I have a lousy handwriting. I begin

People like models. People like models because it gives them a feeling of being in control. When you observe a completely random phenomenon, financial or otherwise, it causes a feeling of unease. You feel uncomfortable that there is something that is beyond the realm of your understanding, which is inherently uncontrollable. And so, in order to get a better handle of what is happening, you resort to a model.

The basic feature of models is that they need not be exact. They need not be precise. They are basically a broad representation of what is actually happening, in a form that is easily understood. As I explained above, the objective is to describe and understand something that we weren’t able to fundamentally comprehend.

All this is okay but the problem starts when we ignore the assumptions that were made while building the model, and instead treat the model as completely representative of the phenomenon it is supposed to represent. While this may allow us to build on these models using easily tractable and precise mathematics, what this leads to is that a lot of the information that went into the initial formulation is lost.

Mathematicians are known for their affinity towards precision and rigour. They like to have things precisely defined, and measurable. You are likely to find them going into a tizzy when faced with something “grey”, or something not precisely measurable. Faced with a problem, the first thing the mathematician will want to do is to define it precisely, and eliminate as much of the greyness as possible. What they ideally like is a model.

From the point of view of the mathematician, with his fondness for precision, it makes complete sense to assume that the model is precise and complete. This allows them to bringing all their beautiful math without dealing with ugly “greyness”. Actual phenomena are now irrelevant.The model reigns supreme.

Now you can imagine what happens when you put a bunch of mathematically minded people on this kind of a problem. And maybe even create an organization full of them. I guess it is not hard to guess what happens here – with a bunch of similar thinking people, their thinking becomes the orthodoxy. Their thinking becomes fact. Models reign supreme. The actual phenomenon becomes a four-letter word. And this kind of thinking gets propagated.

Soon the people fail to  see beyond the models. They refuse to accept that the phenomenon cannot obey their models. The model, they think, should drive the phenomenon, rather than the other way around. The tails wagging the dog, basically.

I’m not going into the specifics here, but this might give you an idea as to why the financial crisis happened. This might give you an insight into why obvious mistakes were made, even when the incentives were loaded in favour of the bankers getting it right. This might give you an insight as to why internal models in Moody’s even assumed that housing prices can never decrease.

I think there is a lot more that can be explained due to this love for models and ignorance of phenomena. I’ll leave them as an exercise to the reader.

Apart from commenting about the content of this post, I also want your feedback on how I write when I write with pencil-on-paper, rather than on a computer.

 


Something’s Itching

  • Recently I read this joke, not sure where, which said that the American and Indian middle classes are feeling sad that they cannot take part in a revolution, unlike their counterparts in Egypt, Tunisia, Libya, Yemen and other similar place. Instead, they can only vote
  • There needs to be some sort of an antitrust law for political parties. There is currently little to distinguish between the policies of various political parties. For example, all parties favour a greater role for the government (more govt => more opportunity to make money on the side => more corruption, etc.) .
  • Given the homogeneity in the political spectrum, there is little incentive to vote. This scoundrel may be only marginally better than that scoundrel, so why bother voting. So we have this large middle class which essentially removes itself from the political process (confession: I’m 28, and I’ve never voted. When my name’s in the list I’ve not been in town, and vice versa.)
  • Now this Anna Hazare tamasha has suddenly woken up people who never bothered to vote, and who are pained with excessive corruption. So they’re all jumping behind him, knowing that this gives them the opportunity to “do something” – something other than something as bland and simple as voting.
  • Supporters of Hazare care little about the implications of what they’re asking for. “Extra constitutional bodies”? “Eminent citizens”? Magsaysay award winners? Have you heard of the National Advisory Council? You seriously think you want more such institutions?
  • The Lok Ayukta isn’t as useless an institution as some critics have pointed out. But then again, this is highly personality-dependent. So you have a good person as a “lok pal”, you can get good results. But what if the government decides to appoint a compliant scoundrel there? Have the protesters considered that?
  • Basically when you design institutions, especially government institutions, you need to take care to build it in such a way that it’s not personality-dependent. Remember that you can have at TN Seshan as Election Commissioner, but you can also have a Navin Chawla.
  • So when you go out in droves and protest, you need to be careful what you ask for. Just make sure you understand that.

Useful links:

http://acorn.nationalinterest.in/2008/02/23/grammar-of-anarchy/

http://openthemagazine.com/article/voices/the-anna-hazare-show

http://calamur.org/gargi/2011/04/06/my-issues-with-the-proposed-jan-lok-pal-bill/

http://realitycheck.wordpress.com/2011/04/06/jan-lok-pal-caveat-emptor/

http://www.indianexpress.com/story-print/772773/

http://www.business-standard.com/india/news/the-hazare-hazard-/431045/

 

Addition to the Model Makers Oath

Paul Wilmott and Emanuel Derman, in an article in Business Week a couple of years back (at the height of the financial crisis) came up with a model-makers oath. It goes:

• I will remember that I didn’t make the world and that it doesn’t satisfy my equations.

• Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.

• I will never sacrifice reality for elegance without explaining why I have done so. Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.

• I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.

While I like this, and try to abide by it, I want to add another point to the oath:

As a quant, it is part of my responsibility that my fellow-quants don’t misuse quantitative models in finance and bring disrepute to my profession. It is my responsibility that I’ll put in my best efforts to be on the lookout for deviant behavour on the part of other quants, and try my best to ensure that they too adhere to these principles.

Go read the full article in the link above (by Wilmott and Derman). It’s a great read. And coming back to the additional point I’ve suggested here, I’m not sure I’ve drafted it concisely enough. Help in editing and making it more concise and precise is welcome.

 

Religion 1

I guess from my posts on religion you people know that I’m not the religious types. I don’t believe in rituals. I don’t believe that saying your prayers daily, or hourly, or monthly has any kind of impact on the orientation of the dice that life rolls out to you.

I believe in randomness. I believe that in every process there is a predictive component and a random component, and that you have no control over the latter. I believe that life can be approximated as a series of toin cosses, er. coin tosses, and some times the coins fall your way, and some times they don’t.

I was brought up in a strange household, in religious terms that is. My mother was crazily religious, spending an hour every day saying her prayers, and performing every conceivable ritual. My father was, for all practical purposes, atheist, and I never once saw him inside the prayer room in the house. I don’t ever remember having to make a conscious choice though, but I somehow ended up becoming like my father. Not believing in prayers or rituals (except for a brief period during my sophomore year at college), not believing that any actions of mine could bias the coin tosses of life.

A couple of years back I bought and read Richard Dawkins’ The God Delusion. I found the book extremely boring and hard to get through. And it really shocked me to read that people actually believe that praying can change the bias of the coins of life. Or that there exist people (most of Americal, shockingly) who think there was a “God” who created the earth, and that evolution doesn’t make sense.

Anyway the point now is that the missus thinks that I’m atheist because it’s the convenient thing to be, and because I haven’t made that extra effort in “finding God”. She things I’m not religious because I’m too lazy to say my prayers, and light incense, and all such. The irony here is that she herself isn’t the ritual types, instead choosing to introspect in quiet temples.

Just want to mention that you might find me write a lot more about religion over the next few days, or weeks, or months, as I try find my bearings and convince myself, and the missus, of my beliefs.

For starters, I’d say that if there exists a god, he does play dice.

When will people come?

So I was trying to estimate how many of my invitees will attend my wedding ceremony and how many will attend the reception (the former is at noon and the latter the same evening). While a large number of people have kindly RSVPd, not too many have really mentioned which event they’ll turn up for. So it’s my responsibility to somehow try and figure out how many will come when, so that the information can be appropriately relayed to the cooks.

Personally, if I’m attending the wedding of someone I don’t know too well, or a wedding I’m attending more out of obligation than out of the desire to be there, I prefer to go to the reception. It’s so much quicker – queue up, gift, wish, thulp, collect coconut, leave. The wedding leads to too much waiting, insufficient networking opportunity, having to wait for a seat in a “batch” for lunch, and the works.

Again, I hope that most people who are coming for my wedding are coming more because they want to attend rather than looking at it as an obligation. Actually I was thinking of a wedding invite as being an option – it gives you the option to attend the wedding, but you pay for it with the “cost” of the obligation to attend. In fact, over the last few days, I’ve felt extremely guilty while inviting people whose weddings I bunked (for one reason or another).

That digression aside, what upsets estimates for my wedding is that it’s on a Sunday, when more people will be inclined to come in the morning rather than at night. For one, they have the day off. Secondly, usually people like to spend Sunday evening at home, ironing clothes and the like, preparing for the grueling work week ahead.

And the fact that the venue is on the northern side of Bangalore, while most of my invitees live in the south (the fiancee and most of her invitees are in the north) makes me want to increase my “lunch” estimate and decrease the dinner estimate. And then the fact that I’m getting married on a seemingly “auspicious” day, when there are lots of functions all around, makes me wonder if I should discount the total attendance also.

After the wedding is over, I’m willing to anonymize and share the spreadsheet I’ve used for my estimates. Ok you might think I’m a geek but what I’ve done is to put an “attendance” probability for each event for each attendee, and then taken expected value to get my estimates. As I write this, I think I should take standard deviation also, and assume the law of large numbers (yes I’ve invited a large number of potential guests) in order to provide my in-laws (who are organizing the whole event) 95% confidence intervals for number of guests..

Anyways, I just hope that my (and my in-laws’) estimates are right and we won’t end up erring in either direction (shortage of food, or wastage) by too much in either direction. And the costs of the two (localized costs – as hosts, our costs of food shortage (in terms of reputation, etc.) is much higher than cost of wastage; though from global sustainability perspective it’s probably the other way round) have led our solution of the Newsboy Problem to be conservative in estimate.

And yesterday I was suggesting to my in-laws that after the wedding lunch, we can revise the estimates for dinner to M – X where M is the total number of guests we expect (counting double for people who we expect to attend both lunch and dinner) and X is the number of people who had lunch. It’s important, I think, to use as much information as possible in making decisions.