Analytics and complexity

I recently learnt that a number of people think that the more the number of variables you use in your model, the better your model is! What has surprised me is that I’ve met a lot of people who think so, and recommendations for simple models haven’t been taken too kindly.

The conversation usually goes like this

“so what variables have you considered for your analysis of ______ ?”
“Why don’t you consider D,E,F,… X,Y,Z also? These variables matter for these reasons. You should keep all of them and build a more complete model”
“Well I considered them but they were not significant so my model didn’t pick them up”
“No but I think your model is too simplistic if it uses only three variables”

This is a conversation i’ve had with so many people that i wonder what kind of conceptions people have about analytics. Now I wonder if this is because of the difference in the way I communicate compared to other “analytics professionals”.

When you do analytics, there are two ways to communicate – to simplify and to complicate (for lack of a better word). Based on my experience, what I find is that a majority of analytics professionals and modelers prefer to complicate – they talk about complicated statistical techniques they use for solving the problem (usually with fancy names) and bulldoze the counterparty into thinking they are indeed doing something hi-funda.

The other approach, followed by (in my opinion) a smaller number of people, is to simplify. You try and explain your model in simple terms that the counterparty will understand. So if your final model contains only three explanatory variables, you tell them that only three variables are used, and you show how each of these variables (and combinations thereof) contribute to the model. You draw analogies to models the counterparty can appreciate, and use that to explain.

Now, like analytics professionals can be divided into two kinds (as above), I think consumers of analytics can also be divided into two kinds. There are those that like to understand the model, and those that simply want to get into the insights. The former are better served by the complicating type analytics professionals, and the latter by the simplifying type. The other two combinations lead to disaster.

Like a good management consultant, I represent this problem using the following two-by-two:


 As a principle, I like to explain models in a simplified fashion, so that the consumer can completely understand it and use it in a way he sees appropriate. The more pragmatic among you, however, can take a guess on what type the consumer is and tweak your communication accordingly.



Valuing a flexible week

For the last couple of weeks my wife has taken time off from work, and given that I’m freelancing, we as a couple now have a flexible week. Yesterday, we went shopping. We were at the Bangalore Central store in JP Nagar, and for the first time in a really long time, were able to shop without bumping into fellow-shoppers every other moment. My wife didn’t have to wait endless hours in the queue just to get into the trial rooms (yeah, this happens at large format apparel stores on weekends). We shopped at Food Bazaar sub-store, and could take our time in deciding what to buy, without sharing aisles with other shoppers. The checkout counter was empty, ABSOLUTELY EMPTY, and we had an extremely peaceful experience there. It was an awesome day of shopping.

There are certain things that are done so much easier on weekdays than on weekends. Shops are significantly less crowded. If you have to get work done in government offices, you are better off going there on a weekday than on Saturday (when there are more consumers, and the employees are pissed off at the end of a long week). You don’t need to book cinema tickets hours before. Restaurants aren’t crowded. If you go for a day trip, you can expect your destination to not be flooded with other tourists. Of course, there are activities which are so much easily done on weekends rather than on weekdays – this involves anything that involves driving across the city in “peak traffic” hours.

So it’s clear that the “flexible working week” that I have provides some intangible value. Of course, since my wife doesn’t have a flexible week, we as a couple don’t always get to enjoy my flexible week, but leave that aside for now. What I’m trying to understand is the extra value that I”m getting thanks to my having a flexible week, and if I can put a number on it.

One way I can think of valuing my flexible week is in terms of optionality. I’ve listed down some of the advantages of doing certain things on a weekday. Maybe I can quantify the value of each of them? Maybe the value of the time I save by not standing in a queue at a checkout counter? The economic value of buying more and better clothes because I can shop peacefully? The additional value I get by having the picnic spot to myself rather than sharing it with a hundred others. The option value of being able to walk into a movie hall and buy tickets a minute before the show. And so on. And all this multiplied, of course, by the probability of my wanting to do each of these activities. Sounds right?

Of course, I’m talking about a flexible week here, and not about a week where you have weekly holiday on a weekday, like my wife had earlier this year. Thanks to some power supply issues, the local electricity distribution company mandated different weekly holidays for heavy industries in different parts of the city, thanks to which my wife had her weekly off on Wednesdays. And they were among the two most disorienting months I’ve been through. We were unable to do all those things that we would have normally done on weekends (and which are more advantageous to be done on weekends). I couldn’t do a full day of work on Saturday to compensate for not working on Wednesday. And I would try to work on Wednesday but wouldn’t be able to because my wife had her weekly off that day. It was absolutely mindfucking.

So yeah, maybe the next time someone asks me how much I”m making as a freelancer, I must include the “value of a flexible week” in the number I tell them!