I can count my last three jobs (including my current one) as being results of my quest to become better at data science and modeling. Unfortunately, none of these jobs have turned out particularly well (this includes my current one). The problem has been that in all these jobs, data science has been tightly coupled with software engineering, and I suck at software engineering.

Let me stop for a moment and tell you that I don’t mind programming. In fact, I love programming. I love writing code that makes my job easier, and automates things, and gives me data in formats that I desire. But I hate software engineering. Of writing code within a particular system, or framework. Or adhering to standards that someone else sets for “good code”. Of following processes and making my code usable by some dumbfuck somewhere else who wouldn’t get it if I wrote it the way I wanted. As I’d mentioned earlier, I like coding for myself. I don’t like coding for someone else. And so I suck at software engineering.

Now I wonder if it’s possible at all to decouple data science from software engineering. My instinct tells me that it should be possible. That I need not write production-level code in order to turn my data-based insights into commercially viable form. Unfortunately, in my search around the corporatosphere thus far, I haven’t been able to find something of the sort.

Which makes me wonder if I should create my own niche, rather than hoping for someone else to create it for me.

Now, a full seven years after I graduated with my Bachelors, I’m not so sure. I think I should’ve gone for a “lighter” course, but then no one told me. So the thing with a B.Tech. in Computer Science and Engineering at IIT Madras is that it is extremely assignment incentive. Computer Science is that kind of a subject, there is very little you can learn in the classroom. The best way to learn stuff is by actually doing stuff, and “lab” is cheap (all you need is a bunch of computers) so most courses are filled with assignments. Probably from the fourth semester onwards, you spend most of your time doing assignments. Yes, you do end up getting good grades on an average, but you would’ve worked for it. And there’s no choice.

The thing with an Undergrad is that you are clueless. You have no clue what you’re interested in, what kind of a career you want to pursue, what excites you and the stuff. Yes, you have some information from school, from talking to seniors and stuff, but still it’s very difficult to KNOW when you are seventeen as to what you want to do in life. From this perspective, it is important for your to keep your options as open as they can be.

Unfortunately most universities in India don’t allow you to switch streams midway through your undergrad (most colleges are siloed into “arts” or “engineering” or “medicine” and the like). IIT Madras, in fact, is better in that respect since it allows you to choose a “minor” stream of study and courses in pure sciences and the humanities. But still, it is impossible for you to change your stream midway. So how do you signal to the market that you are actually interested in something else?

One way is by doing projects in areas that you think you are really interested in. Projects serve two purposes – first they allow you to do real work in the chosen field, and find out for yourself if it interests you. And if it does interest you, you have an automatic resume bullet point to pursue your career on that axis. Course-related projects are fine but since they’re forced, you have no way out, and they will be especially unpleasant if you happen to not like the course.

So why is CS@IITM a problem? Because it is so hectic, it doesn’t give you the time to pursue your other interests. It doesn’t offer you the kind of time that you need to study and take on projects in other subjects (yeah, it still offers you the 3 + 1 months of vacation per year, when you can do whatever you want, but then in the latter stages you’re so occupied with internships and course projects you’re better off having time during the term). So if you, like me, find out midway through the course that you would rather do something else, there is that much less time for you to explore around, study, and do projects in other subjects.

And there is no downside to joining a less hectic course. How hectic a course inherently is only sets a baseline. If you were to like the course, no one stops you from doing additional projects in the same subject. That way you get to do more of what you like, and get additional bullet points. All for the good, right?

After I graduated, IIT Madras reduced its credit requirement by one-twelfth. I don’t know how effective that has been in reducing the inherent workload of students but it’s a step in the right direction. Nevertheless, if you are going to get into college now, make sure you get into a less hectic course so that the cost of making a mistake in selection is not high.

I didn’t know why I’d to spend all my Thursday and Friday afternoons filing away at some piece of iron in the “fitting workshop”. I didn’t have the patience to draw three views of a random object in “engineering drawing”.

And I had the reputation of being one of the studdest programmers in my school. Apart from winning competitions here and there and doing well in acads, I had enormous respect from peers for my programming skills. Given that it was a “high-performance school” (which subjected its own 10th standard students to a test before admitting them to 11th) I guess this peer respect does carry some weight.

So, being good at math, and having the reputation of being a stud programmer, I didn’t know what I was doing studying “engineering”. I wanted to be a programmer, and I wanted to drop out and take up a job. My JEE rank counted almost as much as an IIT degree, I thought. I didn’t have the balls, and I continued.

In hindsight, I’m happy I didn’t drop out. By the end of my second year, I knew for sure that I DIDN’T want to be a programmer. While the theoretical aspects of Computer Science excited me (algo analysis and stuff), I had absolutely no patience for “systems”, or “computer engineering”. I was perhaps alone in my class in my love for Microsoft products (easy to use).

I realized then that I liked only the algorithmic aspect of programming, where one solves a (mostly math) problem and codes it up in a simple program. Huge complicated systems-intensive programming, making GUIs etc. didn’t inspire me at all.

Looking back, all that “major” (i.e. Computer Science and Engineering) stuff that I’ve learnt and internalized was learnt in my first two years of engineering. Of course several concepts that are part of CS&E are taught in the last two years, but I ended up not liking any of that.

Looking back, I do find it positive that I did all those “general engineering” courses. I do find it really positive that we had to do 12 compulsory credits in Humanities and Social Sciences, for that allowed me to discover what I was really interested in, and indirectly led me to doing my MBA.

I have only one regret. That I wasn’t able to switch streams sooner than I could. That IIT, being a one-dimensional technology oriented university, didn’t allow me to transfer credits to a course that I would’ve liked better, simply because it offered undergrad courses only in engineering.

There was a humanities department, where I discovered what I was interested in, but unfortunately it was a “minor” department. It’s been partly rectified now, with the setting up of integrated MA courses, in Economics, etc. (if that course existed back when I was studying, there’s a good chance I’d've transferred to it from CS&E). But it’s not enough.

Kids at 17 have no clue what they want to do. What we need are flexible full-scale universities, which allow you to switch from any branch to any other branch after two years of reasonably generalized study (the earlier branch can then contribute to “minor” credits). We need to stop putting our colleges in silos such as “engineering”, “arts and science”, etc. Only then would our universities be truly world class, even from an undergraduate point of view.

And looking back, I’m really happy I didn’t drop out.

It was my love for coding that meant I fought with my parents (who wanted me to study Electrical) and decided to study Computer Science at IIT Madras. And then I lost it. Somewhere along the way. I didn’t enjoy coding any more. Soon, I began to hate coding. I would love coding when I would write the odd program in “pure” C, or when I would participate in contests such as BITWise. But I’d completely lost it.

So over the last six to seven years (after I graduated from IIT) there have been occasions when I have thought I’ve regained my coding mojo, only to lose it again very soon. I’m still very proud of that Excel+VBA model that I had written in the very first week of my third job. But a couple of months later, I was hating coding again. And so it was while debugging a complicated piece of code at work this morning that I realize why I have this love-hate relationship with coding.

It’s simple – basically I hate coding for others. I hate writing code that others will read or use. I don’t mind writing code that others would use as a black box, of course. But I think writing code that others will read or use puts too many constraints on the way you code. My instinct is always to stop doing something when I’m personally satisfied with it, and with code it seems like I’m satisfied sooner than others would be satisfied with my code.

At a fundamental level, I like coding and I think I’m pretty good at it, so it isn’t something I want to give up. But then the formal processes and endless testing involved with writing code for others really kills joy (as does GUI, and Java). Code saves a lot of time, and helps “studdize” what might be otherwise fighter work, so I like doing it.

In an ideal world, I would be writing code that I would alone be using, AND profiting from it (I never intend to sell code; I intend to sell the results of the said code, however; that would mean no one else would read/use my code per se, so I can write it the way I want). Hopefully I’ll get there, sometime.

Let us suppose Alice’s liking for Bob is a continuous variable between 0 and 1. However, Alice never directly states to Bob how much she likes him. Instead, Bob will have to infer this based on Alice’s actions. Based on a current state of the relationship (also defined as a continuous variable between 0 and 1) and on Alice’s latest action, Bob infers how much Alice likes him. There are a variety of reasons why Bob might want to use this information, but let us not go into that now. I’m sure you can come up with quite a few yourself.

Now, my hypothesis is that the relationship state (which takes into account all past information regarding Alice’s and Bob’s actions towards each other) can be modelled as an exponentially-smoothed variable of the time series of Alice’s historical liking for Bob. To restate in English, consider the last few occasions when Alice and Bob have interacted, and consider the data of how much Alice actually liked Bob during each of these rounds. What I say is that the “current level” that I defined in the earlier paragraph can be estimated using this data on how much Alice liked Bob in the last few interactions. By exponentially smoothed, I mean that the last interaction has greater weight than the one prior to that which has more weight than the interaction three steps back, and so on.

So essentially Alice’s liking for Bob cannot be determined by her latest action alone. You use the latest action in conjunction with her last few actions in order to determine how much she likes Bob. If you think of inter-personal romantic relationships, I suppose you can appreciate this better.

Now that you’ve taken a moment to think about how my above hypotheses work in the context of human romantic relationships, and having convinced yourself that this is the right model, we can move on. To simplify all that I’ve said so far, the same action by Alice towards Bob can indicate several different things about how much she now likes him. For example, Alice putting her arm around Bob’s waist when they hardly knew each other meant a completely different thing from her putting her arm around his waist now that they have been married for six months. I suppose you get the drift.

So what I’m trying to imply here is that if you are going through a rough patch, you will need to try harder and send stronger signals. When the last few interactions haven’t gone well, the “state function of the relationship” (defined a few paragraphs above) will be at a generally low level, and the other party will have a tendency to under-guess your liking for them based on your greatest actions. What might normally be seen as a statement of immense love might be seen as an apology of an apology when things aren’t so good.

It is just like an economy in depression. If the government sits back claiming business-as-usual it is likely that the economy might just get worse. What the economy needs in terms of depression is a strong Keynesian stimulus. It is similar with bilateral relationships. When the value function is low, and the relationship is effectively going through a depression, you need to give it a strong stimulus. When Alice and Bob’s state function is low, Alice will have to do something really really extraordinary to Bob in order to send out a message that she really likes him.

And just one round of Keynesian stimulus is unlikely to save the economy. There is a danger that given the low state function, the economy might fall back into depression. Similarly when you are trying to get a relationship out of a “depressed” state, you will need to do something awesome in the next few rounds of interaction in order to make an impact. If you, like Little Bo Peep, decide that “leave ‘em alone, they will come home”, you are in danger of becoming like Japan in the 90s when absolute stagnation happened.

As I had explained in a post a long time ago, this whole business of louvvu and marriage and all such follows a “Monte Carlo approach“. When you ask yourself the question “Do I want a long-term gene-propagating relationship with her?” , the answer is one of “No” or “Maybe”. Irrespective of how decisive you are, or how perceptive you are, it is impossible for you to answer that question with a “Yes” with 100% confidence.

Now, in Computer Science, the way this is tackled is by running the algorithm a large number of times. If you run the algo several times, and the answer is “Maybe” in each iteration, then you can put an upper bound on the probability that the answer is “No”. And with high confidence (though not 100%) you can say “Probably yes”. This is reflected in louvvu also – you meet several times, implicitly evaluate each other on several counts, and keep asking yourselves this question. And when both of you have asked yourselves this question enough times, and both have gotten consistent maybes, you go ahead and marry (of course, there is the measurement aspect also that is involved).

Now, the deal with the arranged marriage market is that you aren’t allowed to have too many meetings. In fact, in the traditional model, the “darshan” lasts only for some 10-15 mins. In extreme cases it’s just a photo but let’s leave that out of the analysis. In modern times, people have been pushing to get more time, and to get more opportunities to run iterations of the algo. Even then, the number of iterations you are allowed is bounded, which puts an upper bound on the confidence with which you can say yes, and also gives fewer opportunity for “noes”.

Management is about finding a creative solution to a system of contradictory constraints
- Prof Ramnath Narayanswamy, IIMB

So one way to deal with this situation I’ve described is by what can be approximately called “pruning”. In each meeting, you will need to maximize the opportunity of detecting a “no”. Suppose that in a normal “louvvu date”, the probability of a “no” is 50% (random number pulled out of thin air). What you will need to do in order to maximize information out of an “arranged date” (yes, that concept exists now) is to raise this probability of a “no” to a higher number, say 60% (again pulled out of thing air).

If you can design your interaction so as to increase the probability of detecting a no, then you will be able to extract more information out of a limited number of meetings. When the a priori rejection rate per date is 50%, you will need at least 5 meetings with consistent “maybes” in order to say “yes” with a confidence of over 50% (I’m too lazy to explain the math here), and this is assuming that the information you gather in one particular iteration is independent of all information gathered in previous iterations.

(In fact, considering that the amount of incremental information gathered in each subsequent iteration is a decreasing function, the actual number of meetings required is much more)

Now, if you raise the a priori probability of rejection in one particular iteration to 60%, then you will need only 4 independent iterations in order to say “yes” with a confidence of over 95% (and this again is by assuming independence).

Ignore all the numbers I’ve put, none of them make sense. I’ve only given them to illustrate my point. The basic idea is that in an “arranged date”, you will need to design the interaction in order to “prune” as much as possible in one particular iteration. Yes, this same thing can be argued for normal louvvu also, but there I suppose the pleasure in the process compensates for larger number of iterations, and there is no external party putting constraints.

We were divided into groups of four, and each assignment used to have a “part A” and a “part B”, which carried equal weightage. The former was common to all groups, and would have fairly straightforward stuff – minor variants of what we discussed in class, etc. There would be several problems, and it was frankly a bit of a pain.

Part B gave a separate problem for each group, and this would be usually non-straightforward and required some bit of thinking. There was a good chance that the group never solved it at all, while on the other hand, at times it would hardly take time.

Looking back, I think in more than half the assignments, I ended up doing part B. The problems used to be fairly interesting, and I’d somehow end up solving them before the group even met to discuss the work. And given that all that was required to solve the problem was a moment of inspiration, the process of solving them were, in hindsight, interesting.

One problem was solved when I had taken one hostelmate’s new Bajaj Eliminator for a test ride. Another got solved when I was playing table tennis. Yet another while I was perched on the parapet reading the newspaper.

I also remember this particular incident. In the first assignment, we managed to find a fairly simple and intuitive solution to our part B problem. Now, two guys in my group were topper-types and fighter, and writing a simple and intuitive proof was against their ethos. They said that it would mean that we hadn’t shown much effort, and might result in our getting lesser marks. They finally put enough effort to convert the four line proof into some kind of formal mathematical notation which took four pages. I don’t know if anyone bothered to read that.

On the other hand, the basic assumption behind right-wing economics is that everyone is inherently a bad guy, and will work only for his own benefit. Hence, systems have to be devised so as to align a person’s selfish interests with the system’s interests.

To put it in other words, left-wing stuff is ideal. It assumes best case performance, or marginally below best case performance, from all players in the system. Similarly, in assuming that everyone has only a selfish motive, right-wing economics does what can be called a worst-case design.

Training in Computer Science inherently teaches you to think about the worst-case possibilities in everything.

In this context, isn’t it? surprising that so many people from a computer science background are leftist?

udupendra

pointed out last week that I realized that “Theory Lab” sounds somewhat self-contradictory.