Tag: work

Record of my publicly available work

A few people who I’ve spoken to as part of my job hunt have asked to see some “detailed descriptions” of work that I’ve done. The other day, I put together an email with some of these descriptions. I thought it might make sense to “document” it in one place (and for me, the “obvious one place” is this blog). So here it is. As you might notice, this takes the form of an email.

I’m putting together links to some of the publicly available work that i’ve done.
1. Cricket
I have a model to evaluate and “tell the story of a cricket match”. This works for all limited overs games, and is based on a dynamic programming algorithm similar to the WASP. The basic idea is to estimate the odds of each team winning at the end of each ball, and then chart that out to come up with a “match story”.
And through some simple rules-based intelligence, the key periods in the game are marked out.
The model can also be used to evaluate the contributions of individual batsmen and bowlers towards their teams’ cause, and when aggregated across games and seasons, can be used to evaluate players’ overall contributions.
Here is a video where I explain the model and how to interpret it:
The algorithm runs live during a game. You can evaluate the latest T20 game here:
Here is a more interactive version , including a larger selection of matches going back in time.
Related to this is a cricket analytics newsletter I actively wrote during the World Cup last year. Most Indians might find this post from the newsletter interesting:
2. Covid-19
At the beginning of the pandemic (when we had just gone under a national lockdown), I had built a few agent based models to evaluate the risk associated with different kinds of commercial activities. They are described here.
Every morning, a script that I have written parses the day’s data from covid19india.org and puts out some graphs to my twitter account  This is a daily fully automated feature.
Here is another agent based model that I had built to model the impact of social distancing on covid-19.
A tweetstorm based on Bayes Theorem that I wrote during the pandemic went viral enough that I got invited to a prime time news show (I didn’t go).
3. Visualisations
I used to collect bad visualisations.
I also briefly wrote a newsletter analysing “good and bad visualisations”.
4. I have an “app” to predict which single malts you might like based on your existing likes. This blogpost explains the process behind (a predecessor of ) this model.
5. I had some fun with machine learning, using different techniques to see how they perform in terms of predicting different kinds of simple patterns.
6. I used to write a newsletter on “the art of data science”.
In addition to this, you can find my articles for Mint here. Also, this page on my website  as links to some anonymised case studies.

I guess that’s a lot? In any case, now I’m wondering if I did the right thing by choosing “skthewimp” as my Github username.

Core quants and desk quants on main street

The more perceptive of you might have realised that I’m in the job market.

Over the last one month, my search has mostly be “breadth first” (lots of exploratory conversations with lots of companies), and I’m only now starting to “go deep” into some of them. As part of this process, I need to send out a pitch to a company I’ve been in conversation with regarding what I can do for them.

So I’ve been thinking of how to craft my mandate while keeping in mind that they have an existing data science team. And while I was thinking about this problem, I realised that I can model it like how investment banks (at least one that I worked for) do – in terms of “core quants” and “desk quants”.

I have written about this on my blog before – most “data scientists” in industry are equivalent to what investment banks call “core quants”. They are usually highly technically accomplished people; in many cases they are people who were on an academic path that they left to turn to industry. They do very well in “researchy” environments.

They’re great at running long-gestation-period assignments, working on well defined technical problems and expressing their ideas in code. In general, though (I know I’m massively generalising), they are not particularly close to the business and struggle to deal with the ambiguities that business throws at them from time to time.

What I had mentioned in my earlier post is that “main street” (the American word for “general industry”) lacks “desk quants”. In investment banks, desk quants are attached to trading desks and work significantly closer to the business. They may work less on firmwide or long term strategic projects, but their strength is in blending the models and the markets, and building and making simple tweaks to models so that they remain relevant to the business.

And this is the sort of role in which I’m planning to pitch myself – to all potential employers. That while I’m rather comfortable technically, and all sorts of different modelling techniques, I’m not “deep into tech” and like to work close to the markets. I realise that this analogy will be lost on most people, so I need to figure out a better way of marketing myself. Any ideas will be appreciated.

Over the last month or so I’ve been fairly liberal and using my network to get introductions and references. The one thing I’ve struggled with there is how they describe me as. Most people end up describing me as a “data scientist”, and I’m not sure that’s an accurate description of what I do. Then again, it’s my responsibility to help them figure out how best to describe me. And that’s another thing I’m struggling in. “Desk quant” doesn’t translate well.

Coming back to life

On Sunday, I met a friend for coffee. In normal times that would be nothing extraordinary. What made this extraordinary was that this was the first time since the lockdown started that I was actually meeting a non-family member casually, for a long in-person conversation.

I’m so tired of the three pairs of shorts and five T-shirts that I’ve been wearing every day since the lockdown started that I actually decided to dress up that day. And bothered to take a photo at a signal on the way to meeting him.

We met at a coffee shop in Koramangala, from where we took away coffees and walked around the area for nearly an hour, talking. No handshakes. No other touches. Masks on for most of the time. And outdoors (I’m glad I live in Bangalore whose weather allows you to be outdoors most of the year). Only issue was that wearing a mask and walking and talking for an hour can tire you out a bit.

The next bit of resurrection happened yesterday when I had an in-person business meeting for the first time in three months. Parking the car near these people’s office was easier than usual (less business activity I guess?), though later I found that my windshield was full of bird shit (I had parked under a tree).

For the first time ever while going into this office, I got accosted by a security guard at the entrance, asking where I was headed, taking my temperature and offering me hand sanitiser. Being a first time, I was paranoid enough to use the umbrella I was carrying to operate the lift buttons, and my mask was always on.

There were no handshakes. The room was a bit stuffy and I wasn’t sure if they were using the AC, so I asked for the windows to be opened (later they turned on the AC saying it’s standard practice there nowadays). Again, no handshakes or anything. We kept our masks on for a long time. They offered water in a bottle which I didn’t touch for a long time.

Until one of them suggested we could order in dosas from a rather famous restaurant close to their office (and one that I absolutely love). The dosas presently arrived, and then all masks were off. For the next half hour as the dosas went down it was like we were back in “normal times” again, eating together and talking loudly without masks. I must say I missed it.

I took the stairs down to avoid touching the lift. Walked back to the car (and birdshit-laden windshield) and quickly used hand sanitiser. I hadn’t carried my laptop or notebook for the meeting, and I quickly made notes using the voice notes app of my phone.

Yes, in normal times, a lot of this might appear mundane. But given that we’re now sort of “coming back to life” after a long and brutal lockdown, a lot of this deserves documentation.

Oh, and I’m super happy to meet people now. Given a choice, I prefer outdoors. Write in if you want to meet me.

Meetings from home

For the last eight years, I’ve worked from home with occasional travel to clients’ offices. How occasional this travel has been has mostly depended on how far away the client is, and how insistent they are on seeing my face. Nevertheless, I’ve always made it a point to visit them for any important meetings, and do them in person.

Now, with the Covid-19 crisis, this hybrid model has broken down. Like most other people in the world, I work entirely from home nowadays, even for important meetings.

At the face of this, this seems like a good thing – for example, nowadays, however important a meeting is, the transaction cost is low. An hour long meeting means spending an hour for it (the time taken for prep is separate and hasn’t changed), and there’s no elaborate song-and-dance about it with travel and dressing up and all that.

While this seems far more efficient use of my time, I’m not sure I’m so happy about it. Essentially, I miss the sense of occasion. Now, an important meeting feels no different from an internal meeting with partners, or some trivial update.

Travel to and from an important meeting was a good time to mentally prepare for it, and then take stock of how it was gone. Now, until ten minutes before a meeting, I’m living my life as usual, and the natural boundaries that used to help me prep are also gone.

The other problem with remotely being there in large but important meetings is that it’s really easy to switch off. If you’re not the one who is doing a majority of the talking (or even the listening), it becomes incredibly hard to focus, and incredibly easy to get distracted elsewhere in the computer (it helps if your camera is switched off).

In a “real” physical meeting, however, large the gathering is, it is naturally easy for you to focus (and naturally more difficult to be distracted), and also easier to get involved in the meeting. An online meeting sometimes feels a bit too much like a group discussion, and without visual cues involved, it becomes really hard to butt in and make a point.

So once we are allowed to travel, and to meet, I’m pretty certain that I’ll start travelling a bit for work again. I’ll start with meetings in Bangalore (inter-city travel is likely to be painful for a very long time).

It might involve transaction cost, but a lot of the transaction cost gets recovered in terms of collateral benefits.

Ganesha Workflow

I have a problem with productivity. It’s because I follow what I call the “Ganesha Workflow”.

Basically there are times when I “get into flow”, and at those times I ideally want to just keep going, working ad infinitum, until I get really tired and lose focus. The problem, however, is that it is not so easy to “get into flow”. And this makes it really hard for me to plan life and schedule my day.

So where does Ganesha come into this? I realise that my workflow is similar to the story of how Ganesha wrote the Mahabharata.

As the story goes, Vyasa was looking for a scribe to write down the Mahabharata, which he knew was going to be a super-long epic. And he came across Ganesha, who agreed to write it all down under one condition – that if Vyasa ever stopped dictating, Ganesha would put his pen down and the rest of the epic would remain unwritten.

So Ganesha Workflow is basically the workflow where as long as you are going, you go strong, but the moment you have an interruption, it is really hard to pick up again. Putting it another way, when you are in Ganesha Workflow, context switches are really expensive.

This means the standard corporate process of drawing up a calendar and earmarking times of day for certain tasks doesn’t really work. One workaround I have made to accommodate my Ganesha Workflow is that I have “meeting days” – days that are filled with meetings and when I don’t do any other work. On other days I actively avoid meetings so that my workflow is not disturbed.

While this works a fair bit, I’m still not satisfied with how well I’m able to organise my work life. For one, having a small child means that the earlier process of hitting “Ganesha mode” at home doesn’t work any more – it’s impossible to prevent context switches on the child’s account. The other thing is that there is a lot more to coordinate with the wife in terms of daily household activities, which means things on the calendar every day. And those will provide an interruption whether I like it or not.

I’m wondering what else I can do to accommodate my “Ganesha working style” into “normal work and family life”. If you have any suggestions, please let me know!

Context switches and mental energy

Back in college, whenever I felt that my life needed to be “resurrected”, I used to start by cleaning up my room. Nowadays, like most other things in the world, this has moved to the virtual world as well. Since I can rely on the wife (:P) to keep my room “Pinky clean” all the time, resurrection of life nowadays begins with going off social media.

My latest resurrection started on Monday afternoon, when I logged off twitter and facebook and linkedin from all devices, and deleted the instagram app off my phone. My mind continues to wander, but one policy decision I’ve made is to both consume and contribute content only in the medium or long form.

Regular readers of this blog might notice that there’s consequently been a massive uptick of activity here – not spitting out little thoughts from time to time on twitter means that I consolidate them into more meaningful chunks and putting them here. What is interesting is that consumption of larger chunks of thought has also resulted in greater mindspace.

It’s simple – when you consume content in small chunks – tweets or instagram photos, for example, you need to switch contexts very often. One thought begins and ends with one tweet, and the next tweet is something completely different, necessitating a complete mental context switch. And, in hindsight, I think that is “expensive”.

While the constant stream of diverse thoughts is especially stimulating (and that is useful for someone like me who’s been diagnosed with ADHD), it comes with a huge mental cost of context switch. And that means less energy to do other things. It’s that simple, and I can’t believe I hadn’t thought of it so long!

I still continue to have my distractions (my ADHD mind won’t allow me to live without some). But they all happen to be longish content. There are a few blog posts (written by others) open in my browser window. My RSS feed reader is open on my browser for the first time since possibly my last twitter break. When in need of distraction, I read chunks of one of the articles that’s open (I read one article fully until I’ve finished it before moving on to the next). And then go back to my work.

While this provides me the necessary distraction, it also provides the distraction in one big chunk which doesn’t take away as much mental energy as reading twitter for the same amount of time would.

I’m thinking (though it may not be easy to implement) that once I finish this social media break, I’ll install apps on the iPad rather than having them on my phone or computer. Let’s see.

Housewife Careers

This is something I’ve been wanting to write about for a very long time, but have kept putting it off. The ultimate trigger for writing this is this article about women with children in Amazon asking for backup child care at work. Since this hits rather close home, this is a good enough trigger to write.

Quoting the article:

“Everyone wants to act really tough and pretend they don’t have human needs,” says Kristi Coulter, who worked in various roles at Amazon for almost 12 years and observed that many senior executives had stay-at-home wives.

(emphasis mine)

While this might be true of Amazon (though not necessarily for other large tech companies), it is true for other careers as well. The nature of the job means that it is impossible to function if you even have partial child-care responsibilities. And that implies that the only way you can do this job is if you have a spouse whose full time job is bringing up the kids.

Without loss of generality (considering that in most cases it’s the women who give up their careers for child-rearing), we can call these jobs “housewife jobs”.

Housewife jobs are jobs where you can do a good job if an only if you have a spouse who spends all her time taking care of the kids. 

The main feature (I would say it is a bug, but whatever) of such a job is usually long work hours that require you to “overlap both ways” – both leave home early in the morning and return late every night, implying that even if you have to drop your kid to day care, it is your spouse who has to do so. And as I’ve found from personal experience, it is simply not possible to work profitably when you have both child-dropping and child-picking-up duties on a single day (unless you have zero commute, like I’ve had for the last eight months).

Housewife jobs also involve lots of travel. Whether it is overnight or not doesn’t matter, since you are likely to be away early mornings and late evenings at least, and this means (once again) that the spouse has to pick up the slack.

Housewife jobs also involve a lot of pressure, which means that even when you are done with work and want to relax with the kids, you are unable to take your mind off work. So it turns out to be rather unprofitable time with the kids – so you might as well spend that working. Which again means the spouse picks up the slack.

Sometimes a job may not be inherently stressful or require long hours, but might be housewife because the company is led by a bunch of people with housewives (the article linked above claims this about Amazon). What this means is that when there is a sufficient number of (mostly) men in senior management who have housewives taking care of kids, their way of working percolates through the culture of the organisation.

These organisations are more likely to demand “facetime” (not the Apple variety). They are more likely to value input more than output (thus privileging fighter work?). And soon people without housewives get crowded out of such organisations, making it even more housewife organisations.

Finally, you may argue that I’ve used UK-style nurseries as the dominant child care mechanism in my post (these usually run 8-6), and that it might be possible to hedge the situations completely with 24/7 nannies or Singapore-style “helpers”. Now, even with full time child care, there are some emergencies that occur from time to time which require the presence of at least one parent. And it can’t be the same parent providing that presence all the time. So if one of the parents is in a “housewife job”, things don’t really work out.

I guess it is not hard to work out a list of jobs or sectors which are inherently “housewife”. Look at where people quit once they have kids. Look at where people quit once they get married. Look at jobs that are staffed by rolling legions of fresh graduates (if you don’t have a kid, you don’t need a housewife).

The scary realisation I’m coming to is that most jobs are housewife jobs, and it is really not easy being a DI(>=1)K household.

Just Plot It

One of my favourite work stories is from this job I did a long time ago. The task given to me was demand forecasting, and the variable I needed to forecast was so “micro” (this intersection that intersection the other) that forecasting was an absolute nightmare.

A side effect of this has been that I find it impossible to believe that it’s possible to forecast anything at all. Several (reasonably successful) forecasting assignments later, I still dread it when the client tells me that the project in question involves forecasting.

Another side effect is that the utter failure of standard textbook methods in that monster forecasting exercise all those years ago means that I find it impossible to believe that textbook methods work with “real life data”. Textbooks and college assignments are filled with problems that when “twisted” in a particular way easily unravel, like a well-tied tie knot. Industry data and problems are never as clean, and elegance doesn’t always work.

Anyway, coming back to the problem at hand, I had struggled for several months with this monster forecasting problem. Most of this time, I had been using one programming language that everyone else in the company used. The code was simultaneously being applied to lots of different sub-problems, so through the months of struggle I had never bothered to really “look at” the data.

I must have told this story before, when I spoke about why “data scientists” should learn MS Excel. For what I did next was to load the data onto a spreadsheet and start looking at it. And “looking at it” involved graphing it. And the solution, or the lack of it, lay right before my eyes. The data was so damn random that it was a wonder that anything had been forecast at all.

It was also a wonder that the people who had built the larger model (into which my forecasting piece was to plug in) had assumed that this data would be forecast-able at all (I mentioned this to the people who had built the model, and we’ll leave that story for another occasion).

In any case, looking at the data, by putting it in a visualisation, completely changed my perspective on how the problem needed to be tackled. And this has been a learning I haven’t let go of since – the first thing I do when presented with data is to graph it out, and visually inspect it. Any statistics (and any forecasting for sure) comes after that.

Yet, I find that a lot of people simply fail to appreciate the benefits of graphing. That it is not intuitive to do with most programming languages doesn’t help. Incredibly, even Python, a favoured tool of a lot of “data scientists”, doesn’t make graphing easy. Last year when I was forced to use it, I found that it was virtually impossible to create a PDF with lots of graphs – something that I do as a matter of routine when working on R (I subsequently figured out a (rather inelegant) hack the next time I was forced to use Python).

Maybe when you work on data that doesn’t have meaningful variables – such as images, for example – graphing doesn’t help (since a variable on its own has little information). But when the data remotely has some meaning – sales or production or clicks or words, graphing can be of immense help, and can give you massive insight on how to develop your model!

So go ahead, and plot it. And I won’t mind if you fail to thank me later!

Elegant and practical solutions

There are two ways in which you can tie a shoelace – one is the “ordinary method”, where you explicitly make the loops around both ends of the lace before tying together to form a bow. The other is the “elegant method” where you only make one loop explicitly, but tie with such great skill that the bow automatically gets formed.

I have never learnt to tie my shoelaces in the latter manner – I suspect my father didn’t know it either, because of which it wasn’t passed on to me. Metaphorically, however, I like to implement such solutions in other aspects.

Having been educated in mathematics, I’m a sucker for “elegant solutions”. I look down upon brute force solutions, which is why I might sometimes spend half an hour writing a script to accomplish a repetitive task that might have otherwise taken 15 minutes. Over the long run, I believe, this elegance will pay off, in terms of scaling easier.

And I suspect I’m not alone in this love for elegance. If the world were only about efficiency, brute force would prevail. That we appreciate things like poetry and music and art and what not means that there is some preference for elegance. And that extends to business solutions as well.

While going for elegance is a useful heuristic, sometimes it can lead to missing the woods for the trees (or missing the random forests for the decision trees if you may will). For there are situations that simply don’t, or won’t, scale, and where elegance will send you on a wild goose chase while a little fighter work will get the job done.

I got reminded of this sometime last week when my wife asked me for some Excel help in some work she was doing. Now, there was a recent article in WSJ which claimed that the “first rule of Microsoft Excel is that you shouldn’t let people know you’re good at it”. However, having taught a university course on spreadsheet modelling, there is no place to hide for me, and people keep coming to me for Excel help (though it helps I don’t work in an office).

So the problem wasn’t a simple one, and I dug around for about half an hour without a solution in sight. And then my wife happened to casually mention that this was a one-time thing. That she had to solve this problem once but didn’t expect to come across it again, so “a little manual work” won’t hurt.

And the problem was solved in two minutes – a minor variation of the requirement was only one formula away (did you know that the latest versions of Excel for Windows offer a “count distinct” function in pivot tables?). Five minutes of fighter work by the wife after that completely solved the problem.

Most data scientists (now that I’m not one!)  typically work in production environments, where the result of their analysis is expressed in code that is run on a repeated basis. This means that data scientists are typically tuned to finding elegant solutions since any manual intervention means that the code is not production-able and scalable.

This can mean finding complicated workarounds in order to “pull the bow of the shoelaces” in order to avoid that little bit of manual effort at the end, so that the whole thing can be automated. And these habits can extend to the occasional work that is not needed to be repeatable and scalable.

And so you have teams spending an inordinate amount of time finding elegant solutions for problems for which easy but non-scalable “solutions exist”.

Elegance is a hard quality to shake off, even when it only hinders you.

I’ll close with a fairytale – a deer looks at its reflection and admires its beautiful anchors and admonishes its own ugly legs. Lion arrives, the ugly legs help the deer run fast, but the beautiful antlers get stuck in a low tree, and the lion catches up.

 

Triangle marketing

This blog post is based more on how I have bought rather than how I have sold. The basic concept is that when you hear about a product or service from two or more independent sources, you are more likely to buy it.

The threshold varies by the kind of product you are looking at. When it is a low touch item like a book, two independent recommendations are enough. When it involves higher cost and has higher impact, like a phone, it might be five recommendations. For something life changing like a keto diet, it might be ten (I must mention I tried keto for half a day and gave up, not least because I figured I don’t really need it – I’m barely 3-4 kg overweight).

The important point to note is that the recommendations need to come from independent sources – if two people who you didn’t expect to have a similar taste in books were to recommend the same book, the second of these recommendations is likely to create an “aha moment” (ok I’m getting into consultant-speak now), and that is likely to drive a purchase (or at least trying a Kindle sample).

In some ways, exposure to the same product through independent sources is likely to create a feeling of a self-fulfilling prophecy. “Alice is also using this. Bob is also using this” will soon go into “everybody seems to be using it. I should also use it”.

So what does this mean to you if you are a seller? Basically you need to hit your target audience through various channels. I had mentioned in my post earlier this week about how branding creates a “position of strength“, and how direct sales is normally hard because it is done through a position of weakness.

The idea is that before you hit your audience with a direct sale, you need to “warm them up” with your brand, and you need to do this through various channels. Your brand needs to impact on your audience through multiple independent channels, so that it has become a self-fulfilling prophecy before you approach to make the sale.

What these precise channels are depends on your business and the product that you’re trying to sell, but the important thing is that they are independent. So for example, putting advertisements in various places won’t help since the target will treat all of them as coming from the same source.

Finally, where is the “triangle” in this marketing? It is in the idea that you complete the branding and sales by means of “triangulation”. You send out vectors in seemingly random directions trying to build your brand, and they will get reflected till a time when they intersect, or “triangulate”. Ok I know my maths here is messy ant not up to my usual standard, but I guess you know what I’m getting at!