The missing middle in data science

Over a year back, when I had just moved to London and was job-hunting, I was getting frustrated by the fact that potential employers didn’t recognise my combination of skills of wrangling data and analysing businesses. A few saw me purely as a business guy, and most saw me purely as a data guy, trying to slot me into machine learning roles I was thoroughly unsuited for.

Around this time, I happened to mention to my wife about this lack of fit, and she had then remarked that the reason companies either want pure business people or pure data people is that you can’t scale a business with people with a unique combination of skills. “There are possibly very few people with your combination of skills”, she had said, and hence companies had gotten around the problem by getting some very good business people and some very good data people, and hope that they can add value together.

More recently, I was talking to her about some of the problems that she was dealing with at work, and recognised one of them as being similar to what I had solved for a client a few years ago. I quickly took her through the fundamentals of K-means clustering, and showed her how to implement it in R (and in the process, taught her the basics of R). As it had with my client many years ago, clustering did its magic, and the results were literally there to see, the business problem solved. My wife, however, was unimpressed. “This requires too much analytical work on my part”, she said, adding that “If I have to do with this level of analytical work, I won’t have enough time to execute my managerial duties”.

This made me think about the (yet unanswered) question of who should be solving this kind of a problem – to take a business problem, recognise it can be solved using data, figuring out the right technique to apply to it, and then communicating the results in a way that the business can easily understand. And this was a one-time problem, not something you would need to solve repeatedly, and so without the requirement to set up a pipeline and data engineering and IT infrastructure around it.

I admit this is just one data point (my wife), but based on observations from elsewhere, managers are usually loathe to get their hands dirty with data, beyond perhaps doing some basic MS Excel work. Data science specialists, on the other hand, will find it hard to quickly get intuition for a one-time problem, get data in a “dirty” manner, and then apply the right technique to solving it, and communicate the results in a business-friendly manner. Moreover, data scientists are highly likely to be involved in regular repeatable activities, making it an organisational nightmare to “lease” them for such one-time efforts.

This is what I call as the “missing middle problem” in data science. Problems whose solutions will without doubt add value to the business, but which most businesses are unable to address because of a lack of adequate skillset in solving the issue; and whose one-time nature makes it difficult for businesses to dedicate permanent resources to solve.

I guess so far this post has all the makings of a sales pitch, so let me turn it into one – this is precisely the kind of problem that my company Bespoke Data Insights is geared to solving. We specialise in solving problems that lie at the cusp of business and data. We provide end-to-end quantitative solutions for typically one-time business problems.

We come in, understand your business needs, and use a hypothesis-driven approach to model the problem in data terms. We select methods that in our opinion are best suited for the precise problem, not hesitating to build our own models if necessary (hence the Bespoke in the name). And finally, we synthesise the analysis in the form of recommendations that any business person can easily digest and action on.

So – if you’re facing a business problem where you think data might help, but don’t know how to proceed; or if you are curious about all this talk about AI and ML and data science and all that, and want to include it in your business; or you want your business managers to figure out how to use the data  teams better, hire us.

Profit and politics

Earlier today I came across this article about data scientists on LinkedIn that I agreed with so much that I started wondering if it was simply a case of confirmation bias.

A few sentences (possibly taken out of context) from there that I agree with:

  • Many large companies have fallen into the trap that you need a PhD to do data science, you don’t.
  • There are some smart people who know a lot about a very narrow field, but data science is a very broad discipline. When these PhD’s are put in charge, they quickly find they are out of their league.
  • Often companies put a strong technical person in charge when they really need a strong business person in charge.
  •  I always found the academic world more political than the corporate world and when your drive is profits and customer satisfaction, that academic mindset is more of a liability than an asset.

Back to the topic, which is the last of these sentences. This is something I’ve intended to write for 5-6 years now, since the time I started off as an independent management consultant.

During the early days I took on assignments from both for-profit and not-for-profit organisations, and soon it was very clear that I enjoyed working with for-profit organisations a lot more. It wasn’t about money – I was fairly careful in my negotiations to never underprice myself. It was more to do with processes, and interactions.

The thing in for-profit companies is that objectives are clear. While not everyone in the company has an incentive to increase the bottom-line, it is not hard to understand what they want based on what they do.

For example, in most cases a sales manager optimises for maximum sales. Financial controllers want to keep a check on costs. And so on. So as part of a consulting assignment, it’s rather easy to know who wants what, and how you should pitch your solution to different people in order to get buy-in.

With a not-for-profit it’s not that clear. While each person may have their own metrics and objectives, because the company is not for profit, these objectives and metrics need not be everything they’re optimising for.

Moreover, in the not for profit world, the lack of money or profit as an objective means you cannot differentiate yourself with efficiency or quantity. Take the example of an organisation which, for whatever reason, gets to advice a ministry on a particular subject, and does so without a fee or only for a nominal fee.

How can a competitor who possibly has a better solution to the same problem “displace” the original organisation? In the business world, this can be done by showing superior metrics and efficiency and offering to do the job at a lower cost and stuff like that. In the not-for-profit setup, you can’t differentiate on things like cost or efficiency, so the only thing you can do is to somehow provide your services in parallel and hope that the client gets it.

And then there is access. If you’re a not-for-profit consultant who has a juicy project, it is in your interest to become a gatekeeper and prevent other potential consultants from getting the same kind of access you have – for you never know if someone else who might get access through you might end up elbowing you out.

Freelancing and transaction costs

In the six years of running my own consulting business, I’d forgotten about an essential part that you need to endure as part of a job – piecemeal work. It is fairly often when you’re working for someone else that you get work that is so tiny or insignificant that you can hardly take ownership of it. The best strategy for dealing with it is to quietly get it over with and hope you won’t get such stuff again.

However, sometimes you can get caught in a rut of continuously getting this kind of work, and start wondering what you actually signed up for. And this is one thing I hadn’t expected to encounter when I got back to full time working earlier this year.

Thinking about why I never had to encounter such stuff during my consulting life, I realised there’s a fairly simple explanation – transaction costs.

Being a consultant is high transaction cost business. Every time you need to take on a new piece of work, you need to go through the charade of negotiating specifics with the client, pricing and drawing up a contract. All put together, the effort is not insignificant.

Moreover, in the line of work that I used to do, there was this massive overhead cost of understanding, cleaning and getting comfortable with the client’s data  – the effort involved in that meant that after a particular point in time I stopped taking work that wasn’t chunky enough. For a while I started refusing such work, but then got smarter and started pricing myself out of such work (though some clients were generous enough to meet that price to get their little tasks done – effectively I’d passed on the transaction costs to them).

The downside of this, of course, was that there was a fair amount of money I could have made taking up small works which I didn’t since the transaction cost was too high – this can be thought of as potential lost revenues. The upside was that whatever work I did was of high quality and (hopefully) made a big impact on the client’s business.

In the nature of the firm, Ronald Coase wrote that the purpose of the corporation was that transaction cost of dealing with co-workers can be eliminated. But then, I realise that sometimes this transaction cost can also be a good thing!

Oh, and obligatory plug here – my book Between the buyer and the seller deals with transaction costs, among other things. It’s available for sale (both in print and digital) on Amazon.


Letters to my wife

As I turned Thirty Three yesterday, my wife dug up some letters (emails to be precise) I’d written to her over the years and compiled them for me, urging me to create at “Project Thirty Four” (on the lines of my Project Thirty). What is pleasantly surprising is that I’ve actually managed to make a life plan for myself, and execute it (surprising considering I don’t consider myself to be too good a planner in general).

In February 2011, after having returned from a rather strenuous work trip to New York, this is what I had to say (emphasis added later, typos as in original):

For me steady state is when I’ll be doing lots of part-time jobs, consulting gigs, where I’m mostly owrking from home, getting out only to meet people, getting to meet a lot of people (somethign taht doesn’t happen in this job), having fun in the evenings and all that

I wrote this six months before I exited my last job, and it is interesting that it almost perfectly reflects my life nowadays (except for the “have fun in the evenings” bit, but that can be put down to being long distance).

I’ve just started a part time job. I have a couple of consulting gigs going. I write for a newspaper (and get paid for it). I mostly work from home. I’ve had one “general catch up” a day on an average (this data is from this Quantified Life sheet my wife set up for me).

A week later I had already started planning what I wanted to do next. Some excerpts from a letter I wrote in March 2011:

Ok so I plan to start a business. I don’t know when I’ll start, but I’m targeting sometime mid 2012.

I want to offer data consultancy services.

Basically companies will have shitloads of data that they can’t make sense of. They need someone who is well-versed in working with and looking at data, who can help them make sense of all that they’ve got. And I’m going to be that person.

Too many people think of data analysis as a science and just through at data all the analytical and statistical weapons that they’ve got. I believe that is the wrong approach and leads to spurious results that can be harmful for the client’s business.

However, I think it is an art. Making sense of data is like taming a pet dog. There is a way you communicate with it. There is a way you make it do tricks (give you the required information). And one needs to proceed slowly and cautiously in order to get the desired results.

I think of myself as a “semi-quant”. While I am well-versed in all the quantitative techniques in data analysis and financial modeling, I’m also deeply aware that using quantitative tools indiscriminately can lead to mismanagement of risks, which can be harmful to the client. I believe in limited and “sustainable” use of quantitative tools, so that it can lead without misleading.


My past experience with working with data is that data analysis can be disruptive. I don’t promise results that will be of particular liking for the client – but I promise that what I diagnose is good for the client’s business. When you dig through mountains of data, you are bound to get some bitter pills. I expect my clients to handle the bad news professionally and not shoot the messenger.

I don’t promise to find a “signal” in every data set that I’m given. There are chances that what I’m working with is pure noise, and in case I find that, I’ll make efforts to prove that to the client (I think that is also valuable information).

And these paragraphs, written a full year before I started out doing what I’m doing now, pretty much encapsulate what I’m doing now. Very little has changed over nearly five years! I feel rather proud of myself!

And a thousand thanks to my wife for picking out these emails I had sent her and showing me that I can work to a plan.

Now on to making Project Thirty Four, which I hope to publish by the end of today, and hope to execute by the end of next year.

Back to bachelorhood

Starting tonight I’ll be a bachelor once again. For the next nineteen months or so. No it’s not that I’m returning my post graduate diploma and hence getting this downgrade (it’s been a while since I cracked a bad joke here so I’m entitled). It’s that the wife is going away. To get herself an MBA (yes I know that after this she will be better qualified than me since she’ll be getting a proper MBA while i have a post graduate diploma only. Maybe I can retire soon? ).

She’ll be going off to Barcelona tonight. The original plan had me moving there too. But then classic old NED happened and I ended up not looking for a job or assignment there and since it’s not an inexpensive place to stay I’m staying back. Plan to visit her every once in a while. And even though tickets to Europe are prohibitively expensive I now have a place to go to in case I need a break.

But for that I need to first get myself a visa. I guess one of my chief tasks in the next few days will be to get this bit of business done. But then I have my own business.

Regulars on this blog might be aware that I haven’t had formal employment for close to three years now. I freelance as a quAnt consultant – helping companies figure out how to make use of the volumes of data they collect in improving their business decision making. It’s been doing quite okay but my plan is to use the next few months when I don’t have any domestic commitments to see if I can take it to the next level.

It might also be pertinent to mention here that the first bit if business I got for this particular venture was through this blog – the last time I put out a post like this one a long time reader who was looking for quant assistance left a comment here and that led to a rather fruitful assignment. Perhaps mentioning this here might result in a repeat?

Now that I’m blogging more than I used to in the recent past I’ll also be using these pages to keep you updated on the long distanceness. I’ve also noticed that since I last put the update on leaving twitter and Facebook that there’s some more activity here. Keep that flowing and I hope for some good conversations on the comment pages here.

Countercyclical business

I realize being a freelance management consultant is countercyclical business. For two years in succession, I’ve had a light March – both years I’ve ended up finishing projects in Jan/Feb. With March being the end of the Indian financial year, most companies are loathe to commit additional spending in March, and it is a bad time to start new projects!

This is counter-cyclical because most other businesses end up having a bumper March, since they have end-of-year targets, and with a short sales cycle, they push their salespersons hard to achieve this target in March!


In the wake of the passing of Ronald Coase, two incidents, both professional. The first was with an established company to whom I suggested a partnership – they are in a space where I don’t have much skill, but have access to companies who I would love to sell to, and they don’t have my skill and our skills are complementary. So I reached out to them (through common contacts) suggesting that we could work together. They came back to me saying they would love to work with me, but would want me to join them as an employee.

The second was an incoming lead. This was a rather small company doing something similar to what I’m doing but with bigger ideas. They want me to join this “innovation hub” they are trying to create. This is a loose federation they are creating including professionals from various fields. Nobody is obliged to work full time for the hub, but this gives people an opportunity to get together and work together on projects where their respective expertise can combine well.

As the more perceptive of you who would have read every Coase obituary in the last two weeks would have figured out, the piece of work that Coase is most well known for is about the theory of the firm. The question is rather simple – why should you and I get together and form a firm if we have to work together, if we can remain independent and just come together for projects. The answer lies in transaction costs.

The advantage of coming together as a firm is that you negotiate only once. Let us suppose you are a graphic designer and I’m a data scientist. If we decide to work together on a visualization project, how do we decide how much you get and how much I get? We will need to negotiate. Let’s say we negotiate and agree on a price. And complete a project and split the spoils. What would happen the next time we were to bid for a project? We will need to negotiate again on how we will share the spoils.

If on the other hand we were to form a partnership firm, then for every project that we do, our respective share is fixed! Thus we don’t have to negotiate before every single projects. Thus, firms exist so that you don’t have to repeatedly negotiate.

However, there is a downside to this. What if I form a firm with a graphic designer, and then we see a significant opportunity in projects that involve a lot of analysis but little visualization? In that case, I have no use of my partner, and would loathe to pay him his share of the profits. Or consider if I were to somehow become much better at my job, while my partner stagnates. There is little I can do, for we’ve been locked in into the financial arrangement.

These are only some of the complications that arise when you need to decide whether you want to become a firm. I just thought it is pertinent that I’m having some of these dilemmas (employee versus consultant versus partner versus member of federation) in the few weeks after Coase’s passing.