consulting – Pertinent Observations

Paying doctors

Back in 2011-12, when I was about to go freelance, a friend told me about a simple formula on how I should price my services. “Take your expected annual income and divide it by 1000. That will be your hourly rate”, he said. I followed this policy fairly well, with reasonable success (though I think I shortchanged myself in some situations by massively underestimating how long a task would take – but that story is for another day).

The longer term effect of that has been that every time I see someone’s hourly rate, I multiply it by 1000 to guess that person’s approximate annual income (the basis being that as a full time worker, you “bill” for 2000 hours a year. As a freelancer you have “50% utilisation” and so you work 1000 hours).

And one set of people who have fairly transparent hourly rates are doctors – you know the number of appointments they give per hour, and what you paid for that, and you can back calculate their annual income based on that. The interesting thing is, for most doctors I’ve seen, based on this metric, what they earn for their level of eduction and years of experience seems rather low.

“So how do doctors earn?”, I wonder. Why is it still a prized profession while you might have a much better life being an engineer, for example?

Now you should remember that consultations are only one income stream for doctors. Those that practice surgery as well have a more lucrative stream – the hourly rates for surgeries far exceeds hourly rates of consultation. And so surgeons make far more than what I impute from what I’ve paid them for a consultation.

One possible reason for this arbitrage is the way insurance deals are structured – at least in India, out patient care is seldom paid for by insurance. As a consequence, hospitals and doctors cross-subsidise consultations with surgeries. They are able to get away with higher rates for surgeries because insurers are bearing the cost. Consultations, where patients generally pay out of their own pockets, are far more elastic.

This, however, leads to a problem for doctors who don’t do surgeries. Psychiatrists, for example. If they have to make money solely through consultations, their hourly rate must be far higher than that of doctors who also do surgeries. But then, is the market willing to bear this cost?

Now, I’m getting into conspiracy theory mode. If the amount non-surgeon doctors make is limited (thanks to market dynamics), the only way they can make sure they earn a decent living is by limiting supply. Could this be one reason India is under-supplied in a lot of non-surgical doctors? Again this is pure pure speculation, and not based in any fact.

Continuing with conspiracy theories, even for doctors who are surgeons, the only way to make a certain income is to have a threshold on the ratio of surgeries to consultations. And if this ratio (surgeries / consultations) goes too low, the doctors’ income suffers. Again, hippocratic oath aside, do hospitals try to game this metric, based on the current incentives?

On a more serious note, this distortion in the hourly earnings for surgeries versus consultations is one reason that India is also undersupplied with good general practitioners (GPs). Because GPs don’t do surgeries (though the Indian system means they are all licensed to perform surgeries, to the best of my knowledge), their earning potential is naturally capped. So the better doctors don’t want to be GPs.

How can we fix this distortion? How can we make sure we have better GPs? Insurance cover for outpatient care is one thing, but I’m not sure it is the silver bullet I’ve been making it out to be (and it will come with its own set of market distortions).

This entire post is me shooting from my hip. So please feel free to correct me iff I’m wrong.

The missing middle in data science

Over a year back, when I had just moved to London and was job-hunting, I was getting frustrated by the fact that potential employers didn’t recognise my combination of skills of wrangling data and analysing businesses. A few saw me purely as a business guy, and most saw me purely as a data guy, trying to slot me into machine learning roles I was thoroughly unsuited for.

Around this time, I happened to mention to my wife about this lack of fit, and she had then remarked that the reason companies either want pure business people or pure data people is that you can’t scale a business with people with a unique combination of skills. “There are possibly very few people with your combination of skills”, she had said, and hence companies had gotten around the problem by getting some very good business people and some very good data people, and hope that they can add value together.

More recently, I was talking to her about some of the problems that she was dealing with at work, and recognised one of them as being similar to what I had solved for a client a few years ago. I quickly took her through the fundamentals of K-means clustering, and showed her how to implement it in R (and in the process, taught her the basics of R). As it had with my client many years ago, clustering did its magic, and the results were literally there to see, the business problem solved. My wife, however, was unimpressed. “This requires too much analytical work on my part”, she said, adding that “If I have to do with this level of analytical work, I won’t have enough time to execute my managerial duties”.

This made me think about the (yet unanswered) question of who should be solving this kind of a problem – to take a business problem, recognise it can be solved using data, figuring out the right technique to apply to it, and then communicating the results in a way that the business can easily understand. And this was a one-time problem, not something you would need to solve repeatedly, and so without the requirement to set up a pipeline and data engineering and IT infrastructure around it.

I admit this is just one data point (my wife), but based on observations from elsewhere, managers are usually loathe to get their hands dirty with data, beyond perhaps doing some basic MS Excel work. Data science specialists, on the other hand, will find it hard to quickly get intuition for a one-time problem, get data in a “dirty” manner, and then apply the right technique to solving it, and communicate the results in a business-friendly manner. Moreover, data scientists are highly likely to be involved in regular repeatable activities, making it an organisational nightmare to “lease” them for such one-time efforts.

This is what I call as the “missing middle problem” in data science. Problems whose solutions will without doubt add value to the business, but which most businesses are unable to address because of a lack of adequate skillset in solving the issue; and whose one-time nature makes it difficult for businesses to dedicate permanent resources to solve.

I guess so far this post has all the makings of a sales pitch, so let me turn it into one – this is precisely the kind of problem that my company Bespoke Data Insights is geared to solving. We specialise in solving problems that lie at the cusp of business and data. We provide end-to-end quantitative solutions for typically one-time business problems.

We come in, understand your business needs, and use a hypothesis-driven approach to model the problem in data terms. We select methods that in our opinion are best suited for the precise problem, not hesitating to build our own models if necessary (hence the Bespoke in the name). And finally, we synthesise the analysis in the form of recommendations that any business person can easily digest and action on.

So – if you’re facing a business problem where you think data might help, but don’t know how to proceed; or if you are curious about all this talk about AI and ML and data science and all that, and want to include it in your business; or you want your business managers to figure out how to use the data teams better, hire us.

Profit and politics

Earlier today I came across this article about data scientists on LinkedIn that I agreed with so much that I started wondering if it was simply a case of confirmation bias.

A few sentences (possibly taken out of context) from there that I agree with:

Many large companies have fallen into the trap that you need a PhD to do data science, you don’t.
There are some smart people who know a lot about a very narrow field, but data science is a very broad discipline. When these PhD’s are put in charge, they quickly find they are out of their league.
Often companies put a strong technical person in charge when they really need a strong business person in charge.
I always found the academic world more political than the corporate world and when your drive is profits and customer satisfaction, that academic mindset is more of a liability than an asset.

Back to the topic, which is the last of these sentences. This is something I’ve intended to write for 5-6 years now, since the time I started off as an independent management consultant.

During the early days I took on assignments from both for-profit and not-for-profit organisations, and soon it was very clear that I enjoyed working with for-profit organisations a lot more. It wasn’t about money – I was fairly careful in my negotiations to never underprice myself. It was more to do with processes, and interactions.

The thing in for-profit companies is that objectives are clear. While not everyone in the company has an incentive to increase the bottom-line, it is not hard to understand what they want based on what they do.

For example, in most cases a sales manager optimises for maximum sales. Financial controllers want to keep a check on costs. And so on. So as part of a consulting assignment, it’s rather easy to know who wants what, and how you should pitch your solution to different people in order to get buy-in.

With a not-for-profit it’s not that clear. While each person may have their own metrics and objectives, because the company is not for profit, these objectives and metrics need not be everything they’re optimising for.

Moreover, in the not for profit world, the lack of money or profit as an objective means you cannot differentiate yourself with efficiency or quantity. Take the example of an organisation which, for whatever reason, gets to advice a ministry on a particular subject, and does so without a fee or only for a nominal fee.

How can a competitor who possibly has a better solution to the same problem “displace” the original organisation? In the business world, this can be done by showing superior metrics and efficiency and offering to do the job at a lower cost and stuff like that. In the not-for-profit setup, you can’t differentiate on things like cost or efficiency, so the only thing you can do is to somehow provide your services in parallel and hope that the client gets it.

And then there is access. If you’re a not-for-profit consultant who has a juicy project, it is in your interest to become a gatekeeper and prevent other potential consultants from getting the same kind of access you have – for you never know if someone else who might get access through you might end up elbowing you out.

Freelancing and transaction costs

In the six years of running my own consulting business, I’d forgotten about an essential part that you need to endure as part of a job – piecemeal work. It is fairly often when you’re working for someone else that you get work that is so tiny or insignificant that you can hardly take ownership of it. The best strategy for dealing with it is to quietly get it over with and hope you won’t get such stuff again.

However, sometimes you can get caught in a rut of continuously getting this kind of work, and start wondering what you actually signed up for. And this is one thing I hadn’t expected to encounter when I got back to full time working earlier this year.

Thinking about why I never had to encounter such stuff during my consulting life, I realised there’s a fairly simple explanation – transaction costs.

Being a consultant is high transaction cost business. Every time you need to take on a new piece of work, you need to go through the charade of negotiating specifics with the client, pricing and drawing up a contract. All put together, the effort is not insignificant.

Moreover, in the line of work that I used to do, there was this massive overhead cost of understanding, cleaning and getting comfortable with the client’s data – the effort involved in that meant that after a particular point in time I stopped taking work that wasn’t chunky enough. For a while I started refusing such work, but then got smarter and started pricing myself out of such work (though some clients were generous enough to meet that price to get their little tasks done – effectively I’d passed on the transaction costs to them).

The downside of this, of course, was that there was a fair amount of money I could have made taking up small works which I didn’t since the transaction cost was too high – this can be thought of as potential lost revenues. The upside was that whatever work I did was of high quality and (hopefully) made a big impact on the client’s business.

In the nature of the firm, Ronald Coase wrote that the purpose of the corporation was that transaction cost of dealing with co-workers can be eliminated. But then, I realise that sometimes this transaction cost can also be a good thing!

Oh, and obligatory plug here – my book Between the buyer and the seller deals with transaction costs, among other things. It’s available for sale (both in print and digital) on Amazon.

Letters to my wife

As I turned Thirty Three yesterday, my wife dug up some letters (emails to be precise) I’d written to her over the years and compiled them for me, urging me to create at “Project Thirty Four” (on the lines of my Project Thirty). What is pleasantly surprising is that I’ve actually managed to make a life plan for myself, and execute it (surprising considering I don’t consider myself to be too good a planner in general).

In February 2011, after having returned from a rather strenuous work trip to New York, this is what I had to say (emphasis added later, typos as in original):

For me steady state is when I’ll be doing lots of part-time jobs, consulting gigs, where I’m mostly owrking from home, getting out only to meet people, getting to meet a lot of people (somethign taht doesn’t happen in this job), having fun in the evenings and all that

I wrote this six months before I exited my last job, and it is interesting that it almost perfectly reflects my life nowadays (except for the “have fun in the evenings” bit, but that can be put down to being long distance).

I’ve just started a part time job. I have a couple of consulting gigs going. I write for a newspaper (and get paid for it). I mostly work from home. I’ve had one “general catch up” a day on an average (this data is from this Quantified Life sheet my wife set up for me).

A week later I had already started planning what I wanted to do next. Some excerpts from a letter I wrote in March 2011:

Ok so I plan to start a business. I don’t know when I’ll start, but I’m targeting sometime mid 2012.

I want to offer data consultancy services.

Basically companies will have shitloads of data that they can’t make sense of. They need someone who is well-versed in working with and looking at data, who can help them make sense of all that they’ve got. And I’m going to be that person.

Too many people think of data analysis as a science and just through at data all the analytical and statistical weapons that they’ve got. I believe that is the wrong approach and leads to spurious results that can be harmful for the client’s business.

However, I think it is an art. Making sense of data is like taming a pet dog. There is a way you communicate with it. There is a way you make it do tricks (give you the required information). And one needs to proceed slowly and cautiously in order to get the desired results.

I think of myself as a “semi-quant”. While I am well-versed in all the quantitative techniques in data analysis and financial modeling, I’m also deeply aware that using quantitative tools indiscriminately can lead to mismanagement of risks, which can be harmful to the client. I believe in limited and “sustainable” use of quantitative tools, so that it can lead without misleading.

My past experience with working with data is that data analysis can be disruptive. I don’t promise results that will be of particular liking for the client – but I promise that what I diagnose is good for the client’s business. When you dig through mountains of data, you are bound to get some bitter pills. I expect my clients to handle the bad news professionally and not shoot the messenger.

I don’t promise to find a “signal” in every data set that I’m given. There are chances that what I’m working with is pure noise, and in case I find that, I’ll make efforts to prove that to the client (I think that is also valuable information).

And these paragraphs, written a full year before I started out doing what I’m doing now, pretty much encapsulate what I’m doing now. Very little has changed over nearly five years! I feel rather proud of myself!

And a thousand thanks to my wife for picking out these emails I had sent her and showing me that I can work to a plan.

Now on to making Project Thirty Four, which I hope to publish by the end of today, and hope to execute by the end of next year.

Back to bachelorhood

Starting tonight I’ll be a bachelor once again. For the next nineteen months or so. No it’s not that I’m returning my post graduate diploma and hence getting this downgrade (it’s been a while since I cracked a bad joke here so I’m entitled). It’s that the wife is going away. To get herself an MBA (yes I know that after this she will be better qualified than me since she’ll be getting a proper MBA while i have a post graduate diploma only. Maybe I can retire soon? ).

She’ll be going off to Barcelona tonight. The original plan had me moving there too. But then classic old NED happened and I ended up not looking for a job or assignment there and since it’s not an inexpensive place to stay I’m staying back. Plan to visit her every once in a while. And even though tickets to Europe are prohibitively expensive I now have a place to go to in case I need a break.

But for that I need to first get myself a visa. I guess one of my chief tasks in the next few days will be to get this bit of business done. But then I have my own business.

Regulars on this blog might be aware that I haven’t had formal employment for close to three years now. I freelance as a quAnt consultant – helping companies figure out how to make use of the volumes of data they collect in improving their business decision making. It’s been doing quite okay but my plan is to use the next few months when I don’t have any domestic commitments to see if I can take it to the next level.

It might also be pertinent to mention here that the first bit if business I got for this particular venture was through this blog – the last time I put out a post like this one a long time reader who was looking for quant assistance left a comment here and that led to a rather fruitful assignment. Perhaps mentioning this here might result in a repeat?

Now that I’m blogging more than I used to in the recent past I’ll also be using these pages to keep you updated on the long distanceness. I’ve also noticed that since I last put the update on leaving twitter and Facebook that there’s some more activity here. Keep that flowing and I hope for some good conversations on the comment pages here.

Countercyclical business

I realize being a freelance management consultant is countercyclical business. For two years in succession, I’ve had a light March – both years I’ve ended up finishing projects in Jan/Feb. With March being the end of the Indian financial year, most companies are loathe to commit additional spending in March, and it is a bad time to start new projects!

This is counter-cyclical because most other businesses end up having a bumper March, since they have end-of-year targets, and with a short sales cycle, they push their salespersons hard to achieve this target in March!

Coase

In the wake of the passing of Ronald Coase, two incidents, both professional. The first was with an established company to whom I suggested a partnership – they are in a space where I don’t have much skill, but have access to companies who I would love to sell to, and they don’t have my skill and our skills are complementary. So I reached out to them (through common contacts) suggesting that we could work together. They came back to me saying they would love to work with me, but would want me to join them as an employee.

The second was an incoming lead. This was a rather small company doing something similar to what I’m doing but with bigger ideas. They want me to join this “innovation hub” they are trying to create. This is a loose federation they are creating including professionals from various fields. Nobody is obliged to work full time for the hub, but this gives people an opportunity to get together and work together on projects where their respective expertise can combine well.

As the more perceptive of you who would have read every Coase obituary in the last two weeks would have figured out, the piece of work that Coase is most well known for is about the theory of the firm. The question is rather simple – why should you and I get together and form a firm if we have to work together, if we can remain independent and just come together for projects. The answer lies in transaction costs.

The advantage of coming together as a firm is that you negotiate only once. Let us suppose you are a graphic designer and I’m a data scientist. If we decide to work together on a visualization project, how do we decide how much you get and how much I get? We will need to negotiate. Let’s say we negotiate and agree on a price. And complete a project and split the spoils. What would happen the next time we were to bid for a project? We will need to negotiate again on how we will share the spoils.

If on the other hand we were to form a partnership firm, then for every project that we do, our respective share is fixed! Thus we don’t have to negotiate before every single projects. Thus, firms exist so that you don’t have to repeatedly negotiate.

However, there is a downside to this. What if I form a firm with a graphic designer, and then we see a significant opportunity in projects that involve a lot of analysis but little visualization? In that case, I have no use of my partner, and would loathe to pay him his share of the profits. Or consider if I were to somehow become much better at my job, while my partner stagnates. There is little I can do, for we’ve been locked in into the financial arrangement.

These are only some of the complications that arise when you need to decide whether you want to become a firm. I just thought it is pertinent that I’m having some of these dilemmas (employee versus consultant versus partner versus member of federation) in the few weeks after Coase’s passing.

Fighterization of food

One of the topics that I’d introduced on my blog not so long ago was “fighterization“. The funda was basically about how professions that are inherently stud are “fighterzied” so that a larger number of people can participate in it, and a larger number of people can be served. In the original post, I had written about how strategy consulting has completely changed based on fighterization.

After that, I pointed out about how processes are set – my hypothesis being that the “process” is something that some stud would have followed, and which some people liked because of which it became a process. And more recently, I wrote about the fighterization of Carnatic music, which is an exception to the general rule. Classical music has not been fighterized so as to enable more people to participate, or to serve a larger market. It has naturally evolved this way.

And even more recently, I had talked about how “stud instructions” (which are looser, and more ‘principles based’) are inherently different from “fighter instructions” (which are basically a set of rules). Ravi, in a comment on Mohit‘s google reader shared items, said it’s like rule-based versus principles-based regulation.

Today I was reading this Vir Sanghvi piece on Lucknowi cuisine, which among other things talks about the fact that it is pulao that is made in Lucknow, and now biryani; and about the general declining standards at the Taj Lucknow. However, the part that caught my eye, which has resulted in this post with an ultra-long introduction was this statement:

The secret of good Lucknowi cooking, he said, is not the recipe. It is the hand. A chef has to know when to add what and depending on the water, the quality of the meat etc, it’s never exactly the same process. A great chef will have the confidence to improvise and to extract the maximum flavour from the ingredients.

This basically states that high-end cooking is basically a stud process. That the top chefs are studs, and can adapt their cooking and methods and styles to the ingredients and the atmosphere in order to churn out the best possible product.You might notice that most good cooks are this way. There is some bit of randomness or flexibility in the process that allows them to give out a superior product. And a possible reason why they may not be willing to give out their recipes even if they are not worried about their copyright is that the process of cooking is a stud process, and is hence not easily explained.

Publishing recipes is the attempt at fighterization of cooking. Each step is laid down in stone. Each ingredient needs to be exactly measured (apart from salt which is usually “to taste”). Each part of the process needs to be followed properly in the correct order. And if you do everything perfectly, you will get the perfect standardized product.

Confession time. I’ve been in Gurgaon for 8 months and have yet to go to Old Delhi to eat (maybe I should make amends this saturday. if you want to join me, or in fact lead me, leave a comment). The only choley-bhature that I’ve had has been at Haldiram’s. And however well they attempt to make it, all they can churn out is the standardized “perfect” product. The “magic” that is supposed to be there in the food of Old Delhi is nowhere to be seen.

Taking an example close to home, my mother’s cooking can be broadly classified into two. One is the stuff that she has learnt from watching her mother and sisters cook. And she is great at making all of these – Bisibelebhath and masala dosa being her trademark dishes (most guests usually ask her to make one of these whenever we invite them home for a meal). She has learnt to make these things by watching. By trying and erring. And putting her personal touch to it. And she makes them really well.

On the other hand, there are these things that she makes by looking at recipes published in Women’s Era. Usually she messes them up. When she doesn’t, it’s standardized fare. She has learnt to cook them by a fighter process. Though I must mention that the closer the “special dish” is to traditional Kannadiga cooking (which she specializes in), the better it turns out.

Another example close to home. My own cooking. Certain things I’ve learnt to make by watching my mother cook. Certain other things I’ve learnt from this cookbook that my parents wrote for me before I went to England four years ago. And the quality of the stuff that I make, the taste in either case, etc. is markedly different.

So much about food. Coming to work, my day job involves fighterization too. Stock trading is supposed to be a stud process. And by trying to implement algorithmic trading, my company is trying to fighterize it. The company is not willing to take any half-measures in fighterization, so it is recruiting the ultimate fighter of ’em all – the computer – and teaching it to trade.

Preliminary reading on studs and fighters theory:

Studs and Fighters

Extending the studs and fighters theory

Meeting Sickness

Ok here is another reason I can think of as to why I didn’t do well in my consulting career. This is based on something I’ve been observing at office over the last week or two. I suffer from what I call as “meeting sickness”. The inability to work immediately after a meeting.

Rough empirical analysis tells me that for every meeting of N minutes that I sit through, I need another N minuts of downtime following it before I can get back to work. I don’t know why this happens to me. I don’t know if I’m having to spend too much willpower inside the meeting. Or if it is just that at meetings i get into high-intensity mode and that drains me out.

Whatever it is, in a typical consulting environment, you are expected to attend lots of meetings. If you work for a company that believes in the philosophy that all work is to be done at the client’s location (such as AT Kearney) then you have meetings throughout the day. it is only in between meetings that you get time to work, and usually the way the projects have been sold means that you can’t afford any downtime.

So that explains it. The other big reasons I’ve come up with for my failure in consulting environment are it requires a high degree of willpower which I dont’ have; and that it is an essentially fighter job. Maybe these are inter-related. Need to think on these lines and come up with something.

And if you didn’t like this post, my apologies. I had an extra-long meeting at work this evening from 4 to 7 (and had sat through three other meetings since morning). I fled immediately after, but I’m yet to recover.