Rating systems need to be designed carefully

Different people use the same rating scale in different ways. Hence, nuance is required while aggregating ratings taking decisions based on them

During the recent Times Lit Fest in Bangalore, I was talking to some acquaintances regarding the recent Uber rape case (where a car driver hired though the Uber app in Delhi allegedly raped a woman). We were talking about what Uber can potentially do to prevent bad behaviour from drivers (which results in loss of reputation, and consequently business, for Uber), when one of them mentioned that the driver accused of rape had an earlier complaint against him within the Uber system, but because the complainant in that case had given him “three stars”, Uber had not pulled him up.

Now, Uber has a system of rating both drivers and passengers after each ride – you are prompted to give the rating as soon as the ride is done, and you are unable to proceed to your next booking unless you’ve rated the previous ride. What this ensures is that there is no selection bias in rating – typically you leave a rating only when the product/service has been exceptionally good or bad, leading to skewed ratings. Uber’s prompts imply that there is no opportunity for such bias and ratings are usually fair.

Except for one problem – different people have different norms for rating. For example, i believe that there is nothing “exceptional” that an Uber driver can do for me, and hence my default rating for all “satisfactory” rides is a 5, with lower scores being used progressively for different levels of infractions. For another user, for example, the default might be 1, with 2 to 5 being used for various levels of good service. Yet another user might use only half the provided scale, with 3 being “pathetic”, for example. I once worked for a firm where annual employee ratings came out on a similar five-point scale. Over the years so much “rating inflation” had happened that back when I worked there anything marginally lower than 4 on 5 was enough to get you sacked.

What this means is that arithmetically averaging ratings across raters, and devising policies based on particular levels of ratings is clearly wrong. For example, when in the earlier case (as mentioned by my acquaintance) a user rated the offending driver a 3, Uber should not have looked at the rating in isolation, but in relation to other ratings given by that particular user (assuming she had used the service before).

It is a similar case with any other rating system – a rating looked at in isolation tells you nothing. What you need to do is to look at it in relation to other ratings by the user. It is also not enough to look at a rating in relation to just the “average” rating given by a user – variance also matters. Consider, for example, two users. Ramu uses 3 for average service, 4 for exceptional and 2 for pathetic. Shamu also uses 3 for average, but he instead uses the “full scale”, using 5 for exceptional service and 1 for pathetic. Now, if a particular product/service is rated 1 by both Ramu and Shamu, it means different things – in Shamu’s case it is “simply pathetic”, for that is both the lowest score he has given in the past and the lowest he can give. In Ramu’s case, on the other hand, a rating of 1 can only be described as “exceptionally pathetic”, for his variance is low and hence he almost never rates someone below 2!

Thus, while a rating system is a necessity in ensuring good service in a two-sided market, it needs to be designed and implemented in a careful manner. Lack of nuance in designing a rating system can result in undermining the system and rendering it effectively useless!

Matching problem in the Indian Dating Market

And no, this has nothing to do with Hall’s Marriage Problem.

As the more perceptive of you might have noticed, about a year and a bit back the wife started this initiative called “Marriage Broker Auntie”. Basically she thinks she is a good judge of people and a good judge of “matches” between people. As a consequence of this, there were many friends and relatives who would approach her to “set them up”. Having (successfully or unsuccessfully) made a few matches among such applicants, she decided to institutionalise it, and thus Marriage Broker Auntie was born.

The explicit objective of the initiative was to broker marriages (as the name clearly suggests. The wife was the “auntie”. The plural in the title was because briefly there was one more (mad) auntie involved but she’s since moved on). The methodology was to “know the customers” and then use an intelligent human process in order to find pairs who might be interested in each other and then set them up for a date. As simple and basic as it gets. The quant in me had already started dreaming up of an expanded business where I could use “big data” and “analytics” and whatnot to “understand” large sets of people and match them up.

But there was a small problem – ok the fundamental problem was that the number of people who had signed up was not very large, but let’s assume that can be solved through marketing – the problem was that the sex ratio on the website was skewed. Heavily. At one point in time I remember there being 20 girls and 5 boys being registered on the website (all heterosexual, perhaps a consequence of “marriage” in the title)! The ratio remained thus as long as the initiative was in existence.

Now this contrasts heavily with other “dating” sites that are operational in India, primarily global sites such as OkCupid and Tinder. The gender ratio on these sites is heavily skewed, too, except that it is heavily skewed in the opposite direction (too many boys, hardly any girls). For example, check out this piece in Man’s World on Tinder, which talks about users who think there is a “permanent bug” in the site that doesn’t allow matches, and of “all girls on the site being bots”.

The problem with most dating sites in India is that there are way too many boys and way too few girls (I should add Orkut also to this list, and should mention that I met my wife through a combination of Orkut and LiveJournal). This leads to girls on the site feeling like they’re “being stalked”, and getting freaked out and getting out of the site. A girl I know signed up on OkCupid India, just on a whim, only to find a hundred “interests” from boys within a few minutes of logging on.

Reading stories like this, you might be bound to imagine that there are no girls in India interested in dating, or getting married. But if you were to look at sites like Marriage Broker Auntie (small sample, I know, but significant gender bias) you know that this is simply untrue. There are girls out there who are looking for flings and relationships, to date and to get married. And they are short of ideas on how to meet such men. “Traditional” dating sites such as OkCupid or Tinder intimidate them, and shaadi.com and bharatmatrimony.com are in a completely different business altogether – they make matches on a different set of variables such as caste and gotra and stars and so forth.

So what we have here is classic market failure – of the Indian dating market (this, however, is NOT a call for government regulation! 😛 ). The market is surely fertile (no pun intended), and there is plenty of opportunity to make fat profits if someone can get the matching right. There are a number of players looking to enter the market as I write this (I’ve spoken to some of them), but none look particularly promising.

Oh, and you might want to know why Marriage Broker Aunties gets all the chicks – it’s because of a complicated sign up process (a five page google docs form if i remember right) which puts off any non-serious players. Also there is the promise that until matched, potential counterparties cannot see your profile, and there is a “trusted third party” (in the MBA case, my wife, but an algorithm should do reasonably well to scale) who does the matchings.

The most important bit here is the anonymity – the ground reality in India is that online dating is still seen as a “last resort” – to be resorted to only if you can’t find a match through your network. With Tinder and OkCupid being exclusively dating sites (unlike Orkut which was fundamentally a social network), signing up on one of these two sites is an admission of a degree of desperation (in the eyes of most people), and there is a chance people might see you differently after they know that you’re a member on such sites.

While this explains reasonably well why chicks flock to Marriage Broker Auntie, why is there a shortage of guys on the site? It can’t be that there are no serious guys around for whom the 5-page form is a massive transaction cost. The wife’s (and my) perception is that fundamentally guys want to “check out a girl” (i.e. know well what she looks like, etc.) before agreeing to meet her on a date (I remember scouring Orkut and Facebook for all possible pictures of my then-future-wife and “checking her out” before meeting for the first time). And in an anonymised matching site, this experience is not there. So men don’t like this!

It’s a hard problem but not intractable. There are many companies that are coming into this space now. Hopefully someone will get it right!

How much surge is too much surge?

I had gone for a wedding in far-off Yelahanka and hailed an Uber on the way back. The driver was bragging about how it’s easy to find an Uber at any time anywhere in Bangalore, when I pointed out to him that earlier in the evening when I was on my way to the wedding I’d failed to find one, and had taken an Ola instead.

He was surprised that an Uber wasn’t available in Jayanagar when I told him that there were cars available but at a 1.7X surge, and given the distance I was to travel I found it more economical to take an Ola which was offering a ride at a flat Rs. 50 premium. To this, the driver said that he had also noticed that demand sharply dropped off once the level of surge went beyond 1.5X, and at such surges supply would easily outstrip demand.

Now I’m no fan of Ola’s pricing – I think the flat Rs. 50 premium during peak hours is unscientific, but I wonder if the level of Uber’s surges makes sense. From a pure microeconomic standpoint, it is easy to see where Uber is coming from – raise price until quantity demanded matches quantity supplied and let the market clear. The question, however, is if this kind of a surge makes sense from a behavioural standpoint.

The point is that the “base fare” (“1X”) is “anchored” in the customer’s mind, and thus any decision he takes in terms of willingness to pay is made keeping this “anchor” in mind. And when the quoted price moves too far from the anchor (beyond 1.5X, say), the customer deems that it is “too expensive”, and decides that waiting for a few minutes for fares to drop (or using a competing app) is superior to paying the massive premium.

I suppose that Uber would have noticed this. That there is a “cliff” surge price beyond which there is a massive drop off in volume of matchings. The problem is that if they restrict their surges to this “cliff value” they might be leaving money on the table by not being able to match the market. On the other side, though, if the surge is so high that the volume of transactions drops sharply, it results in much lower commissions for Uber! I’m assuming that a solution to this problem is on the way!

And I’ve found that it’s always harder to find a taxi on a Sunday. The problem is that because demand is lower, supply is also lower (this is a unique characteristic of “two-sided markets”) because of which the chances of finding a match are harder, and transaction costs are higher. I wonder if it makes sense for taxi aggregators to levy a “Sunday premium” (perhaps with Uber holding a day-long minimum of 1.2X surge or something) to compensate for this lack of liquidity!

Why the proposed Ola-TaxiForSure merger is bad news

While a merger intuitively makes economic sense, it’s not good for the customers. The industry is consolidating way too fast, and hopefully new challengers will arise soon

Today’s Economic Times reports that Ola Cabs is in the process of buying out competitor TaxiForSure. As a regular user of such services, I’m not happy, and I think this is a bad move. I must mention upfront, though, that I don’t use either of these two services much. I’ve never used TaxiForSure (mostly because I never find a cab using its service), and have used Ola sparingly (it’s my second choice after Uber, so use it only when Uber is not available).

Now, intuitively, consolidation in a platform industry is a good thing. This means that more customers and more drivers are on the same platform, and that implies that the possibility of finding a real-time match between a customer who wants a ride and a driver who wants to offer one is enhanced. The two-sided network effects that are inherent in markets like this imply super-linear returns to scale, and so such models work only at scale. This is perhaps the reason as to why this sector has drawn such massive investments.

While it is true that consolidation will mean lower matching cost for both customers and drivers, my view on this is that it’s happening too soon. The on-demand taxi market in India is still very young (it effectively took off less than a year back when Uber made its entry here. Prior to that, TaxiForSure was not “on demand” and Ola was too niche), and is still going through the process of experimentation that a young industry should.

For starters, the licensing norms for this industry are not clear (and it is unlikely they will be for a long time, considering how disruptive this industry is). Secondly, pricing models are still fluid and firms are experimenting significantly with them. As a corollary to that, driver incentive schemes (especially to prevent them from “multihoming”) are also  rather fluid. The process of finding a match (the process a customer and a driver have to go through in order to “match” with each other), is also being experimented with, though the deal indicates that the verdict on this is clear. Essentially there are too many things in the industry that are still fluid.

The problem with consolidation at a time when paradigms and procedures are still fluid is that current paradigms (which may not be optimal) will get “frozen”, and customers (and drivers) will have to live with the inefficiencies and suboptimalities that are part of the current paradigms. It looks as if after this consolidation the industry will settle into a comfortable duopoly, and comfortable duopolies are never great for innovation and for finding more optimal solutions.

Apart from the network effects, the reasons for the merger are clear, though – in the mad funding cycle unleashed by investors into this industry, TaxiForSure was a clear loser and was finding itself unable to compete against the larger better-funded rivals. Thus, it was a rational decision for the company to get acquired at this point in time. From Ola’s point of view, too, it is rational to do the deal, for it would give them substantial inorganic growth and undisputed number one position in the industry. For customers and drivers, though, now faced with lower choice, it is not a great deal.

This consolidation doesn’t mean the end, though. The strength of a robust industry is one where weak firms go out of business and new firms spring up in their place in their attempt to make a profit. That three has become two doesn’t mean that it should remain at two. There is room in the short term for a number three and even possibly a number four, as the Indian taxi aggregation industry tries to find its most efficient level.

I would posit that the most likely candidates to emerge as new challengers are companies such as Meru or EasyCabs, which are already in the cab provider business but only need to tweak their model to include an on-demand component. A wholly new venture to take up the place that is being vacated by TaxiForSure, however, cannot be ruled out. The only problem is that most major venture capitalists are in on either Uber or Ola, so it’s going to be a challenge for the new challenger to raise finances.

\begin{shameless plug}
I’m game for such a venture, and come on board to provide services in pricing, revenue management, availability management and driver incentive optimisation. 🙂
\end{shameless plug}


Finally some sensible Uber regulation

Ever since Uber launched, regulators worldwide haven’t had a clue as to how to regulate it – it has been such a big disruptor in the taxicab market. Some countries and cities have taken to banning it outright (the list is too long to post links here). Others (such as some states in India) have tried to get Uber to register itself as a “taxicab company”.

The problem with all these regulations is that the Uber model (being replicated by firms such as Ola and TaxiForSure in India) is a fundamental gamechanger. As I have written in this earlier post, the on-demand model propagated by Uber implies that a number of the inefficiencies in the taxicab market don’t exist any more. In this context, trying to regulate it by moving it back to the earlier (extremely inefficient) model is extremely regressive. The right way to regulate is to create a level playing field for taxicab aggregators (which includes Uber) and move the market to a regime where the new technology-enabled efficiencies are made good use of.

And that is precisely what Los Angeles has done. In a rather progressive move (which ought to be copied by other states and cities and countries), the city has decreed that all city-based cab operators need to offer app-based booking services. The interesting bit in the regulation (see link above) is that drivers who fail to install the e-hail app are actually going to be fined.

What this will lead to is that the local taxi market is itself going to become more efficient which should definitely increase both profitability for the local cab industry and also availability of local cabs to the people of LA. What this will also do is to give people of LA a choice between using Uber and the traditional taxi app, which will lead to an improvement in Uber’s service levels. As things stand now I don’t see any downside from this LA regulation.

I hope the model succeeds in LA and other cities see the brilliance of the model and accept the efficiencies brought into the market thanks to this model and adopt similar regulation. I see this kind of regulation coming into the Bangalore market though the backdoor though. Ola already helps match auto rickshaws to customers and now TaxiForSure is also getting into that market. Will this mean that autos won’t have to line up for hours together in front of Lalbagh gate for passengers arriving in the city by bus?

Oh, and LA is not the first city to implement regulations requiring taxis to be “hailable” via an app. When I visited Singapore in November 2013, I found that cabs in the city worked the same way. Locals had an app which they would use to call taxis. The problem there though was that the app was only available to locals (your android/iOS had to be registered in Singapore for you to be able to even install the app), which made it a nightmare for us tourists to move around.

Oh, and while on the topic, a good revenue source for companies such as Ola or TaxiForSure would be to provide the technology backbone to cities that are seeking to use app-based hailing services for their cabs.


On getting fired

On Capital Mind, Deepak Shenoy has a great post out on the TCS layoffs. TL;DR: TCS could have handled it better, but getting fired is a part of corporate life. And 3 months’ severance is generous. He also adds that we should hedge – build your brand, build savings, build skills so that getting fired won’t hit you so hard.

An argument that is being bandied about in relation to the TCS layoffs that if you need job mobility, then job insecurity is a related price you have to pay. For example, check out these tweets from Raj:

So the basic argument here (which I completely agree with) is that you can’t have one-way optionality. A generation ago, there was almost no optionality. You couldn’t get sacked, and it was very difficult for you to leave. That was the way the world worked back then.

Soon, the economy expanded, and you started seeing mobility. You started seeing optionality – the job was a one-way option. You could choose when you wanted to leave, but given the high growth and general shortage of skilled talent back in the days, companies couldn’t sack you. That sweet spot existed for a short while.

In the last decade or so, though, this has started changing. Companies realised that keeping deadwood on the books is a lot more expensive than their financial cost-to-company. A “no firing” policy sends out the wrong incentives – people without motivation are more likely to stick around than the ambitious. And that can never be good for the company. So now companies want optionality both ways. And as the TCS episode illustrates, people are not liking that the optionality exists both ways now. It seems like they were used to the one-way optionality street that existed for a short while during the rapid expansion of the IT sector.

The problem with the above argument (encapsulated in Raj’s tweets, which I agree with), however, is that it assumes that employees have a choice. When you say that “if you want mobility, you get insecurity as part of the package”, the subliminal message is that it there exist jobs where you can choose to forego your mobility in order to save yourself from insecurity. Unfortunately not too many such jobs exist. And it is a matter of liquidity.

Yes, there still exist plenty of jobs where there is strong two-way commitment. However, they are nowhere as numerous as jobs where there exist two-way optionality. The simple matter is that the “market has moved”. Most people are comfortable with the “latest” arrangement, where you can leave easily but also get sacked easily. Given that most people are comfortable with this arrangement, companies are also comfortable with this and have moved to this arrangement. And that has led to a virtuous cycle and the number of companies and number of people who like this arrangement have hit a critical mass.

In other words, if you want an “optionless” job, that is like living in the world until yesterday. But it is not enough that you want to live in that world. The world as we know it is social, and for us to live a certain way, we need other people to agree to live the same way. In other words, we can’t live our chosen lifestyle in isolation without counterparties living that way too.  And when most employers have moved on from the optionless regime to the two-way optionality regime, even if you want to live in yesterday’s world, there aren’t too many companies that still live that way. So you don’t have a choice!

So you need to learn to adapt to live and thrive in the new regime. And it is not that this regime will last forever. I’m sure people will innovate and other regimes might supersede this regime. Some people are slow to react to change, but liquidity makes the world ruthless, and punishes you badly for not adapting. That is the hard truth that some of these people who are cribbing about getting fired from TCS need to digest.

Inefficiencies in the auto rickshaw market and Uber

Taxi marketplaces such as Uber and Ola address inefficiencies and failures in the auto rickshaw / taxi market

Weary after a long cold night journey you get off the overnight bus from Chennai at Lalbagh’s Double Road gate, and look around for auto rickshaws. There are some ten of them around. The drivers are equally weary, having woken up early and left their homes to stand in the cold, hoping to find passengers alighting from buses. They want to get compensated for this, and quote you a fare that includes such compensation. All of them quote similar fares. You grudgingly bargain and agree, and conclude that Bangalore’s auto drivers are bastards.

Alternate scenario: as the bus reached Madivala, ten minutes away from Lalbagh Double Road gate at that time of the morning, you pull out your app and ask for a taxi to pick you up from Double Road gate in ten minutes’ time. The driver has been up, but resting at home. He leaves home now, just in time to be there at Double Road gate by the time you get off there. Off you get into the car and go.

You have to get to work and try catching an auto rickshaw. The guy asks for extra money for he has to take you through traffic-laden roads, which are a tax on his time, which the regulated fare doesn’t compensate him for. You bargain, get in, and conclude that auto drivers are bastards.

In an alternate scenario, you use an app-based taxi which calculates the fare as a linear combination of distance travelled and time taken, which means that the driver gets compensated for getting stuck in traffic without having to bargain for it. And without you having to think that the driver is a bastard.

In the evening you are trying to get an auto rickshaw from MG Road, and the guy asks for a premium. This premium is not reflective of costs, but the fact that demand for auto rickshaws in that area at that time is high, and that there will be customers willing to pay that premium. You conclude that the auto rickshaw driver is a bastard. Uber’s surge pricing (which can be steep at times) doesn’t evoke the same reaction from you. Uber has centralised knowledge of demand and supply so they can clear prices better, while the auto driver, lacking that knowledge, quotes a price that reflects his lack of market knowledge. And not having a good idea of what to charge, he might try to charge above market price.

What I’m trying to say here is that the local taxi/auto rickshaw market is inefficient, and ridden with failures. There is lack of information flow between demand and supply, which leads to inferior price negotiation, and the transaction cost of time and effort wasted on negotiation as opposed to using that time to travel! And when a market fails, the classic economic response is regulation, but in the case of taxi markets regulation is so poor (regulated prices do not reflect costs) that it enhances the market failure. The (badly) regulated prices anchor into people’s minds unrealistic expectations, and when auto drivers nudge them towards more realistic market prices, passengers assume that they (drivers) are bastards.

It is in this context that players like Uber and Ola (I’m not a fan of Ola’s pricing model, though) step in and try to resolve the market failure by improving flow of demand-supply information and setting “clearing prices” that compensate the driver in line with his costs. If you look closely, these companies are actually rescuing the local taxi market from its inherent inefficiencies and failures and bad regulation!

It is important, however, that no one market place ends up becoming a monopoly. As long as we have two or three different marketplaces, both customers and drivers have the choice of moving between one and the other, and this will ensure that these market places face market pressures from the two sides of the market, and if they “regulate” in an unfair manner, their participants will move to a competing marketplace, resulting in loss of business for the marketplace.

But then, considering the inherent network effects of the marketplace model, I don’t know how we can ensure that competition exists!