Selection bias and recommendation systems

Yesterday I was watching a video on youtube, and at the end of it it recommended another (the “top recommendation” at that point in time). This video floored me – it was a superb rendition of Endaro Mahaanubhaavulu by Mandolin U Shrinivas. Listen and enjoy as you read the rest of the post.

I was immediately bowled over by youtube’s recommendation system. I had searched for both Shrinivas and Endaro … in the not-so-distant past so Youtube had put two and two together and served me up an awesome rendition! I was so happy that I went to town twitter about it.

It was then that I realised that this was the firs time ever that I had noticed the top recommendation of Youtube. In other words, every time I use youtube, it recommends a video to me, but I seldom notice it. And I seldom notice it for a reason – they’re usually irrelevant and crap. The one time I like the video it throws up, though, I feel really happy and go gaga over the algorithm!

In other words, there’s a bias which I don’t know what its exactly called – the bias that when event happens in a certain direction, you tend to notice it and give credit where you think it’s due. And when it doesn’t happen that way, you simply ignore it!

In terms of larger implications, this is similar to how legends such as “lucky shirts” are born. When something spectacular happens, you notice everything that is associated with that spectacular event and give credit where you think it’s due (lucky shirt, lucky pen, etc.). But when things don’t go your way you think it’s despite the lucky shirt, not because the shirt has become unlucky.

It’s the same thing with belief in “god”. When you pray and something good happens to you after that, you believe that your prayers have been answered. However, when you pray and something good doesn’t happen, you ignore the fact that you prayed.

Coming back to recommendation systems such as Youtube’s, the problem is that it is impossible for a recommendation system to get recommendations right all the time. There will be times when you get it wrong. In fact, going by my personal experience with Youtube, Amazon, etc. most of the time you will get your recommendation wrong.

The key to building a recommendation system, thus, is to build it such that you maximise the chances of getting it right. Going one step further I can say that you should maximise the chances of getting it spectacularly right, in which case the customer will notice and give you credit for understanding her. Getting it “partly right” most of the time is not enough to catch the customer’s attention.

Putting marketing jargon on it, what you should focus on is delighting the customer some of the time rather than keeping her merely happy most of the time!

Ganeshana Madhuve and Challenge Gopalakrishna

Scenes from these two movies were enacted out at our wedding.

So in certain cultures (such as my wife’s; this isn’t practiced in my mother’s house at least) there is a uniform that brides need to wear – a white or off-white sari with a red border. I think this uniform is there in my father’s family also, but I’m not sure. I’m sure this is not there on my mother’s side.

Anyway, Priyanka was in her uniform, in the “bride’s room” doing “gowri pooje” that is supposed to be done before a girl gets married. There were several other women around, and for the wedding, they had all chosen to wear their own wedding saris – white or off-white with a red border. This included mostly Priyanka’s aunts and cousins and one of my aunts.

So there is this scene in Ganeshana maduve where Ganesha (YG Rao) is told that the girl in red sari is Shruti (his “pen lover”). And he goes into the bride’s room to find that everyone there was in a red sari, so he has no clue in figuring out who Shruti is. Similarly, if someone had come to the bride’s room searching for one particular woman who was in the white-red uniform, they would’ve been thoroughly confused indeed.

Then there is this scene that is oft-repeated in the other classic Anantnag starrer Challenge Gopalakrishna (I’m not able to find the link on youtube). Whenever Gopalakrishna’s dad abuses him, he reminds him of his lineage. (translating) “Being the great-grandson of Justice Gopalakrishna (loud temple gong), being the grandson of Major Radhakrishna (another loud temple gong), being the son of Rotarian Muralikrishna (yet another gong) you dare to behave like this… “. This scene is played out several times in the movie, and towards the latter half, as soon as Mukhyamantri Chandru utters “Justice Gopalakrishna”, Anantnag runs.

So as part of the wedding rituals, the bride and groom are anointed as Lakshmi and Narayana (the gods). So while I was being anointed such, the priest chanted “Venkataramanasharma nautram, Suryanarayanasharma poutram, Shashidharasharma putram Shri Karthika Sharma … ” and similarly for Priyanka (that way I got to know her great-grandfather’s name). The first couple of times it was ok. But when this bit came up later on in the rituals, we couldn’t help but burst out laughing. Thankfully there were no temple gongs to punctuate the recital.

I’m not sure if Lakshmi and Narayana are supposed to laugh.

Business Model for DD

Flipping channels an hour back, I happened to stop at this never-heard-before channel called “DD Bharati”. Usha Uthup was giving a concert that was  clearly recorded for television. Looking at her, and considering that the program had been recorded in black-and-white film, I would suppose that it was ancient indeed. Maybe from some time in the 70s.

The program itself was nice. The sets, for the time, were excellent. Usha was backed up by a bunch of men clad in suits – one on keyboard, a couple on guitars, one on trumpet, one drumming, one on the cymbals and another just swaying from side to side. The songs were all quite good, most of them Usha’s own compositions, and I didn’t think twice about giving up on ESPN Sportscenter Asia, Roland Garros and three not-so-bad Hindi movies in order to watch this program. And while I was watching I thought of this business model for Doordarshan.

The basic idea is that there is a whole lot of footage – all that was shown all through the 70s and 80s – that is quite popular among people and nostalgia-inducing, which is held by Doordarshan. I would be surprised if DD would have licensed out any of its old content to any other channel, if not for any other reason but because so much bureaucracy would have to move for that to happen. Stuff like 80s soaps and sitcoms, shows like the Usha Uthup performance I watched today, etc.

So I think DD can truly profitably run a “nostalgia” kind of channel. The market of people who grew up on these programs in the 70s and 80s is large, and most would prefer to watch re-runs of those ancient shows rather than watch the tripe that is dished out by most channels today. And then there is an opportunity for people to catch up on stuff they missed out on back then for various reasons – for example I missed out on so many cool programs back in the late 80s because our antenna didn’t catch DD2, and I wouldn’t really mind watching those today.

And then those ads – yeah they are available on youtube (and on dd’s own site) but then I’m sure it would be profitable to run those ads now as programs in themselves! The opportunities, I think, are endless. Unfortunately it is a sarkari company that is not interested in profits that is sitting on all these options. The loss, I think, is for us potential viewers.

Introducing: NED Talks

I suppose a number of you have heard of TED Talks (ted.com). You might have seen it in the news recently, for it is going to come to Mysore sometime later this year. TED talks are available online (on the ted website), and are in general extremely informative and entertaining.

When TED can have talks, can NED be far behind? So I, along with Kodhi, hereby invite you to the first of the series of NED talks. The beauty of NED talks is that nothing is known yet. So far we’ve been putting NED to figure out what we are going to have as part of those talks. However, wee are confident that we won’t put NED to organize the talks themselves.

So here are a few salient features. You might notice that nothing is concrete yet, heck nothing is even the steel framework yet. I request you to add your own concrete to this, and put in your ideas.

  • The first of the series of NED talks is likely to be held in Bangalore in October 2009. I have picked the date randomly, and is subject to change.
  • The first of the series will be a day-long event. However, if registrations exceed our expectations, the event might last a whole weekend.
  • We have no clue what is going to be there as part of these talks. We know that it is going to be a bunch of mango people (aam junta for the uninitiated) who will be talking, but we dont’ know what they are going to be talking about. We don’t even know who are the people who are going to be talking, though I’m sure I’m going to be one of them.
  • If you have any bright ideas as to what we can do at the NED talks, please let us know.
  • NED is not rich enough to put talks on its own website, hence the videos will be put up on youtube. However, we cannot promise the same quality that the TED videos.
  • Please note that NED talks are only loosely inspired by TED talks, and has nothing to do with the latter. I also wish to clarify that NED talks are not a spoof of TED talks.
  • At the risk of repeating myself – if you have any bright ideas as to what can be done at the NED talks, plis to be cantributing.
  • Oh and at the risk of repeating myself again – we are really serious about organizxing the NED talks. And we promise that we won’t put NED for NED talks.