Randomness and sample size

I have had a strange relationship with volleyball, as I’ve documented here. Unlike in most other sports I’ve played, I was a rather defensive volleyball player, excelling in backline defence, setting and blocking, rather than spiking.

The one aspect of my game which was out of line with the rest of my volleyball, but in line with my play in most other sports I’ve played competitively, was my serve. I had a big booming serve, which at school level was mostly unreturnable.

The downside of having an unreturnable serve, though, is that you are likely to miss your serve more often than the rest – it might mean hitting it too long, or into the net, or wide. And like in one of the examples I’ve quoted in my earlier post, it might mean not getting a chance to serve at all, as the warm up serve gets returned or goes into the net.

So I was discussing my volleyball non-career with a friend who is now heavily involved in the game, and he thought that I had possibly been extremely unlucky. My own take on this is that given how little I played, it’s quite likely that things would have gone spectacularly wrong.

Changing domains a little bit, there was a time when I was building strategies for algorithmic trading, in a class known as “statistical arbitrage”. The deal there is that you have a small “edge” on each trade, but if you do a large enough number of trades, you will make money. As it happened, the guy I was working for then got spooked out after the first couple of trades went bad and shut down the strategy at a heavy loss.

Changing domains a little less this time, this is also the reason why you shouldn’t check your portfolio too often if you’re investing for the long term – in the short run, when there have been “fewer plays”, the chances of having a negative return are higher even if you’re in a mostly safe strategy, as I had illustrated in this blog post in 2008¬†(using the Livejournal URL since the table didn’t port well to wordpress).

And changing domains once again, the sheer number of “samples” is possibly one reason that the whole idea of quantification of sport and “SABRmetrics” first took hold in baseball. The Major League Baseball season is typically 162 games long (and this is before the playoffs), which means that any small edge will translate into results in the course of the league. A smaller league would mean fewer games and thus more randomness, and a higher chance that a “better play” wouldn’t work out.

This also explains why when “Moneyball” took off with the Oakland A’s in the 1990s, they focussed mainly on league performance and not performance in the playoffs – in the latter, there are simply not enough “samples” for a marginal advantage in team strength to necessarily have the impact in terms of results.

And this is the problem with newly appointed managers of elite football clubs in Europe “targeting the Champions League” – a knockout tournament of that format means that the best team need not always win. Targeting a national league, played out over at least 34 games in the season is a much better bet.

Finally, there is also the issue of variance. A higher variance in performance means that observations of a few instances of bad performance is not sufficient to conclude that the player is a bad performer – a great performance need not be too far away. For a player with less randomness in performance – a more steady player, if you will – a few bad performances will tell you that they are unlikely to come good. High risk high return players, on the other hand, need to be given a longer rope.

I’d put this in a different way in a blog a few years back, about Mitchell Johnson.

Carbon taxes and mental health

The beautiful thing about mid-term elections in the USA is that apart from the “main elections” for senators, congresspersons and governors, there were also votes on “auxiliary issues” – referenda, basically, on issues such as legalisation of marijuana.

One such issue that went to the polls was in Washington State, where there was a proposal for imposition of carbon taxes, which sought to tax carbon dioxide emissions at $15 a tonne. The voters rejected the proposal, with the proposal getting only 44% of the polled votes in favour.

The defeat meant that another attempt at pricing in environmental costs, which could have offered significant benefits to ordinary people in terms of superior mental health, went down the drain.

Chapter 11 of Jordan Peterson’s 12 Rules for Life is both the best and the worst chapter of the book. It is the best for the reasons I’ve mentioned in this blog post earlier – about its discussions of risk, and about relationships and marriage in the United States. It is the worst because Peterson unnecessarily lengthens the chapter by using it to put forward his own views on several controversial issues – such as political correctness and masculinity – issues which only have a tenuous relationship with the meat of the chapter, and which only give an opportunity for Peterson’s zillion critics to downplay the book.

Among all these unnecessary digressions in Chapter Eleven, one stood out, possibly because of the strength of the argument and my own relationship with it – Peterson bullshits climate change and environmentalism, claiming that it only seeks to worsen the mental health of ordinary people. As a clinical psychologist, he can be trusted to tell us what affects people’s mental health. However, dismissing something just because it affects people negatively is wrong.

The reason environmentalism and climate change play a negative impact on people’s mental health, in my opinion, is that there is no market based pricing in these aspects. From childhood, we are told that we should “not waste water” or “not cut trees”, because activities like this will have an adverse effect on the environment.

Such arguments are always moral, about telling people to think of their descendants and the impact it will have. The reason these arguments are hard to make is because they need to persuade people to act contrary to their self-interest. For example, one may ask me to forego my self-interest of the enjoyment in bursting fireworks in favour of better air quality (which I may not necessarily care about). Someone else might ask me to forego my self-interest of a long shower, because of “water shortages”.

And this imposition of moral arguments that make us undertake activities that violate our self-interset is what imposes a mental cost. We are fundamentally selfish creatures, only indulging in activities that benefit us (either immediately or much later). And when people force us to think outside this self-interest, it comes with the cost of increased mental strain, which is reason enough for Jordan Peterson to bullshit environmentalism itself.

If you think about this, the reason we need to use moral arguments and make people act against their self-interest for environmental causes is because the market system fails in these cases. If we were able to put a price on environmental costs of activities, and make entities that indulge in such activities pay these costs, then the moral argument could be replaced by a price argument, and our natural self-interest maximising selves would get aligned with what is good for the world.

And while narrowly concerned with the issue of climate change and global warming, carbon taxes are one way to internalise the externality of environmental damage of our activities. And by putting a price on it, it means that we don’t need to think in terms of our everyday activities and thus saves us a “mental cost”. And this can lead to superior overall mental health.

In that sense, the rejection of the carbon tax proposal in Washington State is a regressive move.

Human, Animal and Machine Intelligence

Earlier this week I started watching this series on Netflix called “Terrorism Close Calls“. Each episode is about an instance of attempted terrorism that has been foiled in the last 2 decades. For example, there is one example of the plot to bomb a set of transatlantic flights from London to North America in 2006 (a consequence of which is that liquids still aren’t allowed on board flights).

So the first episode of the series involves this Afghani guy who drives all the way from Colorado to New York to place a series of bombs in the latter’s subways (metro train system). He is under surveillance through the length of his journey, and just as he is about to enter New York, he is stopped for what seems like a “routine drugs test”.

As the episode explains, “a set of dogs went around his car sniffing”, but “rather than being trained to sniff drugs” (as is routine in such a stop), “these dogs had been trained to sniff explosives”.

This little snippet got me thinking about how machines are “trained” to “learn”. At the most basic level, machine learning involves showing a large number of “positive cases” and “negative cases” based on which the program “learns” the differences between the positive and negative cases, and thus to identify the positive cases.

So if you want to built a system to identify cats in an image, you feed the machine a large number of images with cats in them, and a large(r) number of images without cats in them, each appropriately “labelled” (“cat” or “no cat”) and based on the differences, the system learns to identify cats.

Similarly, if you want to teach a system to detect cancers based on MRIs, you show it a set of MRIs that show malignant tumours, and another set of MRIs without malignant tumours, and sure enough the machine learns to distinguish between the two sets (you might have come across claims of “AI can cure cancer”. This is how it does it).

However, AI can sometimes go wrong by learning the wrong things. For example, an algorithm trained to recognise sheep started classifying grass as “sheep” (since most of the positive training samples had sheep in meadows). Another system went crazy in its labelling when an unexpected object (an elephant in a drawing room) was present in the picture.

While machines learn through lots of positive and negative examples, that is not how humans learn, as I’ve been observing as my daughter grows up. When she was very little, we got her a book with one photo each of 100 different animals. And we would sit with her every day pointing at each picture and telling her what each was.

Soon enough, she could recognise cats and dogs and elephants and tigers. All by means of being “trained on” one image of each such animal. Soon enough, she could recognise hitherto unseen pictures of cats and dogs (and elephants and tigers). And then recognise dogs (as dogs) as they passed her on the street. What absolutely astounded me was that she managed to correctly recognise a cartoon cat, when all she had seen thus far were “real cats”.

So where do animals stand, in this spectrum of human to machine learning? Do they recognise from positive examples only (like humans do)? Or do they learn from a combination of positive and negative examples (like machines)? One thing that limits the positive-only learning for animals is the limited range of their communication.

What drives my curiosity is that they get trained for specific things – that you have dogs to identify drugs and dogs to identify explosives. You don’t usually have dogs that can recognise both (specialisation is for insects, as they say – or maybe it’s for all non-human animals).

My suspicion (having never had a pet) is that the way animals learn is closer to how humans learn – based on a large number of positive examples, rather than as the difference between positive and negative examples. Just that the limit of the animal’s communication being limited means that it is hard to train them for more than one thing (or maybe there’s something to do with their mental bandwidth as well. I don’t know).

What do you think? Interestingly enough, there is a recent paper that talks about how many machine learning systems have “animal-like abilities” rather than coming close to human intelligence.

For millions of years, mankind lived, just like the animals.
And then something happened that unleashed the power of our imagination. We learned to talk
– Stephen Hawking, in the opening of a Roger Waters-less Pink Floyd’s Keep Talking

Single Malt Recommendation App

Life is too short to drink whisky you don’t like.

How often have you found yourself in a duty free shop in an airport, wondering which whisky to take back home? Unless you are a pro at this already, you might want something you haven’t tried before, but don’t want to end up buying something you may not like. The names are all grand, as Scottish names usually are. The region might offer some clue, but not so much.

So I started on this work a few years back, when I first discovered this whisky database. I had come up with a set of tables to recommend what whisky is similar to what, and which single malts are the “most unique”. Based on this, I discovered that I might like Ardbeg. And I ended up absolutely loving it.

And ever since, I’ve carried a couple of tables in my Evernote to make sure I have some recommendations handy when I’m at a whisky shop and need to make a decision. But then the tables are not user friendly, and don’t typically tell you what you should buy, and what your next choice should be and so on .

To make things more user-friendly, I have built this app where all you need to enter is your favourite set of single malts, and it gives you a list of other single malts that you might like.

The data set is the same. I once again use cosine similarity to find the similarity of different whiskies. Except that this time I take the average of your favourite whiskies, and then look for the whiskies that are closest to that.

In terms of technologies, I’ve used this R package called Shiny to build the app. It took not more than half an hour of programming effort to build, and most of that was in actually building the logic, not the UI stuff.

So take it for a spin, and let me know what you think.

 

Caffeine kick

Until June or July this year, I firmly believed that well-made South Indian filter coffee was the best form of coffee ever. This belief possibly had to do with my conditioning, having been exposed this to this coffee form from an extremely early age, and the belief sustained even in the face of pretty excellent coffees from quite a few artisanal “Aussie style” cafes here in London.

Then, around then, I decided to embark on “intermittent fasting”, which meant no calorie consumption from 8 in the night to the next noon (each day). The diet permitted me to drink coffee or tea in the mornings as long as no milk or sugar was added to it, and that presented a problem.

For South Indian filter coffee can’t be drunk black. The addition of the chicory, which slows down the pace through which water/steam filters through the beans in order to maximise flavour, adds its own flavour, which when unmasked by milk can be pretty revolting. Though I must mention that chicory powder is sold as a separate “health drink” here in the UK (maybe it needs to be marketed such because its taste is most revolting).

That I couldn’t add milk to my coffee meant that I needed to explore other ways of making good black coffee. Counter top space (or the lack of it) ruled out contraptions such as an espresso machine or even a Nespresso machine. There was an old Braun “coffee maker” (which my mother-in-law reportedly procured two decades ago) at home, but that dished out pretty bad coffee (which only Americans might appreciate).

And so I started exploring, asking around coffee-geek friends (not to be confused with the cafe of a similar name in Victoria). The French Press was quickly ruled out on account of taste. I strongly considered the Aeropress and the Hario V60, and in the spirit of “try before you buy” or even “learn before you buy”, I asked baristas at my favourite local artisanal cafe to show me how to brew in these methods.

I quite liked the output of both methods, but found the aeropress apparatus a bit cumbersome and hard to clean (one reason I didn’t want to use my trusty Bialetti Moka Pot to make non-South Indian coffee as well). The V60 on the other hand offered simplicity of making process as well as extreme ease of cleaning. So quickly after I had tried, I had bought the pourover cup from Amazon, and a bag of beans from Electric (they ground it for me) and I was ready to go.

I’ve since fallen in love with this form of coffee, though when I go to a cafe I order an espresso-based drink (Cortado/Piccolo or Flat White depending on the cafe). And though I gave up on intermittent fasting a month and half after I started it, I continue to make this (I’m sipping on one such cup as I type this). And this is because of the caffeine kick.

I think I had this realisation for the first time back when I was still fasting – I drank a cup of pourover coffee just before I hit the gym (on an otherwise empty stomach), and I was astounded by my own energy levels that day. And I have since tested this in several other situations – before meetings, while doing an important piece of work or simply to stay awake. The caffeine kick from pourover coffee is simply unparalleled compared to any other kind of coffee I’ve had (though espresso-based coffees in cafes come very close).

South Indian filter coffee optimises for flavour at the cost of the caffeine. The decoction is frequently stored for a long time, even overnight. The large amount of milk added means that a given amount of beans can be used to make several more cups. And the chicory addition means that brewing is slower and more flavour gets extracted from the beans, though it’s unlikely that the amount of caffeine extracted is proportionally large.

And all this together means you get incredibly tasty coffee, but not something you can get that much of a caffeine kick out of. And that is possibly why we are conditioned to drinking so many cups of coffee a day – you need so many cups to get the level of caffeine your body “needs” to function.

And this explains why South Indian filter coffee in the evenings has never interfered with my sleep, buy any coffee bought in a good cafe after 5pm has invariably led to sleepless nights!

Do you have anything else to add to this theory?

PS: The first time I made pourover coffee, I used Indian beans from Chickmaglur (that I bought here in the UK), so it’s not to do with the beans. It’s the extraction method.

Voice assistants and traditional retail

Traditionally, retail was an over-the-counter activity. There was a physical counter between the buyer and the seller, and the buyer would demand what he wanted, and the shopkeeper would hand it over to him. This form of retail gave greater power to the shopkeeper, which meant that brands could practice what can be described as “push marketing”.

Most of the marketing effort would be spent in selling to the shopkeeper and then providing him sufficient incentives to sell it on to the customer. In most cases the customer didn’t have that much of a choice. She would ask for “salt”, for example, and the shopkeeper would give her the brand of salt that benefited him the most to sell.

Sometimes some brands would provide sufficient incentives to the shopkeeper to ensure that similar products from competing brands wouldn’t be stocked at all, ensuring that the customer faced a higher cost of getting those products (going to another shops) if they desired it. Occasionally, such strategies would backfire (a client with extremely strong brand preferences would eschew the shopkeeper who wouldn’t stock these brands). Mostly they worked.

The invention of the supermarket (sometime in the late 1800s, if I remember my research for my book correctly – it followed the concept of set prices) changed the dynamics a little bit. In this scenario, while the retailer continues to do the “shortlisting”, the ultimate decision is in the hands of the customer, who will pick her favourite among the brands on display.

This increases the significance of branding in the minds of the customer. The strongest incentives to retailers won’t work (unless they result in competing brands being wiped out from the shelves – but that comes with a risk) if the customer has a preference for a competing product. At best the retailer can offer these higher-incentive brands better shelf space (eye level as opposed to ankle level, for example).

However, even in traditional over-the-counter retail, branding matters to an extent when there is choice (as I had detailed in an earlier post written several years ago). This is in the instance where the shopkeeper asks the customer which brand she wants, and the customer has to make the choice “blind” without knowing what exactly is available.

I’m reminded of this issue of branding and traditional retail as I try to navigate the Alexa voice assistant. Nowadays there are two ways in which I play music using Spotify – one is the “direct method” from the phone or computer, where I search for a song, a list gets thrown up and I can select which one to play. The other is through Alexa, where I ask for a song and the assistant immediately starts playing it.

With popular songs where there exists a dominant version, using the phone and Alexa give identical results (though there are exceptions to this as well – when I ask Alexa to play Black Sabbath’s Iron Man, it plays the live version which is a bit off). However, when you are looking for songs that have multiple interpretations, you implicitly let Alexa make the decision for you, like a shopkeeper in traditional retail.

So, for example, most popular nursery rhymes have been covered by several groups. Some do the job well, singing the rhymes in the most dominant tunes, and using the most popular versions of the lyrics. Other mangle the tunes, and even the lyrics (like this Indian YouTube channel called Chuchu TV has changed the story of Jack and Jill, to give a “moral” to the story. I’m sure as a teenager you had changed the lyrics of Jack and Jill as well :P).

And in this situation you want more control over which version is played. For most songs I prefer the Little Baby Bum version, while for some others I prefer the Nursery Rhymes 123 version, but there is no “rule”. And this makes it complicate to order songs via Alexa.

More importantly, if you are a music publisher, the usage of Alexa to play on Spotify means that you might be willing to give Spotify greater incentives so that your version of a song comes up on top when a user searches for it.

And when you factor in advertising and concepts such as “paid search” into the picture, the fact that the voice assistants dictate your choices makes the situation very complicated indeed.

I wonder if there’s a good solution to this problem.

Elegant and practical solutions

There are two ways in which you can tie a shoelace – one is the “ordinary method”, where you explicitly make the loops around both ends of the lace before tying together to form a bow. The other is the “elegant method” where you only make one loop explicitly, but tie with such great skill that the bow automatically gets formed.

I have never learnt to tie my shoelaces in the latter manner – I suspect my father didn’t know it either, because of which it wasn’t passed on to me. Metaphorically, however, I like to implement such solutions in other aspects.

Having been educated in mathematics, I’m a sucker for “elegant solutions”. I look down upon brute force solutions, which is why I might sometimes spend half an hour writing a script to accomplish a repetitive task that might have otherwise taken 15 minutes. Over the long run, I believe, this elegance will pay off, in terms of scaling easier.

And I suspect I’m not alone in this love for elegance. If the world were only about efficiency, brute force would prevail. That we appreciate things like poetry and music and art and what not means that there is some preference for elegance. And that extends to business solutions as well.

While going for elegance is a useful heuristic, sometimes it can lead to missing the woods for the trees (or missing the random forests for the decision trees if you may will). For there are situations that simply don’t, or won’t, scale, and where elegance will send you on a wild goose chase while a little fighter work will get the job done.

I got reminded of this sometime last week when my wife asked me for some Excel help in some work she was doing. Now, there was a recent article in WSJ which claimed that the “first rule of Microsoft Excel is that you shouldn’t let people know you’re good at it”. However, having taught a university course on spreadsheet modelling, there is no place to hide for me, and people keep coming to me for Excel help (though it helps I don’t work in an office).

So the problem wasn’t a simple one, and I dug around for about half an hour without a solution in sight. And then my wife happened to casually mention that this was a one-time thing. That she had to solve this problem once but didn’t expect to come across it again, so “a little manual work” won’t hurt.

And the problem was solved in two minutes – a minor variation of the requirement was only one formula away (did you know that the latest versions of Excel for Windows offer a “count distinct” function in pivot tables?). Five minutes of fighter work by the wife after that completely solved the problem.

Most data scientists (now that I’m not one!) ¬†typically work in production environments, where the result of their analysis is expressed in code that is run on a repeated basis. This means that data scientists are typically tuned to finding elegant solutions since any manual intervention means that the code is not production-able and scalable.

This can mean finding complicated workarounds in order to “pull the bow of the shoelaces” in order to avoid that little bit of manual effort at the end, so that the whole thing can be automated. And these habits can extend to the occasional work that is not needed to be repeatable and scalable.

And so you have teams spending an inordinate amount of time finding elegant solutions for problems for which easy but non-scalable “solutions exist”.

Elegance is a hard quality to shake off, even when it only hinders you.

I’ll close with a fairytale – a deer looks at its reflection and admires its beautiful anchors and admonishes its own ugly legs. Lion arrives, the ugly legs help the deer run fast, but the beautiful antlers get stuck in a low tree, and the lion catches up.