Tag: measurement

Metrics

Over the weekend, I wrote this on twitter:

 

Surprisingly (at the time of writing this at least), I haven’t got that much abuse for this tweet, considering how “test positivity” has been held as the gold standard in terms of tracking the pandemic by governments and commentators.

The reason why I say this is a “shit metric” is simple – it doesn’t give that much information. Let’s think about it.

For a (ratio) metric to make sense, both the numerator and the denominator need to be clearly defined, and there needs to be clear information content in the ratio. In this particular case, both the numerator and the denominator are clear – latter is the number of people who got Covid tests taken, and the former is the number of these people who returned a positive test.

So far so good. Apart from being an objective measure, test positivity ratio is  also a “ratio”, and thus normalised (unlike absolute number of positive tests).

So why do I say it doesn’t give much information? Because of the information content.

The problem with test positivity ratio is the composition of the denominator (now we’re getting into complicated territory). Essentially, there are many reasons why people get tested for Covid-19. The most obvious reason to get tested is that you are ill. Then, you might get tested when a family member is ill. You might get tested because your employer mandates random tests. You might get tested because you have to travel somewhere and the airline requires it. And so on and so forth.

Now, for each of these reasons for getting tested, we can define a sort of “prior probability of testing positive” (based on historical averages, etc). And the positivity ratio needs to be seen in relation to this prior probability. For example, in “peaceful times” (eg. Bangalore between August and November 2021), a large proportion of the tests would be “random” – people travelling or employer-mandated. And this would necessarily mean a low test positivity.

The other extreme is when the disease is spreading rapidly – few people are travelling or going physically to work. Most of the people who get tested are getting tested because they are ill. And so the test positivity ratio will be rather high.

Basically – rather than the ratio telling you how bad the covid situation is in a region, it is influenced by how bad the covid situation is. You can think of it as some sort of a Schrödinger-ian measurement.

That wasn’t an offhand comment. Because government policy is an important input into test positivity ratio. For example, take “contact tracing”, where contacts of people who have tested positive are hunted down and also tested. The prior probability of a contact of a covid patient testing positive is far higher than the prior probability of a random person testing positive.

And so, as and when the government steps up contact tracing (as it does in the early days of each new wave), test positivity ratio goes up, as more “high prior probability” people get tested. Similarly, whether other states require a negative test to travel affects positivity ratio – the more the likelihood that you need a test to travel, the more likely that “low prior probability” people will take the test, and the lower the ratio will be. Or when governments decide to “randomly test” people (puling them off the streets of whatever), the ratio will come down.

In other words – the ratio can be easily gamed by governments, apart from just being influenced by government policy.

So what do we do now? How do we know whether the Covid-19 situation is serious enough to merit clamping down on people’s liberties? If test positivity ratio is a “shit metric” what can be a better one?

In this particular case (writing this on 3rd Jan 2022), absolute number of positive cases is as bad a metric as test positivity – over the last 3 months, the number of tests conducted in Bangalore has been rather steady. Moreover, the theory so far has been that Omicron is far less deadly than earlier versions of Covid-19, and the vaccination rate is rather high in Bangalore.

While defining metrics, sometimes it is useful to go back to first principles, and think about why we need the metric in the first place and what we are trying to optimise. In this particular case, we are trying to see when it makes sense to cut down economic activity to prevent the spread of the disease.

And why do we need lockdowns? To prevent hospitals from getting overwhelmed. You might remember the chaos of April-May 2021, when it was near impossible to get a hospital bed in Bangalore (even crematoriums had long queues). This is a situation we need to avoid – and the only one that merits lockdowns.

One simple measure we can use is to see how many hospital beds are actually full with covid patients, and if that might become a problem soon. Basically – if you can measure something “close to the problem”, measure it and use that as the metric. Rather than using proxies such as test positivity.

Because test positivity depends on too many factors, including government action. Because we are dealing with a new variant here, which is supposedly less severe. Because most of us have been vaccinated now, our response to getting the disease will be different. The change in situation means the old metrics don’t work.

It’s interesting that the Mumbai municipal corporation has started including bed availability in its daily reports.

Collateralized Death Obligations

When my mother died last Friday, the doctors at the hospital where she had been for three weeks didn’t have a diagnosis. When my father died two and a half years back, the hospital where he’d spent three months didn’t have a diagnosis. In both cases, there were several hypotheses, but none of them were even remotely confirmed. In both cases, there have been a large number of relatives who have brought up the topic of medical negligence. In my father’s case, some people wanted me to go to consumer court. This time round, I had signed several agreements with the hospital absolving them of all possible complications, etc.

The relationship between the doctor and the patient is extremely asymmetric. It is to do with the number of counterparties, and with the diversification. If you take a “medical case”, it represents only a small proportion of the doctor’s total responsibility – it is likely that at any given point of time he is seeing about a hundred patients, and each case takes only a small part of his mind space. On the other hand, the same case represents 100% for the patient, and his/her family. So say 1% on one side and 100% on the other, and you know where the problem is.

The medical profession works on averages. They usually give a treatment with “95% confidence”. I don’t know how they come up with such confidence limits, and whether they explicitly state it out, but it is a fact that no disease has a 100% sure shot cure. From the doctor’s point of view, if he is administering a 95% confidence treatment, he will be happy as long as his success rate is over that. The people for whom the treatment was unsuccessful are just “statistics”. After all, given the large number of patients a doctor sees, there is nothing better he can do.

The problem on the patient’s side is that it’s like Schrodinger’s measurement. Once a case has been handled, from the patient’s perspective it collapses to either 1 or 0. There is no concept of probabilistic success in his case. The process has either succeeded or it has failed. If it is the latter, it is simply due to his own bad luck. Of ending up on the wrong side of the doctor’s coin. On the other hand, given the laws of aggregation and large numbers, doctors can come up with a “success rate” (ok now I don’t kn0w why this suddenly reminds me of CDOs (collateralized debt obligations)).

There is a fair bit of randomness in the medical profession. Every visit to the doctor, every process, every course of treatment is like a toin coss. Probabilities vary from one process to another but nothing is risk-free. Some people might define high-confidence procedures as “risk-free” but they are essentially making the same mistakes as the people in investment banks who relied too much on VaR (value at risk). And when things go wrong, the doctor is the easiest to blame.

It is unfortunate that a number of coins have fallen wrong side up when I’ve tossed them. The consequences of this have been huge, and it is chilling to try and understand what a few toin cosses can do to you. The non-linearity of the whole situation is overwhelming, and depressing. But then this random aspect of the medical profession won’t go away too easily, and all you can hope for when someone close to you goes to the doctor is that the coin falls the right way.