More on covid testing

There has been a massive jump in the number of covid-19 positive cases in Karnataka over the last couple of days. Today, there were 44 new cases discovered, and yesterday there were 36. This is a big jump from the average of about 15 cases per day in the preceding 4-5 days.

The good news is that not all of this is new infection. A lot of cases that have come out today are clusters of people who have collectively tested positive. However, there is one bit from yesterday’s cases (again a bunch of clusters) that stands out.

Source: covid19india.org

I guess by now everyone knows what “travelled from Delhi” is a euphemism for. The reason they are interesting to me is that they are based on a “repeat test”. In other words, all these people had tested negative the first time they were tested, and then they were tested again yesterday and found positive.

Why did they need a repeat test? That’s because the sensitivity of the Covid-19 test is rather low. Out of every 100 infected people who take the test, only about 70 are found positive (on average) by the test. That also depends upon when the sample is taken.  From the abstract of this paper:

Over the four days of infection prior to the typical time of symptom onset (day 5) the probability of a false negative test in an infected individual falls from 100% on day one (95% CI 69-100%) to 61% on day four (95% CI 18-98%), though there is considerable uncertainty in these numbers. On the day of symptom onset, the median false negative rate was 39% (95% CI 16-77%). This decreased to 26% (95% CI 18-34%) on day 8 (3 days after symptom onset), then began to rise again, from 27% (95% CI 20-34%) on day 9 to 61% (95% CI 54-67%) on day 21.

About one in three (depending upon when you draw the sample) infected people who have the disease are found by the test to be uninfected. Maybe I should state it again. If you test a covid-19 positive person for covid-19, there is almost a one-third chance that she will be found negative.

The good news (at the face of it) is that the test has “high specificity” of about 97-98% (this is from conversations I’ve had with people in the know. I’m unable to find links to corroborate this), or a false positive rate of 2-3%. That seems rather accurate, except that when the “prior probability” of having the disease is low, even this specificity is not good enough.

Let’s assume that a million Indians are covid-19 positive (the official numbers as of today are a little more than one-hundredth of that number). With one and a third billion people, that represents 0.075% of the population.

Let’s say we were to start “random testing” (as a number of commentators are advocating), and were to pull a random person off the street to test for Covid-19. The “prior” (before testing) likelihood she has Covid-19 is 0.075% (assume we don’t know anything more about her to change this assumption).

If we were to take 20000 such people, 15 of them will have the disease. The other 19985 don’t. Let’s test all 20000 of them.

Of the 15 who have the disease, the test returns “positive” for 10.5 (70% accuracy, round up to 11). Of the 19985 who don’t have the disease, the test returns “positive” for 400 of them (let’s assume a specificity of 98% (or a false positive rate of 2%), placing more faith in the test)! In other words, if there were a million Covid-19 positive people in India, and a random Indian were to take the test and test positive, the likelihood she actually has the disease is 11/411 = 2.6%.

If there were 10 million covid-19 positive people in India (no harm in supposing), then the “base rate” would be .75%. So out of our sample of 20000, 150 would have the disease. Again testing all 20000, 105 of the 150 who have the disease would test positive. 397 of the 19850 who don’t have the disease will test positive. In other words, if there were ten million Covid-19 positive people in India, and a random Indian were to take the test and test positive, the likelihood she actually has the disease is 105/(397+105) = 21%.

If there were ten million Covid-19 positive people in India, only one-fifth of the people who tested positive in a random test would actually have the disease.

Take a sip of water (ok I’m reading The Ken’s Beyond The First Order too much nowadays, it seems).

This is all standard maths stuff, and any self-respecting book or course on probability and Bayes’s Theorem will have at least a reference to AIDS or cancer testing. The story goes that this was a big deal in the 1990s when some people suggested that the AIDS test be used widely. Then, once this problem of false positives and posterior probabilities was pointed out, the strategy of only testing “high risk cases” got accepted.

And with a “low incidence” disease like covid-19, effective testing means you test people with a high prior probability. In India, that has meant testing people who travelled abroad, people who have come in contact with other known infected, healthcare workers, people who attended the Tablighi Jamaat conference in Delhi, and so on.

The advantage with testing people who already have a reasonable chance of having the disease is that once the test returns positive, you can be pretty sure they actually have the disease. It is more effective and efficient. Testing people with a “high prior probability of disease” is not discriminatory, or a “sampling bias” as some commentators alleged. It is prudent statistical practice.

Again, as I found to my own detriment with my tweetstorm on this topic the other day, people are bound to see politics and ascribe political motives to everything nowadays. In that sense, a lot of the commentary is not surprising. It’s also not surprising that when “one wing” heavily retweeted my article, “the other wing” made efforts to find holes in my argument (which, again, is textbook math).

One possibly apolitical criticism of my tweetstorm was that “the purpose of random testing is not to find out who is positive. It is to find out what proportion of the population has the disease”. The cost of this (apart from the monetary cost of actually testing) are threefold. Firstly, a large number of uninfected people will get hospitalised in covid-specific hospitals, clogging hospital capacity and increasing the chances that they get infected while in hospital.

Secondly, getting a truly random sample in this case is tricky, and possibly unethical. When you have limited testing capacity, you would be inclined (possibly morally, even) to use it on people who already have a high prior probability.

Finally, when the incidence is small, we need a really large sample to find out the true range.

Let’s say 1 in 1000 Indians have the disease (or about 1.35 million people). Using the Chi Square test of proportions, our estimate of the incidence of the disease varies significantly on how many people are tested.

If we test a 1000 people and find 1 positive, the true incidence of the disease (95% confidence interval) could be anywhere from 0.01% to 0.65%.

If we test 10000 people and find 10 positive, the true incidence of the disease could be anywhere between 0.05% and 0.2%.

Only if we test 100000 people (a truly massive random sample) and find 100 positive, then the true incidence lies between 0.08% and 0.12%, an acceptable range.

I admit that we may not be testing enough. A simple rule of thumb is that anyone with more than a 5% prior probability of having the disease needs to be tested. How we determine this prior probability is again dependent on some rules of thumb.

I’ll close by saying that we should NOT be doing random testing. That would be unethical on multiple counts.

Why Trump Will Retain Power

One piece of news that might have gone unnoticed in the middle of all this Covid19 news is that Bernie Sanders has suspended his campaign to be the Democratic nominee for this November’s American Presidential elections. So it looks highly likely that Joe Biden will take on fellow-septuagenarian Donald Trump.

Thinking about it, it doesn’t matter which Democrat takes on Trump. He is going to win. I suspect that Sanders realised this as the covid crisis was panning out, and so decided to fold.

Essentially what the Covid-19 crisis has been largely positive to things that American conservatives traditionally value, and showed the perils of some of the things that American “liberals” have traditionally valued. As a consequence of this, we will find that people who are on the margin (I’m told there are very few fence-sitting voters in the US, compared to India for example) are likely to shift more conservative.

In fact, everyone will become a little more conservative (in the American sense) after this crisis is over (though most Americans have such extreme political opinions that this won’t matter). And that means that in this year’s elections at least, the Republicans are going to win. So assuming he remains healthy, Trump has four more years in the Oval Office.

So what are these “conservative and liberal values” that influenced by this crisis? Let’s make a laundry list.

  • Borders: Open borders, at state and national level are a favourite of liberals (except, in the American context for some strange reason, for skilled labour immigration). They are great for economic growth, but also for pandemic growth. We are surely likely to see tougher border controls (maybe Brexit will be followed by Nordexit? Can’t be ruled out) continuing post this crisis.
  • Cities: Conservatives are all about urban sprawl, owning McMansions and commuting by car. Liberals bat for high density cities and public transport. The tail risk of high density cities as being higher risk for pandemic spread (which had largely been hidden following the rapid advances in medicine in the first half of the 20th century) has been exposed.
  • Families: When you are isolated you would rather be living with your family (“near and dear ones” as some like to put it). The lockdown has been hardest on people living alone or living “with roommates”. American conservatives are all about marrying early and staying married and “two parent families”, which means fairly low chances of living alone. On the margin, people are likely to rediscover “family”.
  • Individualism: Sort of related to the previous one. This is something that is likely to affect me as well. Liberals have been about “breaking free of the community” and living by and for yourself. Crises like this one make you realise the value of having a community, and cultivating relationships in good times that might come of use in bad. So we are likely to see less individualism.Related to this, liberals are far more likely than conservatives to cut ties with families on account of their political leanings. The pandemic might force a rethink on this.
  • Privacy: Countries that have managed to suppress the disease to great extent (such as Singapore, Taiwan, Hong Kong and South Korea) have done so by increasing surveillance on their citizens. As Raghu SJ wrote in this excellent blogpost, countries are facing an “impossible trilemma” in terms of protecting citizens’ privacy, containing the disease and protecting the economy.
    And he wholeheartedly agrees that privacy is the one thing that should be sacrificed now. I’m thinking he’s not alone. Moreover, instruments like Aadhaar and Aadhaar-linked bank accounts, which was vociferously opposed by privacy fundamentalists, can be of excellent use for fast direct transfer of benefits now (that India, which has this infrastructure, is only doing a tiny stimulus is another matter).
    Going forward, people will be more willing to trade off privacy (which a lot of us are already doing with Facebook, etc.) for superior service, and privacy fundamentalists will get less attention.

There are some mitigating factors as well.

  • Church attendances will go down, since religious gatherings have been shown to be a reliable source of infection spread.
  • The health crisis can mean that some sort of Obamacare might make a comeback.

On the balance, though, at least in the social sense, you can expect Americans to become more conservative. Move to smaller towns and suburubs (greater remote working will aid this), keep factories in the US (a favourite Trump theme) and become more family oriented. While all this may not last for too long, it should be enough to win Trump this year’s election.

It doesn’t matter how well or badly his government handles Covid-19.

I deliberately decided to not talk about India, since I’m not sure there’s that much of an ideological difference between political parties here. But similar trends, at the personal level, are likely to happen here as well.

The Two Overton Windows

If you want to appear intelligent when discussing something about public policy, you could do worse than uttering the phrase “Overton Window”. The Overton Window, “invented” by one Joseph Overton, suggests that there is a “range of policies acceptable to political mainstream”.

And so you frequently have political commentators talking about the Overton Window “shifting” whenever a new political idea (or person) comes to the fore. This was bandied about much when Modi became Prime Minister of India, or when Trump became President of the US, or when Jeremy Corbyn became the Labour Party leader.

While “shifting Overton window” is something you come across rather often in policy discourse, my argument is that with the rise of subscription media, the Overton window is not shifting as much as it is “splitting”. In other words, we now have not one but two Overton Windows.

Without loss of generality, let us call them the “Jamie Overton Window” and the “Craig Overton Window”. Since both the twins are right arm fast bowlers, it doesn’t matter which brother is associated with which Overton Window.

So how did we get here, and what does it mean for us?

We started with the classic Overton Window. Let’s assume that all politics can be reduced to one axis (if we do a Principal Component Analysis of political views, the principal axis is certain to account for a large share of the variance, so this is not a bad assumption). So the Overton Window can be referred to by a line which the shifts.

As long as the world was “ruled by mainstream media”, this Overton Window kept moving back and forth, expanding and contracting, but it remained united. And then with the start of subscription ad-free media (maybe a decade or decade and half ago), the Overton Window started expanding.

The “left media” (that’s a convenient term isn’t it?) started admitting stuff that was left to the then Overton Window. The “right media” started admitting stuff that was to the right of the then Overton Window. And so over time, the Overton Window started expanding. And things can’t get into the media Overton Window unless they’re part of the mainstream political Overton Window.

The thing is that as the media became subscription-heavy and hence biased, political ideas that were once on the fringe now got a voice. And so the Overton Window got larger and larger.

Until a point when it got so unwieldy that it split, giving rise to Jamie and Craig. The image on the right is an approximate illustration of what happened.

And once the Overton Window split, there was no looking back. They started moving away from each other well-at-a-faster-rate. The Jamies could not come to terms with the policies of the Craigs, and vice versa. Political analysts and commentators started getting associated with the Jamie and Craig camps.

For a while, a few commentators continued to write for both sides, but the extreme fringes, which were getting more and more extreme, started overreacting. “How can we have someone who has written 10 articles for Craigs write for us”, the Jamies asked. “Most of our commentators are Craigs, so we might as well become a Craig newspaper”, the other side reasoned.

And that’s where mainstream media is going. The Overton Window has split down the middle. Crossing this gap is considered a crime worse than crossing the floor in Parliament.

Sadly, it is not just media that is getting Jamie and Craig. Mainstream politics reflects this as well, and so across countries we get political opponents who just cannot talk to each other, since everything one says is outside the Overton Window of the other.

Maybe the only way this can end is by going across axes, or inventing a new axis even. With the current spectrum politics, there is no hope of the two Overton Windows coming to meet.

 

Distribution of political values

Through Baal on Twitter I found this “Political Compass” survey. I took it, and it said this is my “political compass”.

Now, I’m not happy with the result. I mean, I’m okay with the average value where the red dot has been put for me, and I think that represents my political leanings rather well. However, what I’m unhappy about is that my political views have been all reduced to one single average point.

I’m pretty sure that based on all the answers I gave in the survey, my political leaning across both the two directions follows a distribution, and the red dot here is only the average (mean, I guess, but could also be median) value of that distribution.

However, there are many ways in which people can have a political view that lands right on my dot – some people might have a consistent but mild political view in favour of or against a particular position. Others might have pretty extreme views – for example, some of my answers might lead you to believe that I’m an extreme right winger, and others might make me look like a Marxist (I believe I have a pretty high variance on both axes around my average value).

So what I would have liked instead from the political compass was a sort of heat map, or at least two marginal distributions, showing how I’m distributed along the two axes, rather than all my views being reduced to one average value.

A version of this is the main argument of this book I read recently called “The End Of Average“. That when we design for “the average man” or “the average customer”, and do so across several dimensions,  we end up designing for nobody, since nobody is average when looked at on many dimensions.

Dreamers and Dignity

If I’d picked up Snigdha Poonam’s Dreamers before I had read Chris Arnade’s Dignity, I might have liked it better. As it happened, having read Dignity, I found Dreamers to be unnecessarily judgmental and prescriptive, and was unable to read it beyond the first two chapters. It is now there on my goodreads page, as a book that I “finished” and gave one star.

Dignity is a book I highly recommend. Chris Arnade, a former investment banker with a PhD in astrophysics wanders around and hangs around in what he calls as “back row America”, and chronicles people’s lives there. The entire book is simply a set of chronicles, garnished with beautiful photos he has taken of his interviewees.

While Arnade makes no secrets of his own political leaning, he doesn’t let that affect his book. Rather, he keeps his own politics to the minimum and lets his interviews do the talking, literally. There are no policy prescriptions in the book, and the reader is simply presented a set of lives and asked to draw her own conclusions. And that means that even if you don’t agree with the politics of the author (I certainly don’t), the book is an incredibly compelling read.

I picked up Snigdha Poonam’s Dreamers about a month or so after I’d finished Dignity. The premise is sort of similar – except that given that India has recently had far higher growth than the US, the “back row Indians” can be classified as “dreamers” who are seeking a better life. And in this book, Poonam chronicles the stories of some of these dreamers, and what they are doing to get themselves a better life.

Poonam is clear about her politics as well (“my family has always voted for the Congress Party”), but what makes her book different from Arnade’s is that she lets her politics take over her narrative. While telling the story of Moin Khan, who runs a spoken English class in Ranchi, she doesn’t hesitate to make snide remarks about either the teacher or any of his students.

Rather than letting her characters talk, Poonam talks on their behalf and overlays her politics to pretty much everything she is talking about. “This is how you are expected to get ahead in Modi’s India” is a refrain through the book.

And even leaving the politics aside, what made me uncomfortable with Dreamers is that the author seems to talk down to the interviewees. The tone throughout the parts of the book that I read is one of moral superiority and smugness of being part of “front row India”.

Maybe if I had read Dreamers before I read Dignity, I would have appreciated it for what it is, and for the stories that it told. I might have discarded the politics and the tone and just enjoyed the stories (I see the book has got 4 stars on Goodreads from a lot of my friends).

Having read Dignity, however, I perhaps had this image in my head of how these stories can be told well. And that meant that I was simply unable to look beyond the overt politics and smug tone in Dreamers. And that meant I abandoned it midway, and gave it a low rating.

Baskets of deplorables

OK this is a political post. You might infer something about my political leanings from this, and you might classify me as a “deplorable”, but I run that risk.

I don’t like the way our politics is turning out nowadays. You are free to interpret “our” and “nowadays” in whatever way you want. What I don’t like is that people seem to wear their political beliefs on their sleeve, and think it is okay to shame and cut contact with people who don’t share their beliefs.

I don’t know when exactly this started – but it was surely sometime between 2013 and 2016. The culmination of this attitude was US Presidential candidate Hillary Clinton describing her opponent Donald Trump’s supporters as a “basket of deplorables“. And that attitude seems to be being taken forward by people of various political dispensations three years on.

I long for the days when people treated their political opinions like their private parts – stuff that existed and was put to good use when required, but not put on display. Nowadays, though, trawl through any social media platform, and you find people making political statements all the time. If you aren’t in a filter bubble, you will surely be seeing flamewars. And a difference in political opinion is no longer just a difference in opinion – you consider someone with differing views as despicable.

I’m friends with a lot of people who hold strong political opinions, and whose opinions might differ from mine. I don’t care about it – since there is plenty to them otherwise that makes them valuable to me, and so I continue to hang out with them.

Some people, on the other hand, don’t think like this. According to them, some political positions are so horrible that anyone who endorses that position is necessarily a horrible person, and not worth engaging with. For them, their political axis is fundamentally uni-dimensional – the world doesn’t exist outside of the dimension that they consider to be a dealbreaker.

As a consequence, any stand endorsed by a politician who endorses their dealbreaker position also becomes a dealbreaker. Political commentary and evaluation is based on who takes the stand, rather than the stand itself. Everything is seen through a political lens, and anyone who disagrees with them is worthy of ridicule.

It is sad that politics has taken over our lives so much, and people consider other people’s political opinions as such an important part of their lives. And the way social media and feedback loops work, I see no way out of this.

Mass marketing and objective journalism

This is a fascinating essay by Antonio García Martinez on the history and future of journalism (possibly paywalled). The money paragraph is this:

The bigger switch happened as a national market for consumer goods opened after the Civil War, when purveyors like department stores wanted to reach large urban audiences. Newspapers responded by increasing the number of ads relative to content, and switched to models that went light on the political partisanship in the interest of expanding circulation. This move was driven not exclusively by lofty ideals but also by mercenary greed. And it worked. Newspapers used to make lots of money. Mountains of money.

Basically, the move to objective journalism came in the late 1800s when advertisers such as Macy’s wanted to take out full page ads, and wanted to do so in newspapers that served the largest sections of the market. And when a newspaper had to reach a large section of the market, it inevitably had to tone down the partisanship, and become more objective.

Over the last decade, we have been witnessing (across the world) the decline of objective media. All media is “#paidmedia” based on which side of the political spectrum you stand on. There aren’t that many truly objective papers around, and social media is bombarded left and right by extremely politicised reporting that goes as “news”.

It is perhaps no coincidence that this period has coincided with a time when print circulation has been dropping steadily (in the developed world at least), and where online advertising can be highly targeted.

In theory, mass marketing is inefficient. When you pay to put up a hoarding somewhere, you’re possibly paying a small amount for each person who sees the hoarding, but not all of them might find it interesting. Consequently, this reflects in a depressed per-person price of the hoarding implying the owner of that real estate can’t make as much as she could if the hoarding were to be more “targeted”.

When you can target your advertisements more precisely, everybody wins. You as the marketer know that your advertisement is only being shown to your intended audience. The owner of the real estate where you put your advertisement can thus charge you more for your advertisement. Even the customer will be less pained by the advertisement if it is highly relevant to her.

Another way of seeing it is – an advertisement shown to a customer who doesn’t want to see it is wasted. The monetary cost of this waste are borne by the owner of the real estate and the advertiser, and the non-monetary cost is borne by the customer (being forced to see something she didn’t want to see). And so one of the biggest technological problems of today is on how we can target advertisements better so that we can minimise such costs – and in the last decade and half, we’ve made significant progress on that front.

The problem with greater efficiency, however, is that it comes with the side-effect of biased media. When Nike knows that it can precisely target an advertisement at American leftwingers, it makes an ad with Colin Kaepernick and shows them to American leftwingers to sell them more shoes.

This doesn’t however, mean that Nike only sells to left-wingers. The same company can make another advertisement targeted precisely at right-wingers and use it to sell shoes to them!

So now that you can make left-wing and right-wing ads, and you have the ability to target them, you want to cut the waste and place the ads so that you can target as best as possible. In other words, you want to place your left-wing ads in places that only left-wingers want to see, and right-wing ads only in places that right-wingers will see. And so you prefer to advertise in CNN and Fox rather than in a hypothetical “broad market” media outlet.

And the reason you created the politically charged ads in the first place was because there were some outlets (Facebook, for example) where you could precisely target people based on their political orientation. And so you see the vicious cycle – that you can target in some places means you want other places where you can target and that creates demand for more polarised media.

It was the opposite cycle that took effect in the late 1800s and early 1900s. There was no way brands could target (also, when you make physical advertisements, with 1900s technology, each advertisement is costly and you don’t want to make one per segment) too effectively, and so they went mass market in their communication.

And this meant advertising in the outlets that could get them the maximum number of eyeballs. When you can’t discriminate between a “right” and a “wrong” eyeball, you pay based on the number of eyeballs. And the way for media organisations to grow then was to cater to everyone. Which meant less less bias and more objectivity and more “features”.

Sadly that cycle is now behind us.

Government and markets

It’s been a while since I wrote a post like this one – I remember a decade ago, I used to flood my blog with such stuff.

In any case, last week, in response to the “10yearchallenge” meme, Nitin Pai of Takshashila wrote an Op-Ed in the Print on how India has changed in 10 years. While he admits that the country has grown and the lives of people has improved in some ways, the article leads with the headline that India should be be ashamed of what has happened in the last 10 years. This paragraph is possibly representative of the article:

While individual Indians seem to have done well over the past decade, India is more or less where it was. Worse, politics and policy priorities seem to have regressed to 1989.

Reading through the article (I encourage you to read it, it’s good – never mind the headline), I found a clear and distinct pattern in the kind of things where things have gotten better in India and where things have gotten worse.

Everything where markets function, or where the government doesn’t have much of a role, things have changed significantly for the better. Everything where the government has an outsized role, either because it is the government’s job or the sector is overregulated, things have gotten worse. So our cities have gotten more crowded. Infrastructure has gotten worse. Law and order has regressed. And this has had little to do with the party in power – whatever the government touched has regressed.

Looking at it in another way, Indians seem to be highly capable of making their lives better by coordinating using the invisible hand of the market. However, we seem incapable of making our lives better by coordinating using the government process.

From this perspective, there is one easy way to progress – basically reduce the government. Get rid of the overregulations. Get the government out of things where it shouldn’t be. Give a freer hand to the market.

Unfortunately, ahead of general elections this year, we see most parties taking a highly statist line. This is a real tragedy.

On liberalism and government control

My first exposure to political ideologies took place in 2004, when I joined the now-defunct (but then brilliant) social networking site Orkut. While filling up my personal details, I was asked to pick my political beliefs from a drop-down.

It had things such as “left-liberal”, “very left-liberal”, “right-conservative”, etc. Now, while I considered myself liberal back then (I’ve moved far more liberal on personal freedom issues since then), there was no way I could describe myself as “left”, since I’ve always been a free market fundamentalist. Finally I noticed there was something called “libertarian” in the dropdown, and assumed it might stand for my beliefs and chose that. In hindsight, it turns out I was right (no pun intended).

A year or two later, I got introduced to a “libertarian cartel” (I was never a member, so don’t know who were members). Presently, I was invited to join some of them in discussions, and my love for the libertarian philosophy grew (these discussions were instrumental in me moving far more liberal on personal freedom issues). Yet, looking around the political spectrum, you had few libertarian parties (going across countries).

You had the set of parties that can be broadly classified as “Republican” which allowed you to do business the way you liked, but sought to restrict personal freedoms. And there were the parties that can be classified as “Democrat” which promoted personal freedoms, but restricted how you could do business. And you had philosophies such as communism which sought to control both. The “fourth quadrant” was (and is) largely empty.

It is not hard to understand why this fourth quadrant is empty – in exchange for responsibilities of governing, politicians desire power, and this power can only come at the cost of restricting freedoms of the constituents. Different political formations choose to exercise this power along different axes, but little differentiates them – they all seek to control. While libertarianism is appealing for the constituent, it doesn’t make sense for politicians since it doesn’t compensate sufficiently for the responsibility of  governance. Hence you don’t find libertarian political parties.

Yet, we find that slowly but surely, reforms do happen. Over time, restrictions on freedoms (both personal and economic) do get relaxed, albeit at a glacial pace, and this is true across countries, despite there being no “libertarian” politicians. Why does this happen?

The simplistic answer is that politicians in functioning democracies have to face lengthy periods of time in opposition, when they are at the mercy of the party that is then in power. Since politicians tend to be vindictive animals, you don’t want to leave behind any laws that might be used to harass you while you are out of power. So the ruling party should tend to ease restrictions that can be used against its members when they are out of power.

Again, this is fine in theory, but why does it not always happen? The answer is that opposing political parties are not “orthogonal enough”. If politicians on multiple sides of the divide have broadly similar ideas on certain issues, there can be a tacit understanding (a “doctrine of no first use”, perhaps) to not use the laws that they agree on against each other.

When you have parties that have orthogonal philosophies, you can expect them to do their bit while in power to undermine the sources of their rivals’ control, so that their rivals might enjoy less control the next time they are in power. And citizens in such democracies are likely to enjoy greater freedoms.

As the old saying (paraphrased) goes, “when politicians from all parties agree to something, it is unlikely to be in the interests of the people”.