Algorithmic curation

When I got my first smartphone (a Samsung Galaxy Note 2) in 2013, one of the first apps I installed on it was Flipboard. I’d seen the app while checking out some phones at either the Apple or Samsung retail outlets close to my home, and it seemed like a rather interesting idea.

For a long time, Flipboard was my go-to app to check the day’s news, as it conveniently categorised news into “tech”, “business” and “sport” and learnt about my preferences and fed me stuff I wanted. And then after some update, it suddenly stopped working – somehow it started serving too much stuff I didn’t want to read about, and when I tuned (by “following” and “unfollowing” topics) my feed, it progressively got worse.

I stopped using it some 2 years back, but out of curiosity started using it again recently. While it did throw up some nice articles, there is too much unwanted stuff in the app. More precisely, there’s a lot of “clickbaity” stuff (“10 things about Narendra Modi you would never want to know” and the like) in my feed, meaning I have to wade through a lot of such articles to find the occasional good ones.

(Aside: I dedicate about half a chapter to this phenomenon in my book. The technical term is “congestion”. I talk about it in the context of markets in relationships and real estate)

Flipboard is not the only one. I use this app called Pocket to bookmark long articles and read later. A couple of years back, Pocket started giving “recommendations” based on what I’d read and liked. Initially it was good, and mostly curated from what my “friends” on Pocket recommended. Now, increasingly I’m getting clickbaity stuff again.

I stopped using Facebook a long time before they recently redesigned their newsfeed (to give more weight to friends’ stuff than third party news), but I suspect that one of the reasons they made the change was the same – the feed was getting overwhelmed with clickbaity stuff, which people liked but didn’t really read.

Basically, there seems to be a widespread problem in a lot of automatically curated news feeds. To put it another way, the clickbaity websites seem to have done too well in terms of gaming whatever algorithms the likes of Facebook, Flipboard and Pocket use to build their automated recommendations.

And more worryingly, with all these curators starting to do badly around the same time (ok this is my empirical observation. Given few data points I might be wrong), it suggests that all automated curation algorithms use a very similar algorithm! And that can’t be a good thing.

Curation mechanisms

The one thing that is making my stay away from twitter (Flipboard is also gone now, since the iPad has been returned to its rightful owner – the wife) hard is the fact that I’m unable to find a reliable alternate means of curating content. Let me explain.

Basically, how do you find interesting stuff to read? I’m talking about article length pieces here (500-5000 words), and not books – the latter are “easy” in terms of how they’re packaged, etc. Fifteen years back it was quite simple, and not all that simple – in order to find a good piece of writing you needed to be subscribed to the periodical in which it was published.

So you would subscribe to periodicals as long as they published good pieces once in a while – at least for the option value of finding such pieces. This meant that sales of periodicals was inflated – a handful of good pieces here and there would support significant subscription numbers, and they did rather well. Then the internet changed all that.

The beauty of the internet is unbundling – you can read one piece from a periodical without reading the fluff. Even periodicals that have a subscription paywall usually offer a certain number of articles (not certain number of editions, note) free before you pay up. This has turned the magazine business topsy-turvy – if you only have the odd good piece that appears in your magazine, people are going to find it somehow, and are not going to bother subscribing to your magazine just so that they can find it!

The question, thus, arises as to how you can find good pieces that are of interest to you without subscribing to whole magazines themselves (and considering the number of sources from which I’ve consumed content even in the last two weeks it’s impossible to subscribe to all of them).

Close to ten years back you got it by way of an RSS reader – you essentially subscribed to entire periodicals or well-defined subsets of them. You didn’t pay for the subscription and there was no paper – the pieces would come and fall in your “RSS feed”. Feed readers such as Bloglines and Google Reader became big in the mid noughties (I remember switching from the former to the latter in 2006 or something).

You used these readers to subscribe to blogs of interesting people (back then a lot of interesting people blogged), and these blogs would link out to other interesting content, and you would consume it all. Then Google Reader began this thing called “shared items” – where you could share items from your RSS feeds with your Google Talk friend list. This improved curation – for example, I knew that there was this friend who would share all interesting posts from a particular blog, so I didn’t need to subscribe to that blog’s RSS feed any more. Soon you could share items apart from those on your RSS feed – any interesting website you came across, you could share. It was beautiful.

And then in its infinite wisdom, Google decided to kill Google Reader! Like that. Gone.

Thankfully by then we had twitter, where among other things people would share interesting stuff. And there would be enough of those posted through the day every day to keep you busy! All the buried content in the world now started getting dug up thanks to twitter. There was always tonnes of interesting stuff.

But then it comes with a remarkably high degree of outrage – no one can simply share a link any more – there has to be commentary that is outraging about something or the other. The question, thus, is about how we can consume content from twitter without the outrage. That leads to apps such as Flipboard, which presents the content in an interesting format. There was a similar app I tried to write but gave up on.

Now that I don’t have access to flipboard any more (while flipboard for Android is nice, it’s not anything like flipboard for ipad) how do I curate content? How do I get interesting stuff recommended to me without having to trawl infinite websites?

The app that I think is well placed for such curation is Pocket – where you can store articles for reading later. But then its native sharing application isn’t too good. It in fact encourages you to share via twitter and email! If only Pocket can improve upon its native sharing, and thus build a social network around the shared content, it is possible that we could have something like Google Reader shared items once again!

But with everyone on twitter is there a market for this?

A month of detox

I cheated a little bit this morning. Since it’s been a month now since I got off Twitter and Facebook, I logged in to both for about a minute each, to check if I have any messages. The ones on Facebook weren’t of much use – just some general messages. There was one DM on twitter which had value, and I sent the guy an email explaining I don’t use twitter any more. I presently logged out.

The one month off Twitter and Facebook has so far gone off fantastically. For starters it’s given me plenty of time to read, meet people, talk to people and other useful stuff. And apart from some interesting links that people post on Twitter, I haven’t really missed either of them.

There have been times when there have been thoughts that would have earlier led to a tweet. However, given that the option exists no more, I end up doing one of two things – if there is substance to the tweet and I can elaborate on it, then I do so and it results in a blog post (you must have noticed that the frequency of blogging has gone up significantly in the last one month).

If it’s not really blog worthy but just something that I want to share with someone, I think of whose attention I would have liked to have caught by putting that tweet. In most cases I have found that there is a small set of people whose attention I would have liked to catch with a tweet – every time I tweeted, I would think of how a particular set of people would respond. So what I do when I have something to say and a particular set of people to say it to is to just message it to them.

While this gives a much better chance of them responding to the message than if they just saw it on their timeline (or missed seeing it), it also has the added benefit of starting conversations. Which is not a bad thing at all. In the last one month I’ve seen that my usage of WhatsApp and Google Talk has gone up significantly.

The only thing I miss about twitter is the interesting links that people post. I’ve tried a few things to remedy that. Firstly I tried to see if I could write a script that crawls my timeline, gets popular links (based on a set of defined metrics), and then bookmarks the top five each day. I went some way with the code (pasted below the fold here) but couldn’t figure how to post the linked articles to Pocket (my article bookmarker of choice). So I ended up tweeting those chosen links (!!) with a #looksinteresting hashtag, so that ifttt does the job of adding to Pocket.

It went for a bit till multiple people told me the tweets were spammy. And then I realized I needed to tweak the algorithm, and it needed significant improvement. And then I realized the solution was at hand – Flipboard.

If you have an android phone or an iPad and not used FlipBoard you’re really missing something. it’s a great app that curates articles based on your indicated areas of interest and history, and one of the sources it can get links from is your own Twitter and Facebook accounts. It is generally good in terms of its algo and good links usually bubble up there.

When I went off Twitter and Facebook on the 6th of August (in a fit of rage, outraged by all the outrage and negativity on the two media) I wanted complete isolation. And thus I deleted Twitter and Facebook from my FlipBoard also. Now I realized that adding back twitter on FlipBoard will allow me to access the nice links shared there without really getting addicted back to twitter, or partaking all the outrage.

For the last two weeks it’s worked like a charm. That twitter is present only on FlipBoard, which I use not more than twice a day (once in the morning, once at night), means that I’ve had the best of both worlds. And not being on twitter has meant that i’ve been able to get a fair bit of work done, finished three books (my first attempt at reading fiction in ten years fizzled out midway, though – Joseph Conrad’s Heart of Darkness failed to sustain my interest beyond about 40% (I have it on kindle) ), written dozens of blog posts across the three blogs and had more meaningful conversations with people.

I hereby extend my sabbatical from Twitter and Facebook for another month.

Below the fold is the code I wrote. It’s in R. I hope you can make some sense of it.

Continue reading “A month of detox”

Leaving twitter

It has been over three weeks since I signed off twitter. On January 1st, I had left this message on the social network:


After that I logged off twitter on all my computers, deleted tweetdeck from chrome and deleted the twitter app from my phone and iPad. I haven’t changed my twitter password, though, so every time I write a blog post wordpress will send an automatic notification tweet (it is likely that some of you are reading this via the automatic wordpress twitter notification).

The reason I logged off twitter was that I was getting addicted. Every time I had a minute or two of free time I would go check tweets. I was constantly on twitter all my waking hours. I would wake up in the morning to the alarm on my phone, and the first thing I would do was to check twitter. It is not unfair to say that twitter had consumed me.

Hence the effort to log off and delete the apps. So far I haven’t faced any withdrawal symptoms. There are times when I pull out my phone and instinctively go for the twitter app. And then I realize it’s not there, and curb my instincts. While working if I need a break I look for tweetdeck in my Chrome, but then realize it is not there.

So far, Facebook has been a good substitute. The advantage of facebook over twitter is that the former has a much more slow-moving news feed. If you check facebook after an hour or two, there will be two or three status updates on your timeline. Essentially when you instinctively click on facebook, it doesn’t become as much of a time sink as twitter used to.

One of  the reasons I would check twitter was for interesting links and articles. In the last 2-3 years some of the best stuff I’ve read online has been recommended to me by people on twitter. However, I have a way of accessing that without accessing twitter itself – I use this app called Flipboard (on both Android and iPad) and that curates articles that have been recommended by several of my followees and shows them to me. I check Flipboard approximately once a day, read some articles and bookmark some others. Thanks to that, I only get the article content on twitter without all the inane commentary and the PJs.

In my last month on twitter, I had logged off for a day on two-three separate occasions. The problem with twitter of late is that it is turning into yet another TV news channel. When there is an event of some interest, all the diversity on your timeline disappears, and everyone starts talking about the same thing. For a while it is good, for you get different perspectives. And then there is more and more of the sameness and can absolutely drive you nuts.

There is one reason I miss twitter though – for sharing articles. For a long time now I’ve liked to share interesting pieces that I’ve read. Back when Google Reader existed and had the “shared items” feature enabled, a number of people requested to be my GTalk friends just so that they could look at my curated “Google Reader Shared Items” content. Since that feature was taken off, though, I’ve resorted to twitter for sharing interesting articles. Now that I’m off twitter (technically Flipboard and Feedly (my RSS feed reader) allow me to share things on Twitter without logging on, but I don’t want to do that) I need another way.

Facebook doesn’t work, since most facebook friends are of a personal kind and won’t particularly be interested in articles on financial hedging (for example) or football formations. I’m not on any of these link sharing systems such as digg or delicious (assuming I’ve understood correctly how those two work), and I dn’t want to add another social network which can be yet another source of distraction. Hence, I’ve come up with an ingenious solution.

Back when Google took off the sharing feature from Reader, their recommendation was that we use Google Plus instead for sharing links. And that is exactly what I use that social network for. I never log on to that, but every time I read something interesting, it goes there. People say Google Plus is like shouting into an empty room. I don’t know (and don’t care) who reads the links I put there. I don’t share links for popularity. I share it because I think someone might find them interesting.

When I first got off twitter, people told me my resolution won’t last. It’s been three weeks already and i’m happy the way things are. I’m much less distracted, and can work better. I have a lot more time to myself. Time that would earlier be spent saying inane things on twitter is now spent in deep thought – and that is a good thing. I used to be a big fan of long lonely walks. Constant interaction on twitter means I don’t do those any more. But now I get more time for myself. On an auto rickshaw ride to meet some friends last evening for example, I just looked around and thought. It was wonderful!

I don’t rule out ever getting  back to twitter but  I don’t see myself doing so in the near future unless there is a very strong reason, and unless I know I won’t get addicted again. Till both these events happen, I remain away from that social network.