So after much deliberation and procrastination, I’ve finally started a newsletter. I call it “the art of data science” and the title should be self-explanatory. It’s pure unbridled opinion (the kind of which usually goes on this blog), except that I only write about one topic there.
I intend to have three sections and then a “chart of the edition” (note how cleverly I’ve named this section to avoid giving much away on the frequency of the newsletter!). This edition, though, I ended up putting too much harikathe, so I restricted to two sections before the chart.
I intend to talk a bit each edition about some philosophical part of dealing with data (this section got a miss this time), a bit on data analysis methods (I went a bit meta on this this time) and a little bit on programming languages (which I used for bitching a bit).
And that I plan to put a “chart of the edition” means I need to read newspapers a lot more, since you are much more likely to find gems (in either direction) there than elsewhere. For the first edition, I picked off a good graph I’d seen on Twitter, and it’s about Hull City!
Anyway, enough of this meta-harikathe. You can read the first edition of the newsletter here. In case you want to get it in your inbox each week/fortnight/whenever I decide to write it, then subscribe here!
And put feedback (by email, not comments here) on what you think of the newsletter!
A rather uncharitable view of the book advances that legacy publishing houses give out to established (non first-time) authors is to look at it as “convertible debt”, as this piece by Matthew Yglesias points out.
An advance is bundled with a royalty agreement in which a majority of the sales revenue is allocated to someone other than the author of the book. In its role as venture capitalist, the publisher is effectively issuing what’s called convertible debt in corporate finance circles — a risky loan that becomes an ownership stake in the project if it succeeds.
Now, as I consider possibly self-publishing my book, while simultaneously attempting to sell it to established publishing houses, I realise that apart from the convertible debt, publishing a book also involves a massive sale of equity.
I’ve finished the manuscript, and edited it once. It needs further editing, but I’ve put it off so far in the hope that I can sell the book to a mainstream publisher, who will then take care of the publishing. However, given that I might end up self-publishing, and from what I read that publishers don’t do a great job of editing anyway, I might need another pass or two.
And then there are other things to be done before the book comes out in print – a cover needs to be designed, illustrations need to be put in, maybe we should get someone to do an audiobook, and all such. Now, if a mainstream publisher picks up the book, I’d expect them to take care of these. Else I’ll need to spend to get people to do these things for me.
When people first told me that royalties in book publishing are of the order of 7.5% of cover price, it was a little hard to believe. However, looking at the costs involved in the publishing process, it’s not hard to see why publishers take the cut that they take. The problem, though, is that it involves you selling equity in your book.
By going for a mainstream publisher (rather than self-publishing), you are saving yourself the upfront cost of getting your book edited, designed and “typeset”, in exchange for a large portion of the equity of the book.
Looking at it in another way, you are trading in your limited downside (what you spend in designing, printing, etc.) for what might be a massively unlimited upside (in case my book is a runaway success). For the most part, considering that most books don’t do that well, it isn’t a very bad deal. However, considering that downside is limited (in terms of costs) I wonder if it makes sense to trade it in for a large stake of what could be a large upside.
In any case, the main reason I’m still pushing to get mainstream publishers is because the self-publishing market is a “market for lemons“. With barrier to entry not being too high, lots of bad books are self-published, and so anyone who thinks they’ve written a half decent book will try to find a mainstream publisher. And this further diminishes the average quality of self-published books. And further dissuades people like me from self-publishing!
It is now a well-documented fact (that I’m too lazy to google and provide links) that there exists a relationship between mental illness and creative professions such as writing.
Most pieces that talk about this relationship draw the causality in one way – that the mental illness helped the writer (or painter or filmmaker or whoever) focus and channel emotions into the product.
Having taken treatment for depression in the past, and having just finished a manuscript of a book, I might tend to agree that there exists a relationship between creativity and depression. However, I wonder if the causality runs the other way.
I’ve mentioned here a couple of months back that writing a book is hard because you are working months together with little tangible feedback, and there’s a real possibility that it might flop miserably. Soncequently, you put fight to make the product as good as you can.
In the absence of feedback, you are your greatest critic, and you read, and re-read what you’ve written; you edit, and re-edit your passages until you’re convinced that they’re as good as they can be.
You get obsessed with your product. You start thinking that if it’s not perfect it is all doomed. You downplay the (rather large) random component that might affect the success of the product, and instead focus on making it as perfect as you can.
And this obsession can drive you mad. There are days when you sit with your manuscript and feel useless. There are times when you want to chuck months’ effort down the drain. And that depresses you. And affects other parts of your life, mostly negatively!
Again it’s rather early that I’m writing this blog post now – at a time when I’m yet to start marketing my book to publishers. However, it’s important that I document this relationship and causality now – before either spectacular success or massive failure take me over!
While I look for publishers for the manuscript that I’ve just finished (it’s in “alpha testing” now), I think it’s a good time to write about what it was like to write the book. Now, I should ideally be writing this after it has been published and declared a grand success.
But there are two problems with that. Firstly, the book may not be a success of any kind. Secondly, it will be way too long after having finished it to remember what it was like to write it. In fact, a week after the first draft, I’ve almost already forgotten what it was like. So I’m writing this now.
Writing is a full-time job. I got this idea for the book in October 2014 when I was visiting Barcelona for the first time. I wrote the outline in November 2014. Despite several attempts to write, nothing came out of it.
During a break from work in October 2015 I managed to get started, but I’ve re-written all that I wrote then. Part-time effort doesn’t just cut it. It wasn’t until I came to Barcelona in February that I could focus completely on the book and write it.
You need discipline. This probably doesn’t need to be explicitly stated, but writing a book, unlike writing a blog post, is a fighter process, and you need a whole load of discipline and focus. After a week or two of preparing the outline, I prepared fairly strict deadline regarding when I would finish the book. I had to reset the deadline a couple of times, but finally managed it.
There is no feedback. I think I wrote about this a few days back. The big problem with writing a book is that you spend a significant amount of effort before even a small fraction of your customers have seen the product. So you soldier on without any feedback, and it can occasionally be damn frustrating.
You feel useless. Writing a book can introduce tremendous amounts of self-doubt. One day you think you’ve completely cracked it, and your book will change the world. The next day you start wondering if there’s any substance at all to what you’re writing, and there’s any point in going ahead with it. On several occasions, I’ve had thoughts on abandoning it.
Getting away helps. The only reason I didn’t abandon the book when I had my bouts of self-doubt was that I was away in Barcelona with nothing else to do. It wasn’t as if I could ditch the book and find some work to do the next day. Being away meant that the TINA factor pushed me on. There was no alternative but to write the book.
Getting in a draft is important. You are likely to have bad days when you’re writing. On those days you feel like giving up. On putting things off for another day. Reams have been written about great writers stalling their books for several days because they couldn’t find the “right word”. I don’t buy that.
Found that when I’m in a rut, it’s better I simply push through and finish the chapter. Editing it later on is far easier than writing it again from scratch.
There is a limit to how much you can write. When I said it’s a full time job you might think I spent 8 hours a day on the book. I took around 70 days to write it (including a 10-day vacation), and the draft weighs in at 75,000 words (I intend to cut it before publication). So it’s less than 1200 words per day on an average.
That doesn’t sound like a lot, but trust me, writing on a continuous basis is quite hard. A lot of time goes in fact checks and in getting links (I don’t think I still have all the footnotes and endnotes I need for the book). Writing a book is far more complex than writing a blog post.
Writing is tiring. This isn’t something I figured out while writing the 2000 odd posts I’ve put on this blog. When you’re writing a book, and for an audience, you realise that you get tired pretty quickly. I don’t think I was able to work more than four hours a day on any of my “writing days”. And four hour-days would leave me a zombie.
You need a schedule, and a workplace. I did the pseud romantic thing. The entire book was written at this WiFi enabled cafe near my place in Barcelona. Pseud value apart, the point of having the workplace was that it brought a schedule and some discipline to my days. I would go there every morning on writing days (exact time varied), get a coffee and sit down to write. And not rise until I had finished my target for the session.
Two days back I went there to work on something else. I figured I couldn’t – that cafe is now forever tied to my writing the book. The kind of focus required there was of a different kind.
I’ll stop for now. I hope to republish this blog post once the book has hit the stands!
When Dhoni was bantering with the Australian, the other journalists in the hall were laughing. They would, no sports journalist would want to be anything but nice to the formidable Indian captain. That’s why this piece had to be written by someone whose day job is to write on business and economy.
Looking at the reports of the incidents from both Sinha and EspnCricinfo’s standpoints, it is clear to me that Sinha’s view is more logical. That Dhoni’s calling of the journalist to the press conference table and cross-questioning him was unprofessional on the one hand and showed his lack of defences on the other.
Yet, the ending to Sinha’s piece also explains why other sports journalists have taken to lauding Dhoni’s view rather than critisicing him – for them, access to the Indian limited overs captain is important, and they wouldn’t like to damage that by taking an Australian colleague’s side.
The problem with a lot of sports journalism in general, and Indian cricket journalism in particular, is that jingoism and support for one’s team trumps objective reporting and analysis. One example of this was coverage from Indian and Australian newspapers of the Monkeygate scandal in 2007-08 (when Harbhajan Singh called Andrew Symonds a monkey).
More recently, there was the controversy about India losing games because of the tendency of Rohit Sharma (and Indian batsmen in general) to slow down in their 90s. Again, commentary about that took jingoistic tones, with the Indian sports media coming out strongly in favour of Sharma. There were reports defending his “commitment” and “grit” and all such flowery language sports journalists love, and that Glenn Maxwell’s comment was entirely unwarranted. Maxwell even backed down on his comments.
Data, however, showed that Maxwell need not have backed down on his comments. Some analysis based on ball-by-ball data that I published in Mint showed clearly that Indian batsmen do slow down in their 90s, and of all recent players, Sharma was the biggest culprit.
The piece was a hit and was widely shared on social media. What was more interesting, however, was the patterns in which it was shared. For one, the editors at Mint loved it and shared it widely. It was also shared widely by mango people and people with a general interest in cricket.
The class of people which was conspicuous by its absence of commentary on my piece was sports journalists. While it could be reasoned that they didn’t see the piece (appearing as it did in a business publication, though I did send emails to some of them), my reasoning is that this piece didn’t gain much traction among them because it didn’t fit their priors, and didn’t fit the jingoistic narrative they had been building.
It is not necessary, though, that someone only shares pieces that they completely agree with – it is a fairly common practice to share (and abuse) pieces which you vehemently disagree with. The commentary I found about this piece was broadly positive – few people who had shared the piece disagreed with it.
My (untested) hypothesis on this is that this analysis flew in the face of all that mainstream sports journalists had been defending over the previous few days – that Maxwell’s comments were simply not true, or that Sharma was a committed cricketer, and all such hyperbole. With data being harder to refute (only option being to poke holes in the analysis, but this analysis was rather straightforward), they chose to not give it further publicity.
Of course, I might be taking too much credit here, but that doesn’t take away from the fact that there is a problem with sports (and more specifically, cricket) writing. Oh, and as for the ultra-flowery language, I’ll save my comments for another day and another post.
I watched half of Exodus: Gods and Kings last night (I’d DVRd it a few days back seeing it’s by Ridley Scott). The movie started alright, and the story was well told. Of Moses’s fight with Rameses, of Moses being found out, of his exile and struggle and love story and finding god on a mountain. All very nice and well within the realms of good mythology.
And then Moses decides to hear god’s word and goes to Memphis to free his fellow Hebrews. There’s a conspiracy hatched. Sabotage begins. Standard guerrilla stuff that slaves ought to do to revolt against their masters. Up to that point in time I’d classified Exodus as a good movie.
And then things started getting bad. God told Moses that the latter wasn’t “doing enough” and god would do things his way. And so the Nile got polluted. Plants died. Animals died. Insects attacked. Birds attacked (like in that Hitchcock movie). What had been shaping up to be a good slave-revolt story suddenly went awry. The entire movie could be described by this one scene in Indiana Jones and Raiders of the Lost Ark:
When you see the guy twirling the sword, you set yourself up for a good fight. And then Indiana just pulls out a gun and shoots him! As a subplot in that movie, it was rather funny. But if the entire plot of a movie centres around one such incident (god sending the plague to Egypt, in this case), it’s hard to continue watching.
Checking out the movie on IMDB, I realised that it has a pretty low rating and didn’t recover its investment. While this is surprising given the reputation of Scott, and how the first part of the movie is set up and made, looking at the overall plot it isn’t that surprising. The problem with the movie is that it builds on an inherently weak plot, so the failure is not unexpected.
It did not help that I was reading mythology, or a realistic mythological interpretation, earlier in the day – the English translation of SL Bhyrappa’s Parva. In that, Bhyrappa has taken an already complex epic, and added his own degrees of complexity to it by seeking to remove all divinity and humanise the characters. Each major character has a long monologue (I’m about a third into the book), which explores deep philosophical matters such as “what is Dharma”, etc.
While moving directly from humanised philosophical myth to unabashedly religious story might have prevented me from appreciating the latter, it still doesn’t absolve the rather simplistic nature of the latter myth. I admit I’m generalising based on one data point, not having read any Christian myth, but from this one data point, it seems Christian myth seems rather weak compared to Hindu or Greek or Roman myth.
My explanation for this is that unlike other myths, Christian myth didn’t have enough time to evolve before it was written down. While the oral tradition meant that much valuable human memory was wasted in mugging up stories and songs, and that transmission was never exact, it also meant that there was room for the stories to evolve. Having been transmitted through oral tradition for several centuries, Hindu, Greek and Roman stories were able to evolve and become stronger. Ultimately when they got written down, it was in much evolved “best of” form. In fact, some of these myths got written down in multiple forms which allowed them to evolve even after writing came by.
While writing saves human memory space and prevents distortions, it leaves no room for variations or improvisation. Since there is now an “original book”, and such books are determined to be “words of God”, there is no room for improvisation or reinterpretation. So we are left with the same simplistic story that we started of with. I hope this explains why Exodus, despite a stud director, is a weak movie.
There are certain pieces that are a breeze to write – you start writing, the thoughts flow, the words flow, your fingers do the needful and before you know it you’ve written a thousand words. Once you’ve published you’re feeling all good and high and kicked to take on your next task for the day.
But then there’s writing that can drain you. For example I just wrote a piece on my policy blog. It took forty minutes to write. I kept hitting backspace and cancelling out sentences. It was extremely laboured. And having written that I’m feeling all completely drained out. This blog post on the other hand is unlikely to have that effect.
I realise that there are two kinds of pieces that I write – the first are “flow pieces” – where I do the thinking as I write. I start writing only with an initial sketch, or paragraph, or opening line. And then I build the piece linearly thinking as I write along. These pieces are a breeze and a pleasure to write. And it is incredible the number of insights I stumble upon while writing such pieces.
On the other hand you have “planned pieces” – where you know exactly what you want to communicate and how, and you only have to implement it and put it all together while you’re writing. The problem with such writing is that you would have imagined certain sentences at different points in time while thinking of the ideas and now you try and fit all those sentences into a coherent piece. And that leads to a lot of jigsaw-fitting and labouring and backspacing. It’s hell!
For a while I wanted to write Op-Eds, but I’ve now simply given up on those. It requires me to write in an impersonal formal voice which is something I find extremely hard to summon. And such pieces are more likely than not planned pieces, and writing them can be extremely draining. I’d rather write when I want to, building pieces as I please, and publishing them as blog posts! The effort to write Op-Eds is simply not worth it!
Andrew Gelman has a nice piece on why academic writing is bad. Basically two points – writing is hard, and academic pieces are not selected based on their quality of writing. So the quality of writing in such pieces is far inferior to say writing in a newspaper!