In defense of recommendation

I like listening to music a lot. On my desk right now there's a set of Beyerdynamic DT 770s, a popular model of studio headphones that you sometimes see people wearing in music videos. I also have a couple Sennheisers that I've used previously, starting with the HD 212s that I got when I was 14.

As a teenager, my music library expanded greatly after I discovered Pandora, the "personalized internet radio" service that queues up songs based on the seed tracks you select and the thumbs-up-or-down you give. Pandora introduced me to several bands that I still listen to heavily, like Breaking Benjamin, Evanescence and Rise Against. (Especially Breaking Benjamin.)

Pandora had an annoying flaw, though: it always ended up playing indie screamo bands. Why couldn't the algorithm figure out that I only like melodic hard rock? Exacerbating the problem (particularly as I got older) was the fact is that I didn't have much time or energy to manually curate my playlists or go search for new music regularly.

That experience kickstarted my interest in recommender systems, although I didn't know that's what they were called at the time. In college I was able to use various undergrad research opportunities as an excuse to work on my own music recommendation applications, while a technical writing course gave me the chance/a deadline to write a literature review of relevant research. A couple years later I even worked on a (now defunct) startup called Lagukan, which aimed to be a better version of Pandora.

I didn't make much progress, I think mostly due to my inexperience as a founder (e.g. I should've started out making a web frontend for Spotify instead of trying to make my own app). But I've stayed interested in recommendation generally, and each of my subsequent startup ideas have been recommender systems of one kind or another, including the one I'm currently working on.

But many people don't share my interest. In my experience, most commentary views recommender systems as part of the problem, not part of the solution. A common complaint (and in my opinion, the most substantive one) is that algorithmic recommendation often promotes shallow content with little genuine value—the information equivalent of junk food. This is part of Substack's core ideology, according to employee #1 Nathan Baschez: 

All the algorithms see are engagement metrics, they don’t care what they put in front of our faces. But what we read matters, and the most viral content is not the most valuable. Virality favors emotions like outrage, so it manufactures it from nothing if necessary, and we become addicted.

There's truth to that! I too have a great dissatisfaction with the recommender systems available to me. But I suspect that many who dismiss algorithms outright are throwing the baby out with the bath water. Chronological feeds are dominated by those who post frequently, and building an audience from scratch can take years. Recommender systems can help to solve problems like these. Nathan addresses this later in the piece:

In reality, I don’t think people actually want “full control” over what they read, because that’s too much work. What people are more likely to do is over-subscribe themselves, feel overwhelmed and stagnated in their content diet, and move onto a new platform if the old one doesn’t solve their problem for them. I don’t think most people are interested in painstakingly curating their feeds.

So while I do think algorithm-free solutions are valuable and worth building, I'd also love to encourage more people to experiment with new kinds of recommender systems. Recommendation doesn't have to be just a tool that big tech companies use to make a number on a spreadsheet go up. How can we develop algorithms that serve the interests of individuals? How can we make them surface hidden gems instead of whatever's trending? How can we make them resilient to bad actors? How do you create an organization where the incentives and culture support doing those things? Those are the questions I'd like to figure out!

 

Thanks for reading! This is an essay week; next time I'll be back with more half-formed thoughts about stuff I'm building. If you enjoyed this piece, I'd be greatly appreciative if you gave it a share (here's a link).