It’s that time of the year again. On the first day of December, everyone unwraps their “Wrapped” lists from music streaming app Spotify. Every person you follow seems to be screenshotting their top artists and songs, and floating them across social media, providing a condensed aural history of their year.
This year, Spotify’s personalized rundown of your 2021 soundtrack features your most-played artists, songs, genres and even a music mood board in a click-through interactive story. You can now also “blend” your year of music with that of a friend’s into a melded playlist. (The mood board and blended playlists are new this year.)
All of this raises the question: What’s the idea behind showing users how they consumed music? A clue might lie in a study the Spotify research team wrote about last May. The researchers showed 10 users their personal data profiles based on their Spotify account that had information about their top songs (of the past month and all time), top genres, how many playlists they created, and when they listened to Spotify. They discovered that laying out a user’s personal listening history data actually allowed them to “reflect on their identities as listeners,” and it let them see if they were only listening to music while working, or if they had periods of intense obsession with a specific artist.
We care about this kind of insight even if intuitively, we know that it’s just music, and what’s more, we all know that we’re not the only fans of Taylor Swift or Lorde. It might be that seeing a narrative (now with a hint of an emotional arc) created around the songs that shaped your year always feels a bit personal, and at times, revealing. (If you feel up to it, you can let an outside AI judge your Spotify). A FiveThirtyEight writer once mused that Spotify seemed to know him better than he knew himself.
So, how exactly is Spotify doing this? We know that they have reams of data they’ve collected from listeners (381 million monthly active users at last tally). Here’s what kind of analyses they’ve been running behind the scenes to understand what their users like to hear.
From music library to music discovery
When Spotify was first founded in 2006, its aspiration was to be a music library. Personalization came later, when the app’s engineers realized that enabling people to discover new music that they might like could elevate their experience. And this could be done by feeding an algorithm information about a user’s listening history, music choices, duration of play for certain songs, and how they respond to recommendations (are they liking them, skipping them, replaying them, saving them).
“Personalization was an empowering experience for listeners who didn’t have the time or knowledge to create endless unique playlists for every dinner party or road trip,” Oskar Stål, Spotify’s VP of Personalization, said in an October 2021 blog post. “It opened up discovery on a broader level, enabling hundreds of artist discoveries per person per year.”
[Related: A brief history of shuffling your songs, from Apple to Adele]
Their approach to this type of personalization hinged on two main areas of research: user modeling and intricate sonic analysis. Spotify attempts to model user behavior on the app by figuring out methods to project in-app activities into human traits and emotion, and tethering music experiences to mood and situational contexts like time of day, week, or season. Knowing this could allow them to change what they recommend on a Friday night versus a Tuesday afternoon. Recommended playlists can pop up on carousels across your home screen; there are personalized playlists like Discover Weekly, Daily Mix, and Radio playlists.
Plus, a new feature called “enhance” allows you to retrieve recommendations within a playlist you’ve already created, and just this week, Stål said in a video presentation that the team at Spotify was considering an approach that employed a human editor alongside the machine learning algorithm to create audio experiences that could maybe mix and match songs with podcasts and others. Spotify has even been testing a neural network called CoSeRNN that weighs certain features such as past listening history and current context to suggest song recommendations that suit the moment.
As for testing whether music mirrors certain human characteristics, they published results from a small survey-based study last December to see how music preference matched up to certain personality traits. In a blog post, the researchers noted that there appeared to be some correlations between personality and music genre preference. Unsurprisingly, people who identified as “open to new experiences” checked out Discover Weekly more; those who identified them as extroverts listened more to playlists that other people had created, while those who identified as introverts preferred to delve into a newly discovered artist’s discography.
Diving deep into the makeup of your perfect playlist
The team at Spotify appears to be constantly thinking up new ways to sort and recommend different types of music to its users. To get there, first they have to take the different kinds of data that they collect and build models that can analyze, compare, contrast, sort, and group the variety of information they’re getting. Researchers from the company noted in a 2016 paper that they scan the web for artist information and words used in online reviews to describe specific songs. They construct algorithms that can dissect the sound structures of songs and analyze how songs are related by scanning the billions of user-generated playlists already on the platform. And, they approximate a given user’s music taste by analyzing their historical and real-time listening patterns.
Discover Weekly, for example, is powered by close analysis of the songs a user has recently listened to and a scan of all playlists that may contain those songs or songs like them. Spotify uses a machine learning tool called the approximate nearest-neighbor search algorithm to group songs and users together based on shared attributes or qualities.
[Related: This new AI tool from Google could change the way we search online]
“Imagine you and another person have four of the same top artists, but your fifth artists are different. We would take those two near-matches and think, ‘Hmm, maybe each person would like the other’s fifth artist’ and suggest it,” Stål explained in a blog. “Now imagine that process happening at scale—not just one-on-one, but thousands, millions of connections and preferences being considered instantaneously.”
Spotify, for its part, has done a great deal of math separating a song into its individual instrument layers, breaking down its rhythm and structure. In November, the music service published research proposing a new personalized recommendation model called MUSIG that learns “meaningful representations of tracks and users” based on individual features of songs (like genre, acoustics, danceability, the wordiness of lyrics) and how they relate to one another (like if they appear on the same playlist).
Keeping up with the users
However, it’s not enough to just find out what users like at the moment. Our music tastes change over time, and Spotify has to feed users new tunes they like to keep them coming back.
This includes recommending a mix of content that is popular or similar to music that the person has previously listened to along with exploratory content that is more eclectic and unaligned with the user’s usual engagement.
[Related: MIT scientists taught robots how to sabotage each other]
“Maybe you joined Spotify for dance music, but can we help you focus while studying. By balancing these polarities, can we help you get a more fulfilling content diet,” Stål said in the video presentation. “We have to think about your wants, things that will keep you in your comfort zone, and your needs, so things that may enhance your listening down the line but may not be exactly what you expect right now.”
This type of formula “can facilitate exploration by helping users discover new content or inculcate new tastes,” Spotify researchers wrote in a post in March. “It can help the platform spread consumption across artists and facilitate consumption of less popular content.”
Of course, Spotify has its own financial reasons for wanting to diversify its users’ taste palette. Internal studies have shown that active users with more diverse listening habits were “25 percentage points more likely to convert from Free to Premium than those with less diversity in their music consumption.”
To stay ahead of our changing preferences, and keep their recommendations fresh, Spotify also needs to be able to understand how what we like can evolve over time. Earlier this year, researchers there built a model based on a dataset of 100,000 Spotify users who were continuously active from 2016 to 2020. They looked at the entire streaming history of every user, grouped their music into “micro-genres,” and mapped them across time. What they came up with was a connected graph that illustrated the transitions between different genres of music. As an example, their model suggested that to go from liking “EDM” to “nu jazz” or “gospel,” users were likely to pass through a phase of liking “tropical house,” which is chill but upbeat electronic music.
Using the paths laid out in this model, Spotify hopes that they can gradually acclimate users to different genres by journeying them through microgenres that lie in-between what they already like and what they don’t yet know.
Outside of music, Spotify’s decade of personalization research has already extended to podcasts. “It turns out that we can even predict what kinds of podcasts a listener might enjoy based on their taste in music,” Stål noted in a company blog post. But Spotify’s audio expansion plans don’t end there—it recently acquired the audiobook company Findaway.