You tap the Spotify app on your phone. Within seconds, a vast digital library of music and audio programs is there at your fingertips. You search for something that matches your mood, maybe a recommended song or artist based on your personal listening history. As you settle in to enjoy your selection, you’re probably not thinking about the invisible infrastructure that makes this whole interaction possible — not only for you, but for the platform’s 626 million users around the world.
So how does one go about seamlessly connecting such a huge global audience with almost a century’s worth of music and recorded content on demand?
For Austin, Texas-based DigiPen graduate Adam Estela, it’s more than a rhetorical question. As a data engineer and technical owner for the Swedish streaming giant, he’s spent the last two years helping tackle a number of the platform’s most puzzling technical challenges related to data processing, analysis, and integration.
“It’s a pretty niche field with a lot of new developments,” Estela says. “Typically in data engineering, we end up delving into graph theory, discrete math, and specialized algorithms to write solutions in distributed environments, and there’s not a lot of easily available resources for these types of problems.”
Whereas much of web-based app development can be roughly divided between front-end versus back-end engineering — with the former focused on user interfaces and the latter dealing with core functionality — data engineering tends to deal with problems of a deeper, more esoteric nature. For example, what can be done to combine and reconcile multiple massive data sets when simple programming techniques won’t suffice? Or how do we take things from the real world and translate them into useful and reliable digital data in the first place?
As one example of a specific problem he confronted recently, Estela points to the Spotify artist credit feature, which displays all relevant writing, recording, and production credits for each piece of music in the Spotify library.
“You can imagine that record labels and publishers may digitally represent artists and their assets very differently, so it could be hard to accurately represent every contributor, whether that’s the writer or an instrumentalist,” Estela says. “Creating the best credits possible is a problem I was excited to focus on, and there’s still more that can be done.”
At the moment, Estela works on product issues relating to audio books, the newest category of content introduced to the Spotify platform in 2022.
“I can’t say much about it, except that I’m very excited,” he says.
Enthusiastic as he is, Estela says his journey into data engineering was mostly accidental. In fact, his entire career turned out to be quite different from what he initially envisioned when he first enrolled in the BS in Computer Science in Real-Time Interactive Simulation program at DigiPen. Despite his original goal of becoming a game programmer, things quickly took a different turn after his graduation in 2013.
“I got a lot of interviews but always came in second or third place,” Estela says. “So I just decided to round that out with searching for non-game opportunities.”
After brushing up on his web programming knowledge, his expanded search eventually landed him a job with a Kirkland-based startup working on tools and infrastructure for an online advertising service.
“After working there, they were impressed enough with me that they started targeting [hires from] DigiPen,” Estela says.
I said, ‘You know what? Why am I wasting time with this? Why don’t I just go for a dream job?’
From there, he spent the next several years rounding out his programming experience even further — first as a Microsoft developer working on tools and automation for the HoloLens mixed reality headset, followed by stints at a string of tech startups spanning multiple industries. He worked on front-end features for a software system for car dealerships, back-end systems for a cloud-based human resources and payroll management platform, and more. Still, he says, each experience left him wanting something more.
“Every time I learned a framework or a role, I would quickly get to a point where it was like, ‘Argh! This is just front end. It’s just showing what’s in the database.’ I’d move to the database side. ‘Argh! This is just storing data. How’s it important to the business? Where does it come from?’” he says. “At every step, I felt more or less replaceable. I wanted to see the bigger picture outside of a single app or service.”
All of that changed when, while working as a back-end engineer for a healthcare startup, a surprise directive came from the top.
“We were told by our director, ‘Our databases are not going to scale this open-enrollment season.’ The underlying message was to fix it or we’re basically out of a company,’” Estela recalls.
With only three months to react, Estela and his engineering coworkers had to quickly learn and migrate to what’s known as a MapReduce framework, with a ‘data lake’ architecture that could more readily handle the huge influx of customer data.
“MapReduce distributes your data processing and makes that run very, very quickly. We’re talking about database migrations that might take weeks will take minutes,” Estela says. “We did it, it was great, and I felt like I was directly solving the company’s biggest problems.”
For Estela, the experience was also a crash course in data engineering that opened his eyes to a whole new field of programming. He began diving head-first into increasingly complex data pipeline problems, sometimes veering into machine learning applications. For example, how can you algorithmically detect duplicate customers from a combination of messy demographic data sets, so as to avoid the potential for medical insurance fraud?
“When I finally ended up in data, that felt like the thing that’s connected to the real world — very tied in with the business operations and what ends up flowing to the front end,” he says. “How do you even get things from the real world? There’s a lot of creativity here!”
While the healthcare company eventually folded due to unrelated business troubles, it marked a new opportunity for Estela to market his newly acquired data engineering chops to employers. As he started sending out his resume far and wide, however, he quickly became overwhelmed. A different strategy was in order.
“I said, ‘You know what? Why am I wasting time with this? Why don’t I just go for a dream job?’” Estela says. “I had read some Spotify engineering and product blogs way back and had always been interested in their work-life balance, focus on people, everyone being product oriented — engineers included. And they had a lot of really nice data problems.”
He prepared his application materials, submitted them to the company, and within 24 hours of applying, received an invitation to interview.
“It just worked out perfectly,” Estela says.
So what is it like to work on one of the largest and most popular streaming platforms on the planet? For starters, he says, it’s humbling. Any line of code written can have tremendous consequences — a fact that was brought into stark relief during his first days at the company.
“I remember the very first ticket I was working on,” Estela says. “I wrote a pretty unassuming line of code and watched a data job increase by an hour.”
Partly for that reason, Estela says the other main takeaway is Spotify’s heavy emphasis on things like careful planning and deliberation over speed and quantity of code written.
“I’d been mostly conditioned in these startup environments, so when I first got to Spotify, I was like, ‘That’s the bug! The bug is bad! Go!’ I was like SEAL Team 6 on any bug, problem, or request,” Estela says with a laugh. “And I mean, people were not mad at that, of course. But it’s not the sort of thing that I think will make you successful, or necessarily make the company successful.”
Programming is so cool in how flexible it is and how much you can really just steer yourself, where you’re almost your only limiting factor.
Today, Estela says, his typical work routine has more to do with long-term product cycles, where he has both the autonomy and responsibility to investigate platform issues from a holistic point of view. Long before sitting down to write the code — the part of the job that now takes the least amount of time — Estela has a close hand in everything from gathering input from other Spotify teams to designing the actual specs for a new data product feature.
“I think what Spotify does really well is ask ‘Why?’ a lot,” he says. “All the thinking we do ahead of time really is trying to answer a lot of those questions of, ‘Is this worth pursuing at all?’ Because things we pursue are very expensive.”
Reflecting on his strange journey so far, Estela says he’s happy for the chance to have explored so many different sides of the technology sector — the benefit of having a strong foundation in computer science.
“Programming is so cool in how flexible it is and how much you can really just steer yourself, where you’re almost your only limiting factor. I’ve gone from used car sales to healthcare to music. No one’s going to stop you. I’ve gone across so many roles, and I think that would have surprised me because I was so dead set on games,” he says. “DigiPen was the beginning, not the end.”
Of course, Estela’s role also begs the question: Just what does a Spotify employee listen to while on the job?
“Bad Bunny. Bad Bunny, for sure,” he says, smiling and pointing to a framed vinyl album on the wall behind him. “I saw him here in Austin. There was an orchestra, a horse, and a suspended bridge. It was really cool!”