Personalization’s Hulk squad produces Human Understandable Language Knowledge to enrich content understanding. We utilize Large Language Models to understand podcasts and audiobooks, building reliable, scalable systems to distribute that knowledge to Spotify internal teams, users, and creators.

We are looking for a Data Engineer with interest in ML techniques to join our team and help build the future of podcast and audiobook listening experiences for millions of listeners at Spotify. This is a unique opportunity to help develop and shape Spotify recommendations.

You’ll grow your skills in engineering at scale, work with a cross-functional team of Machine Learning Engineers, Backend Engineers, Data Engineers, and researchers, and join a super motivated and supportive team.

What You'll Do

• Work with large-scale data pipelines using data processing frameworks like Scio (built on Apache Beam), Hendrix ML, Ray, PyTorch, and Dataflow and other Google Cloud Platform offerings
• Support streaming and batch inference of core LLM models
• Develop new data pipelines to meet emerging needs and maintain our existing data systems as necessary to ensure they are reliable, scalable, and well tested
• Work with ML engineers to develop training and evaluation datasets for Generative LLM models
• Own the development of data solutions from architecture to delivery
• Collaborate with other backend and data engineers, ML engineers, data scientists, researchers, and other stakeholders
• Develop monitoring tools to automatically track AI product quality. This likely involves the use of LLMs to evaluate results as well as closer monitoring of performance indicators, and creation and maintenance of metrics datasets and dashboards
• Work in an agile team to continuously experiment, iterate and deliver on new product objectives
• Share knowledge, promote standard methodologies, making your team the best version of itself through mentorship and constructive accountability

Who You Are

• You are passionate about data and proficient in data modeling, data access, and data storage optimization
• You have experience building high-scale production data pipelines with one or more data processing frameworks such as Scio (built on Apache Beam), and major props if you know Ray
• You might have worked with orchestration frameworks such as Flyte
• You care about agile software processes, data-driven development, reliability, and responsible experimentation
• You value collaboration and partnership within teams
• Experience, exposure, or curiosity about ML Data processing and inference pipelines, LLM fine-tuning, and/or using LLMs for data analysis
• Bonus if you have experience with NRT (near real time) systems
• Bonus if you experience with prompt engineering for LLMs and prompt optimization tools like DSPy

Where You'll Be

• We offer you the flexibility to work where you work best! For this role, you can be within the North America region as long as we have a work location.
• This team operates within the Eastern Standard time zone for collaboration.