Apache: Big Data 2016 has ended
Register Now or Visit the Website for more Information 
Monday, May 9 • 11:40am - 12:30pm
Migrating Hundreds of Hadoop Pipelines into Docker Containers - Noa Resare, Spotify

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Spotify maintains hundreds of big data pipelines built over a number of years, most of which runs one or more transformations on our 1800 node on-premise Hadoop cluster. There has been steady evolution with regards languages, frameworks and development strategies over those years and the result is a highly heterogenous set of pipelines with lots of specific demands the execution environment. To ensure stability while encouraging innovation, we are now leveraging Docker to contain some of the complexity and have a unified interface for the scheduling infrastructure. This talk is all about what we have learned in the process and how Spotify’s experience in running a large fleet of docker containers for production services has helped shape our efforts.

avatar for Noa Resare

Noa Resare

Free Software Ombudsman, Spotify
Noa Resare is a senior engineer and the Spotify Free Software Ombudsman. Noa is an accomplished public speaker has been giving talks at conferences such as Cloud Open, Usenix Lisa and LinuxCon on a wide variety of technical subjects.

Monday May 9, 2016 11:40am - 12:30pm PDT
Plaza C