Apache: Big Data 2016 has ended
Register Now or Visit the Website for more Information 
Monday, May 9 • 2:00pm - 2:50pm
Will It Scale? The Secrets Behind Scaling Stream-processing Applications - Navina Ramesh, LinkedIn

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Scaling stream processing applications is sometimes seen akin to scaling batch processing applications. You may re-partition your input stream to scale throughput, similar to re-sharding a batch. However, it becomes challenging for "stateful" applications to “stay realtime”, as they frequently require fault-tolerant state-management. Providing low-latency, fault-tolerant processing for high-volume input streams is fundamentally governed by the state-management primitives provided by the stream processing systems. In this talk, we will discuss how such stateful applications are supported in the open-source stream-processing systems, such as Apache Flink, Spark Streaming and Apache Samza. We will, then provide a deep-dive on Apache Samza’s approach for state-management and fault-tolerance and discuss how it can be effectively used to scale stateful applications.

avatar for Navina Ramesh

Navina Ramesh

Navina Ramesh started her career in Yahoo! India, where she contributed on scaling the Yahoo! Search clusters for 3 years. At LinkedIn, she has worked on developing the Feed Personalization pipeline and improved the caching and pagination models in the Feed Infrastructure. She has... Read More →

Monday May 9, 2016 2:00pm - 2:50pm PDT
Plaza A