Apache: Big Data 2016

The last decade as seen a dramatic shift in the complexity of data pipelines. Data is stored in more systems, queried in more ways, and comes from more sources. Complex data pipelines combined with the need for applications that can analyze and respond to that data in real-time leave traditional approach to data integration struggling to keep up.

This talk will describe how data integration is shifting to a streaming model and how Kafka supports this new model. Specifically, it will focus on a new tool included with Kafka, Kafka Connect, that handles streaming "E" and "L". It will describe Kafka Connect’s data and execution models, which provide scalable fault-tolerant import and export between Kafka and other data systems. Finally, it will show how this can be combined with other tools such as stream processing frameworks to create a complete streaming data integration solution.