Apache: Big Data 2016

Currently, there are two competing architectures for how to implement HDFS. The original HDFS approach utilizes storage colocated with the compute servers. An emerging alternative relies on dedicated storage resources shared by the compute cluster. This talk will compare and contrast these two approaches and provide definitive quantitive guidelines to planners and architects to help them identify the best solutions for their needs.

Traditionally, Hadoop clusters have been built using dedicated hardware separated from the rest of the data center IT infrastructure. The rapid growth of HDFS/Hadoop/Spark/Yarn applications makes it desirable to share the services-oriented virtualized infrastructure commonly known as private cloud. While virtualizing compute and network interconnectivity is a relatively well-solved problem. Virtualizing the HDFS storage component into the private cloud work has unique challenges. This talk explores those challenges and offers multiple prescriptive solutions along with criteria to allow planners and architects to meaningfully compare and contrast the different approaches.

YCSB is an open source framework for evaluating data storage systems that has become the de facto standard for use with NoSQL projects. After several quiet years the project has returned to life as a community effort, which has led to substantial utility and system coverage improvements. Sean Busbey will review the last six months of development, explain how users of Apache projects can get a better understanding of their preferred storage system, and discuss ways some folks proactively use YCSB to improve the quality of their storage project.