Apache: Big Data 2016

The Wikimedia Foundation is a charitable organization with a vision of a world where everyone can freely share in the sum of all knowledge. Each month it serves over 18 billion page views to 500 million unique visitors around the world.

Among the resources offered by Wikimedia is an API providing low-latency access to full-history content, in many formats. Its results are often the product of computationally intensive transforms, and must be pre-generated and stored to meet latency expectations. Unsurprisingly, there are many challenges to providing low-latency access to such a large data-set, in a demanding, globally distributed environment.

This talk will cover the Wikimedia content API and its use of Apache Cassandra as storage for a diverse and growing set of use-cases. Trials, tribulations, and triumphs, of both a development and operational nature will be discussed.