Vault 2016

Introduction Very few companies run 24/7 clusters at the scale like Facebook using Hadoop/HDFS. As scale increases HDFS shows its limitations in scalability, performance and availability. HDFS was first designed 10+ years ago based on the assumptions that were valid in 2005. Over the years, numerous significant improvements have been made to the system. However storage, network and compute technology changed at different ratios during last decade. In this paper we describe Warm Storage - grounds up solution that exploits technology inflection points that exist in 2015 and we believe will continue into the future. We describe architecture and some of the implementation details of the novel and more efficient and available storage system optimized for DW workloads.