UniversityofTorontoAbstract.PDF

Simple Testing Can Prevent Most Critical Failures An Analysis of Production Failures in Distributed Data-intensive Systems Ding Yuan, Yu Luo, Xin Zhuang, Guilherme Renna Rodrigues, Xu Zhao, Yongle Zhang, Pranay U. Jain, Michael Stumm University of Toronto Abstract raises the questions of why these systems still experi- ence failures and what can be done to increase their re- Large, production quality distributed systems still fail pe- riodically, and do so sometimes catastrophically, where siliency. To help answer these questions, we studied 198 most or all users experience an outage or data loss. We randomly sampled, user-reported failures of five data- present the result of a comprehensive study investigat- intensive distributed systems that were designed to tol- ing 198 randomly selected, user-reported failures that oc- erate component failures and are widely used in produc- curred on Cassandra, HBase, Hadoop Distributed File tion environments. The specific systems we considered were Cassandra, HBase, Hadoop Distributed File System System (HDFS), Hadoop MapReduce, and Redis, with the goal of understanding how one or multiple faults (HDFS), Hadoop MapReduce, and Redis. eventually evolve into a user-visible failure. We found Our goal is to better understand the specific failure that from a testing point of view, almost all failures re- manifestation sequences that occurred in these systems quire only 3 or fewer nodes to reproduce, which is good