ChukwaA large-scale monitoring system.PDF

Chukwa: A large-scale monitoring system Jerome Boulon Andy Konwinski Runping Qi jboulon@ andyk@ runping@ Yahoo!, inc UC Berkeley Yahoo!, inc Ariel Rabkin Eric Yang Mac Yang asrabkin@ eyang@ macyang@ UC Berkeley Yahoo!, inc Yahoo!, inc Abstract data. Logs are periodically deleted, unless users take the initiative in storing them. We describe the design and initial implementation of We believe that our situation is typical, and that lo- Chukwa, a data collection system for monitoring and an- cal storage of logging data is a common model for very alyzing large distributed systems. Chukwa is built on large deployments. To the extent that more sophisticated top of Hadoop, an open source distributed filesystem and data management techniques are utilized, they are largely MapReduce implementation, and inherits Hadoop’s scal- supported by ad-hoc proprietary solutions. A well docu- ability and robustness. Chukwa also includes a flexible mented open source toolset for handling monitorin