让数据分析框架有意义-Making Sense of Performance in Data Analytics Frameworks.pdfVIP

Making Sense of Performance in Data Analytics Frameworks † † ‡ Kay Ousterhout , Ryan Rasti , Sylvia Ratnasamy , Scott Shenker , Byung-Gon Chun UC Berkeley, †ICSI, VMware, ‡Seoul National University Abstract This paper makes two contributions towards a more There has been much research devoted to improving the comprehensive understanding of performance. First, we performance of data analytics frameworks, but compara- develop a methodology for analyzing end-to-end per- tively little effort has been spent systematically identify- formance of data analytics frameworks; and second, we ing the performance bottlenecks of these systems. In this use our methodology to study performance of two SQL paper, we develop blocked time analysis, a methodology benchmarks and one production workload. Our results run counter to all three of the aforementioned mantras. for quantifying performance bottlenecks in distributed computation frameworks, and use it to analyze the Spark The first contribution of this paper is blocked time framework’s performance on two SQL benchmarks and analysis, a methodology for quantifying performance a production workload. Contrary to our expectations, we bottlenecks. Identifying bottlenecks is challenging for find that (i) CPU (and not I/O) is often the bottleneck, (ii) data analytics frameworks because of pervasive paral- improving network performance can improve job comple- lelism: jobs are composed of many parallel tasks, and tion time by a median of at most 2%, and (iii) the caus