《Dremel__Interactive_Analysis_of_Web-Scale_Datasets》.pdf

Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, Theo Vassilakis Google, Inc. melnik,andrey,jlong,gromer,shiva,mtolton,theov@ ABSTRACT exchanged by distributed systems, structured documents, etc. lend Dremel is a scalable, interactive ad-hoc query system for analy- themselves naturally to a nested representation. Normalizing and sis of read-only nested data. By combining multi-level execution recombining such data at web scale is usually prohibitive. A nested trees and columnar data layout, it is capable of running aggrega- data model underlies most of structured data processing at Google tion queries over trillion-row tables in seconds. The system scales [21] and reportedly at other major web companies. to thousands of CPUs and petabytes of data, and has thousands This paper describes a system called Dremel1 that supports inter- of users at Google. In this paper, we describe the architecture active analysis of very large datasets over shared clusters of com- and implementation of Dremel, and explain how it complements modity machines. Unlike traditional databases, it is capable of op- MapReduce-based computing. We present a novel columnar stor- erating on in situ nested data. In situ refers to the ability to access age representation for nested records and discuss experiments on data ‘in place’, e.g., in a distributed file system (like GFS [14]) or few-thousand node instances of the system.