Impala A Modern SQL Engine for Apache Hadoop.pdfVIP

Cloudera Impala: A Modern SQL Engine for Apache Hadoop Mark Grover Software Engineer, Cloudera January 16, 2013 What is Impala? ● General-purpose SQL engine ● Real-time queries in Apache Hadoop ● Beta version released since October 2012 ● General availability (GA) release slated for April 2013 Overview ● User View of Impala ● Architecture of Impala ● Comparing Impala with Dremel ● Comparing Impala with Hive ● Impala Roadmap Impala Overview: Goals ● General-purpose SQL query engine: ● should work both for analytical and transactional workloads ● will support queries that take from milliseconds to hours ● Runs directly within Hadoop: ● reads widely used Hadoop file formats ● talks to widely used Hadoop storage managers ● runs on same nodes that run Hadoop processes ● High performance: ● C++ instead of Java ● runtime code generation ● completely new execution engine that doesnt build on MapReduce User View of Impala: Overview ● Runs as a distributed service in cluster: one Impala daemon on each node with data ● User submits query via ODBC/Beeswax Thrift API to any of the daemons ● Query is distributed to all nodes with relevant data ● If any node fails, the query fails ● Impala uses Hives metadata interface, connects to Hives metastore ● Supported file formats: ● text files (GA: with compression, including lzo) ● sequence files with snappy/gzip compression ● GA: Avro data files ● GA: Trevni (columnar format; more on that later) User View of Impala: SQL ● SQL support: ● patterned after Hives version of SQL ● limited to Select, Project, Join, Union, Subqueries, Aggregation and Insert ● only equi-joins; no non-equi joins, no cross products ● Order By only with Limit ● GA: DDL support (CREATE, ALTER) ● Functional limitations: ● no custom UDFs, file formats, SerDes ● no beyond SQL (buckets, samples, transforms, arrays, structs, maps, xpath, json) ● only hash j