impala-meetup-20130820英文电子资料.pdf

Parquet Update/UDFs in Impala Nong Li Soware Engineer, Cloudera 1 Agenda •  Parquet •  File format descripon •  Benchmark Results in Impala •  Parquet 2.0 •  UDF/UDAs 2 Parquet 3 4 5 6 7 Data Pages •  Values are stored in data pages as a triple: Definion Level, Repeon Level and Value. •  These are stored conguous on disk = 1 seek to read a column regardless of nesng. •  Data pages are stored with different encodings: •  Bit packing and Run Length Encoding (RLE) •  Diconary for strings •  Extended to all types in Parquet 1.1 •  Plain (lile endian encoding) for nave types. 8 Parquet 2.0 •  Addional Encodings •  Group VarInt (for small ints) •  Improved string storage format •  Delta Encoding (for strings and ints) •  Addional Metadata •  Sorted files •  Page/Column/File Stascs •  Expected to further reduce on disk size and allow for skipping values on the read path. 9 Hardware Setup •  10 Nodes •  16 Core Xeon •  48 GB Ram •  12 Disks •  CDH4.3 •  Impala 1.1 10 TPC-H lineitem table @ 1TB scale factor Size (GB) 800 700 600 500 400 300 200 100 0 Text Text w/ Lzo Seq w/ Snappy