SparkSQL的使用经验分享.pptx

基于Spark/hbase的数据分析平台及 SparkSQl使用经验分享 BUILD FOR ANALYTICS We built a Distributed Graph Database for analytical use cases Fast traversal + Range query + Batch processing Need to take care of the interfaces to users (Analysts) Query Language? (Analysts says SQL please!) Interactive data exploration? Visualization? Dynamic charts? And be flexible to do all sorts of powerful things 3 BUILD FOR SCALABILITY Need to be fully scalable and optimized for Data size (100TB planned, 1PB next year) Ingestion speed (parallel insertion + dynamic splits) Parallel and Batch queries Offline + Online processing Mining 4 ARCHITECTURE DEMO 5 6 7 SPARKSQL 经验分享：Before You Start Build a latest Spark from branch-1.2! Lots of new features, bug fixes Parameters are set to a better default Join optimizations! 8 SPARKSQL 经验分享：数据格式 Use Parquet (Columnar format, dictionary encoding, compression) Data size dropped 40%! when converting from old SequenceFile (supposed to be highly compressed) Columnar format avoids unnecessary IO and deserialization vs (30s vs 25min) 9 SPARKSQL 经验分享：数据格式 Conversion is easy if you already have a Pig/Hive loader for your old data 10 SPARKSQL 经验分享：Load Parquet in HiveContext /huangjs/683a4c85ae14e9ae205b# Note: Column names has to match Parquet schema. Tip: Use CTAS (View) for Column renaming. Try setting spark.sql.hive.convertMetastoreParquet to false to use Hive’s Parquet implementation if not working 11 SPARKSQL 经验分享：Broad-cast Join Optimization Biggest headache : Join optimization support is still far from mature Cannot specify which join algorithm to use In Hive, we can add /*+ MAPJOIN(…) */ Not supported in SparkSQL HiveContext It relies on table metadata retrieved by Use “DESC EXTENDED table” to check metadata 12 SPARKSQL 经验分享：Broad-cast Join Optimization If…{EXTERNAL=TRUE, transient_lastDdlTime=1417763591, totalSize=56166}…works! If…{numFiles=0, EXTERNAL=TRUE, transient_lastDdlTime=1417763892, COLUMN_STATS_ACCURATE=false, totalSize=0, numRows=-1, rawDat