《093 Druid之旅-大数据实时分析数据存储框架》.pdf

THE JOURNEY OF DRUID, A BIG DATA ANALYTICS DATA STORE ERIC TSCHETTER, CREATOR OF DRUID DEMO REQUIREMENTS “ ” REQUIREMENTS • Data Ingestion Rate • Ingest data and make it queryable in real-time •Arbitrary Drill-Downs, Slice-n-Dice • Arbitrary boolean filters •Availability • Downtime is evil WHAT WE TRIED “ ” WHAT WE TRIED I. RDBMS - Relational Database I. RDBMS - THE SETUP • Star Schema •Aggregate Tables • Query Caching I. RDBMS - THE RESULTS • Queries that were cached • fast • Queries against aggregate tables • fast to acceptable • Queries against base fact table • generally unacceptable I. RDBMS - PERFORMANCE Select COUNT(*) scan rate ~5.5M rows / second / core 1 day of summarized aggregates 60M+ rows 1 query over 1 week, 16 cores ~5 seconds Page load with 20 queries over a week long time of data WHAT WE TRIED I. RDBMS - Relational Database WHAT WE TRIED I. RDBMS - Relational Database II. NoSQL - Key/Value Store II. NOSQL - THE SETUP • Pre-aggregate all dimensional combinations (truncate time) • Store results in a NoSQL store Key Value 1 revenue=$1.19 ts! gender age revenue 1,M revenue=$0.15 1 M 18 $0.15 1,F revenue=$1.04 1 F 25 $1.03 1,18 revenue=$0.16 2 F 18 $0.01 1,25 revenue=$1.03 1,M,18 revenue=$0.15