Druid之旅-大数据实时分析数据存储框架.docx

ERICTSCHETTER. CREATOR OF DRUID ■ —■ . ■ k 一 Data Ingestion Rate Ingest data and make it queryable in real-time Arbitrary Drill-Downs, Slice-n-Dice Arbitrary boolean filters Availability Downtime is evil RDBMS - Relational Database I. Star Schema Aggregate Tables Query Caching I. I. Star Schema Aggregate Tables Query Caching ? Queries that were cached ? fast Queries against aggregate tables fast to acceptable Queries against base fact table generally unacceptable Select COUNT。）scan rate ?5.5M rows / second / core 1 day of summarized aggregates 60M+ rows 1 query over 1 week, 16 cores ?5 seconds Page load with 20 queries over a week of data ong time II. NoSQL - Key/Value Store ? Pre-aggregate all dimensional combinations (truncate time) ? Store results in a NoSQL store 1 revenue=$LI9 l,M revenue=$0.15 1 M 18 $0」5 l,F revenue=$ 1.04 1 F 25 $1.03 IJ8 revenue=$0.16 2 F 18 $0.01 1,25 revenue=$L03 I,M,I8 revenue=$0J5 l,EI8 revenue=$0.0l l,E25 revenue=$L03 Queries were fast range scan on primary key Inflexible not aggregated, not available Not continuously updated aggregate first, then display Processing scales exponentially Dimensional combinations = exponential increase Tried limiting dimensional depth still expands exponentially Example:?500k records 11 dimensions, 5-deep 4.5 hours on a 15-node Hadoop cluster 14 dimensions, 5-deep 9 hours on a 25-node Hadoop cluster Mh?:.r -- ? II. NoSQL II. NoSQL III.??? Problem with RDBMS: scans are slow Problem with NoSQL: computationally intractable Problem with RDBMS: scans are slow Problem with NoSQL: computationally intractable Problem with RDBMS: scans are slow Problem with NoSQL: computationally intractable Tackling RDBMS issue seems easier X L 1.Real-Time Ingestion (Indigestion?) 2.Slicing-n-Dicing Drill Down Fruit IMinj切 3.Available R u c CL T 川 c R A ■ ■ Realtime Nodes Query API Realtime Nodes Query API Hand Off Data Historical Nodes Query API Broker Nodes timestamp publisher advertiser gender country . . ? click price 2011