分布式OLAP引擎Pinot的聚合索引—PINOT Real-time Analytics at LinkedIn.pptx

Real-time Analytics at LinkedIn Greg Brandt (2015-­‐09-­‐11) Pinot Agenda Pinot at LinkedIn Design and Architecture Segment Layout StarTree for Fast Aggregation ThirdEye for Business Monitoring Use Cases at LinkedIn Who’s Viewed My Profile – (member) Use Cases at LinkedIn XLNT – A/B tes-ng – (internal) Use Cases at LinkedIn Ad campaigns – (customers) Data Analytics Ecosystem at Li nkedln (Analytics Products) WVMP (Tracking Data) Real-time Offline High-Level Design Goals SQL-like interface Columnar storage and indexing Real-time data load Non-goals JOIN, UDF (unpredictable performance) Not source of truth Mutability – Immutability allows trivial data distribu-on Pinot Architecture 挝蜕毁照E Networked Storage Stream data from Kafka Kafka Segment Assignment (1) Push time segment(s) from t:!到照12. (2) Controller computes segment assignment z。oKeeoer (3) Servers receive notification for new partitions they own Storage (4) Servers pull data from networked storage Apache Helix A framework for building distributed, stateful applications Developed at LinkedIn for Espresso, (distributed OLTP database) Leveraged in Pinot for cluster management (Controller uses Helix libraries) Query Execution: Distributed SElECT SUM(clicks) FROM table WHERE time B盯WEEN 20 AND 60 Broker time- partitions que时， then sca忧ers que即/ gathers results Historical data preferred to real­ time data Controller notifies of segment placement Quer y Execution: Local 1. Parser – SELECT COUNT(*) … = Abstract Syntax Tree Logical Planner Operators like COUNT, WHERE, etc. Physical Planner Consider data layout (sort key, dic-onary encoding, compression, etc.) Layout: Columnar Storage moon-roof，reliable COLUMNAR DATA RAW DATA tags moon-roof，reliable mp3，compact Manual，compact leather，cool automatlc INDEX GENERATION Oocld Model Year Prlce 饱gs Toyota 1997 7500 m∞n-roof ，reliable 2 Toyota 2001 7切。 m阂，compact 3 Mazda 2005 1切。 Manual，compact 4 Honda 2010 8∞o leather ，cool 5 Honda 2011 9∞o aulomalic 100∞ b