（讲义）数据库技术：Apache Kylin 1.5 Updates分享.pptx

The Evolution of Apache Kylin;Agenda;Extreme OLAP Engine for Big Data Kylin is an open source Distributed Analytics Engine from eBay that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets Open Sourced on Oct 1st, 2014 Accepted as Apache Incubator Project on Nov 25th, 2014;Feature – SQL Interface;eBay;90% queries <5s;;Linear scale out with more nodes;A query may consider only 3 dimensions;Base vs. aggregate cells; ancestor vs. descendant cells; parent vs. child cells (9/15, milk, Urbana, Dairy_land) - <time, item, location, supplier> (9/15, milk, Urbana, *) - <time, item, location> (*, milk, Urbana, *) - <item, location> (*, milk, Chicago, *) - <item, location> 5. (*, milk, *, *) - <item>;Agenda;Cube Builder (MapReduce…);;;Engine MR V1 MR V2 Spark (early) Streaming (experimental) Source Hive Kafka Spark SQL & DataFrames Storage HBase ? Kudu ? Cassandra;Freedom Zoo break, not bound to Hadoop any more Free to go to a better engine or storage Extensibility Accept any input, e.g. Kafka Embrace next-gen distributed platform, e.g. Spark Flexibility Choose different engine for different data set;;mapper;If data splits are unique Fast cubing wins If data splits are common Layer cubing wins;A simplified star cubing algorithm Xin, Dong, et al. "Star-cubing: Computing iceberg cubes by top-down and bottom-up integration." Proceedings of the 29th international conference on Very large data bases-Volume 29. VLDB Endowment, 2003. Top-down; Free resource on branch complete Multi-threading if mem available; Ordered output;Slow queries are 5-10x faster.;Near Realtime Incremental Build;Stream Data Consuming;;;Use Case: SEO Operational Dashboard;HyperLogLog Count Distinct TopN BitMap Precise Count Distinct from Sun, Yerui () Raw Records from Wang, Xiaoyu () Domain specific aggregations now become easy aggregate user events to detect time serials or access patterns draw a sketch of certain user groups pre-calculate clusters of