构建基于Apache Kylin的大数据分析平台 讲话.ppt

Use Cases: eBay DSS Nous NRT SEO Dashboard Near Real-time and Historical together eCG GA Deep Dive Pilot Case Extreme challenge for storage and building time Enabled Tableau eCS Tracking Report Hugest Cube: 100+B raw records in one cube Use Case: JD.com JCloud JOS - API Statistics JCloud 云海 – Low latency query engine JCloud Data Cloud – OLAP platform for online analytics tool Use Case: Baidu Map Use Case: VIP.com Use Case: Beijing Mobile Performance and Throughput By NetEase: /2016/01/04/kylin-olap/ Roadmap Roadmap QA 联系信息 shaofeng@kyligence.io 加入社区 user@ 网站 http://kyligence.io 微信公众号 ApacheKylin Thanks for your attention! See you on our next meeting * Deploy Kylin on single instance Prepare a Hadoop CLI which has hive, mapreduce, hbase client installed; Download Kylin binary package from /download; Start Kylin with the shell account which has permission to Hadoop; $KYLIN_HOME/bin/kylin.sh start Visit Kylin web GUI at http://host_name:7070/kylin/ * Expand Kylin into cluster Add more Kylin instances, share the same metadata store; Assign one instance for job engine role, all others as query role; kylin.server.mode=all|job|query Configure the server list kylin.rest.servers=host1:7070,host2:7070,host3:7070 Deploy a Load Balancer for all, dispatch the requests to the cluster; (Optional) Configure LDAP or SSO authentication for centralized user management. (Optional) Enable HTTPS in LB Access Kylin via LB: https://lb_address/kylin * Separate READ and WRITE Hbase is a separated cluster with MR, has dedicated HDFS; Cube calculation only happens in MR cluster, will not impact Hbase; Cube files transfer to Hbase cluster on finish; Deploy Deploy Kylin in a MR CLI machine, with Hbase client points to the Hbase cluster; Configure Hbase HDFS kylin.hbase.cluster.fs=hdfs://hbase-cluster:8020 Kylin should be close to Hbase to gain best network performance. * Staging + Production Within the same Hadoop/HBase cluster Staging acts as pre-production, for create/update/test/