《Developer-Experience-and-Lesson-Learned-in-Building-Real-world-Spark-Applications-Grace-Huang1》.pdf

Experience and Lesson Learned in Building Real-World Spark Applications Grace Huang ( jie.huang@) Intel/SSG/Big Data Technology This is team work Thanks all the great guys’ contributions in last 6 months! Software and Services Spark is Sparkling in China  Spark is not only rising but also sparkling  Evolving from promotion to demand growth  + co-locates there  Intel partnering with several large web sites in China since 2012  Building real-world big data analytic applications using Spark stack  E.g., Alibaba, Baidu iQiyi, Youku, Sina, etc.  Application focuses  Iterative, complex machine learning graph analysis  Real-time analytical processing (RTAP)  Complex, Interactive OLAP/BI /wp-content/uploads/2013/10/Spark_Summit_2013_Jason_Dai.pdf Software and Services 3 Lessons learned 1. Manage Memory 2. Avoid Network 3. Improve Disk I/O 4. Optimize Computations Software and Services 4 Lessons learned 1. Manage Memory 2. Avoid Network 3. Improve Disk I/O 4. Optimize Computations Software and Services 5 Manage Memory M e  Graph analysis – Ndegree association m o r  Computing associations between two vertices that are n-hop away y (E.g., Friends of friend)  Graph-parallel implementation by Bagel or GraphX  High memory utilization for cached messages in Bagel  Memory space not used efficiently huge GC impact  Memory is not freed timely  Vertex duplication for sparse graph in GraphX Software and Services 6 Solution#1 Partition based RDD un-persistent M e  RD