大数据技术基础培训-GraphLab技术培训.pptx

大数据技术基础培训GraphLab 技术培训Social MediaWebAdvertisingScience图用来编码/描述知识和联系：:Big: billions of vertices and edges and rich metadata人产品想法事实兴趣图是机器学习和数据挖掘的重要基础之一发现有影响力的人或者信息发现社区或者社交圈或者团伙精准广告/产品定位 复杂数据/知识依赖性建模图并行算法的特性DependencyGraphFactored Computation IterativeComputationWhat I LikeWhat My Friends Like为什么MapReduce不合适Map-Reduce不能高效表示依赖图User must code substantial data transformations Costly data replicationIndependent Data Rows为什么MapReduce不合适(2)IterationsDataDataDataDataCPU 1CPU 1CPU 1DataDataDataDataDataDataDataDataCPU 2CPU 2CPU 2DataDataDataDataDataDataDataDataCPU 3CPU 3CPU 3DataDataDataSlowProcessorDataDataDataDataDataBarrierBarrierBarrierMap-Reduce 无法高效描述迭代算法:为什么MapReduce不合适(3)图算法通常只有一部分数据（子图）需要计算:IterationsDataDataDataDataCPU 1CPU 1CPU 1DataDataDataDataDataDataDataDataCPU 2CPU 2CPU 2DataDataDataDataDataDataDataDataCPU 3CPU 3CPU 3DataDataDataDataDataDataDataDataBarrierBarrierBarrier为什么MapReduce不合适(4)Hadoop Map/Reduce没有为迭代算法优化IterationsDataDataDataDataCPU 1CPU 1CPU 1DataDataDataDataDataDataDataDataCPU 2CPU 2CPU 2DataDataDataDisk PenaltyStartupPenaltyStartup PenaltyDisk PenaltyStartup PenaltyDisk PenaltyDataDataDataDataDataCPU 3CPU 3CPU 3DataDataDataDataDataDataDataData图并行算法抽象A user-defined Vertex-Program runs on each vertexGraph constrains interaction along edgesUsing messages (e.g. Pregel [PODC’09, SIGMOD’10])Through shared state (e.g., GraphLab [UAI’10, VLDB’12])Parallelism: run multiple vertex programs simultaneouslyDepends on the popularity their followers示例Depends on popularityof her followersWhat’s the popularityof this user?Popular?PageRank AlgorithmRank of user iWeighted sum of neighbors’ ranksUpdate ranks in parallel Iterate until convergencePregel (Giraph)ComputeCommunicateBarrierBulk Synchronous Parallel Model:The Pregel AbstractionVertex-Programs interact by sending messages.Pregel_PageRank(i, messages) : // Receive all the messages total = 0 foreach( msg in messages) : total = total + msg // Update the rank of this vertex R[i] = 0.15 + total // Send new messa