MizanGraphProcessingSystem.pptVIP

用两张slides介绍清楚Pregel模型 Problems：1，是由于Graph分布不均匀，2是由于计算模型不知 静态预先切图无法胜任，所以需要动态的Load Balance策略 影响图的load分布主要有两大因素：1，graph structure, 2, algorithm behavior, * 影响图的load分布主要有两大因素：1，graph structure, 2, algorithm behavior, * * * * * * * * * * * * * * WS:Pregel 有非常粗粒度的Load balance策略，思想是切多分partition，每个机器根据load放置多份。 * Mizan 和Giraph语言不同，所以无可比性 * 主要是从algorithm behavior来看的，pagerank是Stationary的算法，所以Mizan对其优化很少。 Range主要是因为：切分load不均匀 * Cost不大。 * Cost不大。 * Cost不大。 * * Mizan:Graph Processing System xuejilong@ topics Brief introduction on Pregel EuroSys13 paper: Mizan Our ideas Pregel A System for Large-Scale Graph Processing Mapreduce v.s. Pregel Hadoop v.s. Hama,Giraph... Inside Pregel Graph partitioning Graph data Worker1 Worker2 Workern … Pregel output Communication cost Load balance Inside Pregel Problems of Pregel Worker1 Worker2 Workern Worker1 Worker2 Workern …… Computation Communication Mizan A system for Dynamic Load Balancing in Large-scale Graph Processing Load balance Current method for load balance: Hash partition, Giraph, Pregel Range partition, Min-cut partition(Metis) Sophisticated partitioning tech. Mizan goal building a system that is: Adaptive Agnostic to the graph structure requires no a priori knowledge of algorithm behavior Graph algorithms Stationary Graph Algorithms: matrix-vector multiplication PageRank finding weakly connected components Non-stationary Graph Algorithms: DMST: distributed minimal spanning tree graph queries advertisement propagation Mizan Monitoring Migration planning Monitoring Outgoing message Incoming message Response time Migration Planning Five-step migration planning: Identify the source of imbalance Select the migration objective Pair over-utilized workers with under-utilized ones Select vertices to migrate Migrate vertices Step 1 Step 2 Select the migration objective Outgoing msg, incoming msg, response time Compute correlation between: Outgoing msg and response time Incoming msg and response time Default response time Step 3 St