PREGEL Indian Institute of Technology Bombay预凝胶印度技术学院孟买.pptxVIP

PREGEL A System for Large-Scale Graph Processing The Problem Large Graphs are often part of computations required in modern systems (Social networks and Web graphs etc.) There are many graph computing problems like shortest path, clustering, page rank, minimum cut, connected components etc. but there exists no scalable general purpose system for implementing them. 2 Pregel Characteristics of the algorithms They often exhibit poor locality of memory access. Very little computation work required per vertex. Changing degree of parallelism over the course of execution. Refer [1, 2] 3 Pregel Possible solutions Crafting a custom distributed framework for every new algorithm. Existing distributed computing platforms like MapReduce. These are sometimes used to mine large graphs[3, 4], but often give sub-optimal performance and have usability issues. Single-computer graph algorithm libraries Limiting the scale of the graph is necessary BGL, LEDA, NetworkX, JDSL, Standford GraphBase or FGL Existing parallel graph systems which do not handle fault tolerance and other issues The Parallel BGL[5] and CGMgraph[6] Pregel 4 Pregel Google, to overcome, these challenges came up with Pregel. Provides scalability Fault-tolerance Flexibility to express arbitrary algorithms The high level organization of Pregel programs is inspired by Valiant’s Bulk Synchronous Parallel model[7]. Pregel 5 Message passing model A pure message passing model has been used, omitting remote reads and ways to emulate shared memory because: Message passing model was found sufficient for all graph algorithms Message passing model performs better than reading remote values because latency can be amortized by delivering larges batches of messages asynchronously. Pregel 6 Message passing model Pregel 7 Example Find the largest value of a vertex in a strongly connected graph 8 Pregel 6 2 6 6 6 6 6 Blue Arrows are messages Blue vertices have voted to halt 9 Pregel 6 Finding the largest value in a graph Basic Orga