massive streaming data analytics a case strongstudystrong with clustering.pptVIP

Massive Streaming Data Analytics:A Case Study with Clustering Coefficients David Ediger Karl Jiang Jason Riedy David A. Bader Georgia Institute of Technology Atlanta, GA USA * STINGER Data Structure Spatio-temporal Interaction Networks and Graphs (STING) Extensible Representation General-purpose data structure for dynamic graphs Efficient edge insertion/deletion (updates) with concurrent readers (analysis) * STINGER Data Structure Array of linked lists, which may have empty slots (from deleting edges) Additional stored info not in paper Efficient updates Concurrent reads (no locking) * Assumptions for parallelism Single streaming source for inserts/deletes Changes are scattered widely Batches are sufficiently independent Analysis kernels have small range Graph change only requires access to local portions and affects small portion of output * Assumptions (continued) * Case Study:Updating Clustering Coefficients Clustering coefficients measure density of closed triangles: One way of determining if a graph is a small-world graph * Bloom filter Consider an edge list represented as a bit array (1 bit per edge) = O(n) storage space Bloom filter is a bit array with an arbitrary, smaller number of bits A hash function maps a vertex to a specific bit Small number of bits == high collision rate To reduce false-positives, use k independent hash functions to set multiple bits * Bloom filter * Testbed Massively multi-threaded Cray XMT 64 Threadstorm processors Each running at 500MHz Each has 128 hardware streams maintaining a thread context Context switches occur every cycle 512 GiB globally addressable shared memory (holds 2 billion vertices and 17 billion edges) Synthetic data 16 million vertices, ~500 million edges * * * * *