dryad-talk-berkeley09.pptxVIP

Cluster Computing with DryadLINQ Mihai Budiu Microsoft Research, Silicon Valley Cloud computing: Infrastructure, Services, and Applications UC Berkeley, March 4 2009 Goal 2 Design Space 3 Throughput Latency Internet Private data center Data- parallel Shared memory Dryad Search HPC Grid Transaction Execution Application Data-Parallel Computation 4 Storage Language Parallel Databases Map-Reduce GFSBigTable Cosmos AzureSQL Server Dryad DryadLINQScope Sawzall Hadoop HDFS S3 Pig, Hive SQL ≈SQL LINQ, SQL Sawzall SQL Software Stack 5 Windows Server Cluster Services Distributed FS (Cosmos) Dryad Distributed Shell PSQL DryadLINQ SQLserver Windows Server Windows Server Windows Server C++ NTFS legacycode SSIS Scope C# MachineLearning .Net queueing Distributed Data Structures Graphs Datamining Applications Azure XCompute Windows HPC Azure XStore SQL Server Log parsing Introduction Dryad DryadLINQ Conclusions 6 Outline Dryad Continuously deployed since 2006 Running on 104 machines Sifting through 10Pb data daily Runs on clusters 3000 machines Handles jobs with 105 processes each Platform for rich software ecosystem Used by 100 developers Written at Microsoft Research, Silicon Valley 7 Dryad = Execution Layer 8 Job (application) Dryad Cluster Pipeline Shell Machine ≈ 2-D Piping Unix Pipes: 1-D grep | sed | sort | awk | perl Dryad: 2-D grep1000 | sed500 | sort1000 | awk500 | perl50 9 Virtualized 2-D Pipelines 10 Virtualized 2-D Pipelines 11 Virtualized 2-D Pipelines 12 Virtualized 2-D Pipelines 13 Virtualized 2-D Pipelines 14 2D DAG multi-machine virtualized Dryad Job Structure 15 grep sed sort awk perl grep grep sed sort sort awk Inputfiles Vertices (processes) Outputfiles Channels Stage Channels 16 X M Items Finite streams of items distributed filesystem files (persistent) SMB/NTFS files  (temporary) TCP pipes (inter-machine) memory FIFOs  (intra-machine) Dryad System Architecture 17 Files, TCP, FIFO, Network job s