chapter2CloudProgrammingandSoftwareEnvironments研讨.pptxVIP

Cloud Programming and Software Environments Distributed File System Single compute node to multi-compute nodes It needs new File system and computing model Distributed File System(DFS) Big file( TB…) File is rarely updated File is divided into file blocks(chunk), usually 64MB/chunk Cloud Ecosystem Requirements: At the system level, the cloud ecosystem  include the cloud platform and infrastructure,  resource management policies, etc. At the service level, the SLAs, globalized  standards, reputation system, billing and  accounting system, cloud business models, etc. At the user (client) level, Application  programming interfaces (APIs), cloud  programming environment, Quality of Service  control, etc. Cloud Ecosystem Cloud Computing Software Tool Packages from Google  The Google file system (GFS) The MapReduce package The Bigtable package MapReduceSimplified Data Processing on Large Clusters What is MapReduce It is a programming model and an associated implementation for processing and generating large data sets. A web programming model for fast processing large datasets Applied in web-scale search and cloud computing applications Users specify a map function to generate intermediate  key/value pairs Users use a reduce function to merge all intermediate values  with the same key. MapReduce The computing process as follow Map tasks. Runs on thousands of computers Every task has one or more file chunks as input. generate key-value series(users define the Map function) Master controller Collect key-value orders from Maps. And redistribute them to Reduce functions. The orders with the same key will belong to the same Reduce function Reduce Combine the values with the same key. Got the final result. Map Input can be any type, just like a tuple or a document. The key is not unique Shuffle Implementation Word Frequencies in a document Input ：one document map function，input: key = document