GOOGLE集群计算之Hadoop技术概要.ppt

Google Cluster Computing Faculty Training Workshop Module V: Hadoop Technical Review Overview Hadoop Technical Walkthrough HDFS Databases Using Hadoop in an Academic Environment Performance tips and other tools You Say, “tomato…” Some MapReduce Terminology Job – A “full program” - an execution of a Mapper and Reducer across a data set Task – An execution of a Mapper or a Reducer on a slice of data a.k.a. Task-In-Progress (TIP) Task Attempt – A particular instance of an attempt to execute a task on a machine Terminology Example Running “Word Count” across 20 files is one job 20 files to be mapped imply 20 map tasks + some number of reduce tasks At least 20 map task attempts will be performed… more if a machine crashes, etc. Task Attempts A particular task will be attempted at least once, possibly more times if it crashes If the same input causes crashes over and over, that input will eventually be abandoned Multiple attempts at one task may occur in parallel with speculative execution turned on Task ID from TaskInProgress is not a unique identifier; don’t use it that way MapReduce: High Level Node-to-Node Communication Hadoop uses its own RPC protocol All communication begins in slave nodes Prevents circular-wait deadlock Slaves periodically poll for “status” message Classes must provide explicit serialization Nodes, Trackers, Tasks Master node runs JobTracker instance, which accepts Job requests from clients TaskTracker instances run on slave nodes TaskTracker forks separate Java process for task instances Job Distribution MapReduce programs are contained in a Java “jar” file + an XML file containing serialized program configuration options Running a MapReduce job places these files into the HDFS and notifies TaskTrackers where to retrieve the relevant program code … Where’s the data distribution? Data Distribution Implicit in design of MapReduce! All mappers are equivalent; so map whatever data is local to a particular node in HDFS If lots of data does happen