Hadoop Basic.pptVIP

* * Hadoop Formally speaking, Hadoop is an open source framework for writing and running distributed applications that process large amounts of data. * Hadoop * A Hadoop cluster has many parallel machines that store and process large data sets. Client computers send jobs into this computer cloud and obtain results. Hadoop * Accessible—Hadoop runs on large clusters of commodity machines or on cloud computing services such as Amazon’s Elastic Compute Cloud (EC2 ). Robust—Because it is intended to run on commodity hardware, Hadoop is architected with the assumption of frequent hardware malfunctions. It can gracefully handle most such failures. Scalable—Hadoop scales linearly to handle larger data by adding more nodes to the cluster. Simple—Hadoop allows users to quickly write efficient parallel code. Motivation: Large Scale Data Processing Want to process lots of data ( 1 TB) Want to parallelize across hundreds/thousands of CPUs … Want to make this easy * Google Earth uses 70.5 TB: 70 TB for the raw imagery and 500 GB for the index data. From: /2006/09/how-much-data-does-google-store.html MapReduce Automatic parallelization distribution Fault-tolerant Provides status and monitoring tools Clean abstraction for programmers * Programming Model Borrows from functional programming Users implement interface of two functions: map (in_key, in_value) - (out_key, intermediate_value) list reduce (out_key, intermediate_value list) - out_value list * map Records from the data source (lines out of files, rows of a database, etc) are fed into the map function as key*value pairs: e.g., (filename, line). map() produces one or more intermediate values along with an output key from the input. * reduce After the map phase is over, all the intermediate values for a given output key are combined together into a list reduce() combines those intermediate values into one or more final values for that same output key (in practice, usually only one final value per key) * Architecture