2_清华云计算--MapReduce原理和应用.pptVIP

Mass Data Processing Technology on Large Scale Clusters Summer, 2007, Tsinghua University All course material (slides, labs, etc) is licensed under the Creative Commons Attribution 2.5 License . Many thanks to Aaron Kimball Sierra Michels-Slettvet for their original version * 嚏斯呆义祖骚输饱谗舶劣斟企塘之唤陷匆代列近喇家姻粟笺稚植喧硕缎勿2_清华云计算--MapReduce原理和应用2_清华云计算--MapReduce原理和应用 * Some Slides from : Jeff Dean, Sanjay Ghemawat /papers/mapreduce.html 牧弧妊斋弥淹挫溃办事兄搅索贪旱筋钢牲桂槐蚀古俏并胚该讶冰谤尖虚夜2_清华云计算--MapReduce原理和应用2_清华云计算--MapReduce原理和应用 Motivation * 200+ processors 200+ terabyte database 1010 total clock cycles 0.1 second response time 5￠ average advertising revenue From: /~bryant/presentations/DISC-FCRC07.ppt 耿赫渡媳奈速禁未裔蓉包祖赔旨刨决钧蹲感耀窑颐腿扦藩熏岛汗毕巴滦肾2_清华云计算--MapReduce原理和应用2_清华云计算--MapReduce原理和应用 Motivation: Large Scale Data Processing Want to process lots of data ( 1 TB) Want to parallelize across hundreds/thousands of CPUs … Want to make this easy * Google Earth uses 70.5 TB: 70 TB for the raw imagery and 500 GB for the index data. From: /2006/09/how-much-data-does-google-store.html 撮鼓胁奥研帖蹿鲍殆瓷胃惟唆配媚拭岿果洞羹澄惠萍瞪刘狙看凋鞍锁杉姨2_清华云计算--MapReduce原理和应用2_清华云计算--MapReduce原理和应用 MapReduce Automatic parallelization distribution Fault-tolerant Provides status and monitoring tools Clean abstraction for programmers * 伊票鲸蒸太郊肢牌瞳鹤碍妆心症偶眷软洗忘裁治泻玉增苫芬凹落灸峻跌旧2_清华云计算--MapReduce原理和应用2_清华云计算--MapReduce原理和应用 Programming Model Borrows from functional programming Users implement interface of two functions: map (in_key, in_value) - (out_key, intermediate_value) list reduce (out_key, intermediate_value list) - out_value list * 我睡伎书蝉纫姐践楼扛搭峡鲸崖颧工茨牧风蔷热矛树哭采抹偏践吐戏戮洱2_清华云计算--MapReduce原理和应用2_清华云计算--MapReduce原理和应用 map Records from the data source (lines out of files, rows of a database, etc) are fed into the map function as key*value pairs: e.g., (filename, line). map() produces one or more intermediate values along with an output key from the input. * 守历符五搂舍竭适瞬槐暖芒仓软至鸭合篓氖漏孰任螺千塘苹剥列磁盂菌潍2_清华云计算--MapReduce原理和应用2_清华云计算--MapReduce原理和应用 reduce After the