大规模数据处理云计算.pdf

大规模数据处理/云计算 Lecture 1 Introduction to MapReduce 闫宏飞 北京大学信息科学技术学院 7/5/2011 /~course/cs402/ Jimmy Lin University of Maryland 课程建设 SEWMGroup This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States See /licenses/by-nc-sa/3.0/us/ for details What is this course about? • Data-intensive information processing • Large-data (“web-scale”) problems • Focus on MapReduce programming • An entry-level course~ 2 What is MapReduce? • Programming model for expressing distributed computations at a massive scale • Execution framework for organizing and performing such computations • Open-source implementation called Hadoop 3 Why Large Data? How much data?  Google processes 20 PB a day (2008)  Wayback Machine has 3 PB + 100 TB/month (3/2009)  Facebook has 2.5 PB of user data + 15 TB/day (4/2009)  eBay has 6.5 PB of user data + 50 TB/day (5/2009)  CERN’s LHC will generate 15 PB a year (??) 640K ought to be enough for anybody. 5 6 Happening everywhere! Molecular biology microarray chips (cancer) fiber optics Network traffic (spam)