Hadoop技术产品综述.pdf

Hadoop技术产品综述 提纲 大数据与Hadoop概述 • 大数据概念、Hadoop概述 Hadoop体系结构 • 架构、 、 、生态圈 HDFS MR Hadoop 产品 • IDH Hadoop环境搭建 • 软硬件环境、实际部署设计 什么是大数据？ 最初定义: • Big data usually includes data sets with sizes beyond the ability of commonly-used software tools to capture, manage, and process the data within a tolerable elapsed time. 2001年 • Use “3Vs” (volume, velocity and variety) to describe the features of Big Data 2012年：Gartner • Big Data are high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization 视频 大数据的过去与未来 过去： • 科学研究：天文数据，地质数据，气象数据，基因组数据 当前： • 互联网企业： – 搜索与门户网站：Google ，Yahoo ，Baidu ，Sohu ，Sina ， NetEase – 社交媒体：facebook ，twitter, LinkedIn ，Youtube ，flickr ，优酷 – 电子商务：taobao • 传统企业：生产数据，物流数据，交易数据 未来： • 物联网：智能交通，智能医疗，智慧家庭、城市、地球 大数据示例 1.8ZB in 2011 750Million 3B pair 2 Days the dawn Photos every Base pairs of civilization to 2 days in analyzed for 2003 Facebook human genome 2.5 PB 140TB 200PB By 1M + Every 5 days with By 1M sensors transaction every the latest in a Smart City hour in wal-mart telescope in 2016 project 6TB $800B $300B/yr Every day by 20k in personal US healthcare servers in Baidu location data saving from Big within 10 years Data Data are becoming the new raw Info