探秘淘宝Hadoop集群.docVIP

探秘淘宝Hadoop集群 当下中国超大规模的单Master节点Hadoop集群在哪里?在淘宝。据悉，淘宝Hadoop集群拥有2860个节点，清一色基于英特尔处理器的x86服务器，其总存储容量50PB，实际使用容量超过40PB，日均作业数高达15万，为淘宝网的日常运营做出了关键支撑。对了，它还有一个很美丽的名字:“云梯”。 近日，《网络世界》记者有幸采访到了阿里集团技术共享平台核心系统研发部海量数据技术专家罗李。作为淘宝“云梯”集群元老级创建者，以及目前的负责人，他详细讲解了Hadoop在淘宝的应用与发展。 “云梯”的前世今生 “在2008年之前，淘宝内部一些业务团队，纷纷搭建了自己的Hadoop集群，但规模都很小，出了问题也都各自解决。”罗李说，“考虑到淘宝的业务确实对分布式计算系统有需要，再加上这种分散模式小集群的资源利用率不高，整个集团就决定建立一个统一的、大的分布式计算集群，为各业务线服务。” 至于为什么选择Hadoop，罗李表示，当时能够达到商用水准的分布式计算技术也只有Hadoop。不过，他说:“当时整个淘宝并没有一个对Hadoop内核真正了解的人，后来是抽调了雅虎中国的核心技术人员，在2008年10月成立了专门团队。” equipment, components, equipment, temporary construction facilities, processed with strict compliance with provisions of construction floor plans, smooth roads and drainage, site formation. ----3, the site looks dignified and elegant and tidy, advertising slogans to write eye-catching appearance, walls to be clean and tidy. 4, six-second figure----cards to complete, custom, eye-catching, hanging in the obvious place. Management----5, district administration, launch of Qing dynasty focuses on managing the cleaning, recycling, use, four Central Bank; the work is finished, clean, clear, work product protection in all procedures to periodically check the appraisals in a timely manner. 6, clean health----engineering work sites, temporary facilities clean and tidy inside and outside. 7, mechanical health----engineering machinery, vehicles are maintained in good condition, looks clean, and free from dirt, dust, electrical switchgear (box) complete with lock, mechanical safety protection equipment and complete, sensitive and reliable. 8, fire, defend----perfect fire system, set in accordance with requirements of fire equipment, set up obvious fire safety signs and signage, provide effective fire-fighting equipment, establishment of a security system, and out of the site staff to wear to work, marks and signs, implements “2008年我也是刚进淘宝，就直接被分配到这个团队，那时整个团队包括我才四个人。到2009年，我们的?云梯?集群刚搭起来的时候，才300台规模。”罗李告诉记者。 短短不过四年时间，“云梯”集群已经扩张到近2900台规模，成为中国规模最大的单Master节点Hadoop集群