A Tour of Apache Hadoop.pdfVIP

A Tour of Apache Hadoop Tom White Lexeme Ltd tomwhite@ Itinerary • What is Hadoop? • Components – Distributed File System – MapReduce – HBase • Related Projects What is Hadoop? The Problem • Existing tools are struggling to process todays large datasets • How long to grep 1TB of log files? • Why is this a problem for me? How Does Hadoop Help? • Hadoop provides a framework for storing and processing petabytes of data. • Storage: HDFS, HBase • Processing: MapReduce A Brief History of Hadoop • Feb 2003 – First MapReduce library written at Google • Oct 2003 – Google File System paper published • Dec 2004 – Google MapReduce paper published • Jul 2005 – Doug Cutting reports that Nutch now uses new MapReduce implementation • Nov 2006 – Google Bigtable paper published • Feb 2006 – Hadoop code moves out of Nutch into new Lucene sub-project • Feb 2007 – First HBase code drop from Mike Cafarella • Apr 2007 – Yahoo! running Hadoop on 1000-node cluster • Jan 2008 – Hadoop made an Apache Top Level Project Hadoop Organization • Apache Top Level Project • Two sub-projects – Core (15 committers) • HDFS • MapReduce – HBase (3 committers) Hadoop Users • Clusters from 1 to 2000 nodes – A9, Facebook, Joost, Last.fm, Yahoo! and many more • Broad academic interest – IBM/Google cloud computing initiative – CMU/Yahoo! supercomputing cluster • Hadoop Summit hosted by Yahoo! last month attracted over 300 attendees Hadoop Growth Hadoop Distributed File System Hadoop Distributed File System - Goals • Store large data sets • Cope with hardware failure • Emphasise streaming data