中华电信用Hadoop技术分析通话明细-classcloud.PPT

Two Key Elements of Operating System作業系統兩大關鍵組成元素 Scheduler 程序排程 File System 檔案系統 * Distributed Operating System of HadoopHadoop建構成一個分散式作業系統 * Linuux Java Linuux Java Linuux Java Data Task Data Task Namenode JobTracker Hadoop Node1 Node2 Node3 Features of Hadoop ...Hadoop 這套軟體的特色是... 海量 Vast Amounts of Data 擁有儲存與處理大量資料的能力 Capability to STORE and PROCESS vast amounts of data. 經濟 Cost Efficiency 可以用在由一般PC所架設的叢集環境內 Based on large clusters built of commodity hardware. 效率 Parallel Performance 透過分散式檔案系統的幫助，以致得到快速的回應 With the help of HDFS, Hadoop have better performance. 可靠 Robustness 當某節點發生錯誤，能即時自動取得備份資料及佈署運算資源 Robustness to add and remove computing and storage resource without shutdown entire system. Founder of Hadoop – Doug CuttingHadoop 這套軟體的創辦人 Doug Cutting Doug Cutting Talks About The Founding Of Hadoop /watch?v=qxC4urJOchs History of Hadoop … 2002~2004Hadoop 這套軟體的歷史源起... 2002~2004 ? Lucene / 用Java 設計的高效能文件索引引擎API a high-performance, full-featured text search engine library written entirely in Java. 索引文件中的每一字，讓搜尋的效率比傳統逐字比較還要高的多 Lucene create an inverse index of every word in different documents. It enhance performance of text searching. History of Hadoop … 2002~2004Hadoop 這套軟體的歷史源起... 2002~2004 Nutch / Nutch 是基於開放原始碼所開發的網站搜尋引擎 Nutch is open source web-search software. 利用Lucene 函式庫開發 It builds on Lucene and Solr, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats, etc. History of Hadoop … 2004 ~ NowHadoop 這套軟體的歷史源起... 2004 ~ Now Nutch後來遇到儲存大量網站資料的瓶頸，剛好看到 Google在一些會議分享他們的三大關鍵技術 … Added DFS MapReduce implement to Nutch According to user feedback on the mail list of Nutch Hadoop became separated project since Nutch 0.8 Nutch DFS → Hadoop Distributed File System (HDFS) Yahoo hire Dong Cutting to build a team of web search engine at year 2006. Only 14 team members (engineers, clusters, users, etc.) Doung Cutting joined Cloudera at year 2009. Who Use Hadoop ??有哪些公司在用 Hadoop 這套軟體 ?? Yahoo is the key