学习_hive源码.doc

学习 Hive 李建奇 学习 看了一部分代码，感觉，hive 比较复杂，使用场景有限，一般用 hadoop 原生的 map reduce 就可以了。 版本 0.6 目的 学习 facebook 等应用 hive 的经验，以便应用于公司。 学习代码的目的是便于更好的应用，比如debuging , tuning . 以及应用新的patch. 等。 Pig + Hive : ETL + data warehouse The data preparation phase is often known as ETL (Extract Transform Load) or the data factory. Factory is a good analogy because it captures the essence of what is being done in this stage: Just as a physical factory brings in raw materials and outputs products ready for consumers, so a data factory brings in raw data and produces data sets ready for data users to consume. Raw data is loaded in, cleaned up, conformed to the selected data model, joined with other data sources, and so on. Users in this phase are generally engineers, data specialists, or researchers. The data presentation phase is usually referred to as the data warehouse. A warehouse stores products ready for consumers; they need only come and select the proper products off of the shelves. In this phase, users may be engineers using the data for their systems, analysts, or decisionmakers. Given the different workloads and different users for each phase, we have found that different tools work best in each phase. Pig (combined with a workflow system such as Oozie) is best suited for the data factory, and Hive for the data warehouse. data warehouse Data warehouse use cases In the data warehouse phase of processing, we see two dominant use cases: business-intelligence analysis and ad-hoc queries. In the first case, users connect the data to business intelligence (BI) tools — such as MicroStrategy — to generate reports or do further analysis. In the second case, users run ad-hoc queries issued by data analysts or decisionmakers. In both cases, the relational model and SQL are the best fit. Indeed, data warehousing has been one of the core use cases for SQL through much of its history. It has the right constructs to support the types of queries and tools that analysts want to use. And it is a