2013_0301_hawq_sql_engine_hadoop_0.pdfVIP

hawq IntroductIon Today’s most successful companies use data to their advantage. The data are no longer easily quantifiable facts, such as point of sale transaction data. Rather, these companies retain, explore, analyze, and manipulate all the available information in their purview. Ultimately, they search for evidence of facts, insights that lead to new business opportunities or which leverage their existing strengths. This is the business value behind what is often referred to as Big Data. Hadoop has proven to be a technology well suited to tackling Big Data problems. The fundamental principle of the Hadoop architecture is to move analysis to the data, rather than moving the data to a system that can analyze it. Ideally, Hadoop takes advantage of the advances in commodity hardware to scale in the way companies want. There are two key technologies that allow users of Hadoop to successfully retain and analyze data: HDFS and MapReduce. HDFS is a simple but extremely powerful distributed file system. It is able to store data reliably at significant scale. HDFS deployments exist with thousands of nodes storing hundreds of petabytes of user data. MapReduce is parallel programming framework that integrates with HDFS. It allows users to express data analysis algorithms in terms of a small number of functions and operators, chiefly, a map function and a reduce function. The success of MapReduce is a testament to the robustness of HDFS — both as a system to restore and access data, and as an application programming interface (API) for Big Data analysis frameworks. While MapReduce is convenient when performing scheduled analysis or manipulation of data stored on HDFS, it is not suitable for interactive use: it is too slow and lacks the expressive power required. SQL SQL is the language of data analysis. No language is more expressive, or more commonly used, for this task. Companies that bet big on Hadoop — such as Facebook, Twitter and especially Yahoo!