大规模数据分析方法比较.doc

一个方法比较大规模数据分析 ABSTRACT 摘要 There is currently considerable enthusiasm around the MapReduce (MR) paradigm for large-scale data analysis [17]. 目前有相当大的积极性MapReduce模式的数据分析 [17]。 Although the basic control ?ow of this framework has existed in parallel SQL database management systems (DBMS) for over 20 years, some have called MR a dramatically new computing model [8, 17]. 虽然这个框架的基本控制流存在并行SQL数据库管理系统超过20年，有称为新计算模型 [8，17]。 In this paper, we describe and compare both paradigms. 在本文中，我们描述和比较这两个。 Furthermore, we evaluate both kinds of systems in terms of performance and development complexity. 此外，我们系统的复杂。 To this end, we de?ne a benchmark consisting of a collection of tasks that we have run on an open source version of MR as well as on two parallel DBMSs. ，我们定义一个基准数据库管理系统上。 For each task, we measure each systems performance for various degrees of parallelism on a cluster of 100 nodes. 对于每个任务，我们100台机子的集群上衡量每个系统的各种程度的并行。 Our results reveal some interesting trade-offs. 我们的研究结果揭示了一些有趣的取舍。 Although the process to load data into and tune the execution of parallel DBMSs took much longer than the MR system, the observed performance of these DBMSs was strikingly better. 虽然加载数据并行数据库管理系统执行，这些数据库管理系统观察到的效果改善。 We speculate about the causes of the dramatic performance difference and consider implementation concepts that future systems should take from both kinds of architectures. 我们推测性能差异的原因，并考虑将来的系统应该从两种架构实施，。 1. 1。 INTRODUCTION 简介 Recently the trade press has been ?lled with news of the revolution of “cluster computing”. 最近商业媒体上已经充满了“计算集群”。 This paradigm entails harnessing large numbers of (low-end) processors working in parallel to solve a computing problem.这种模式利用大量的（低）处理器并行工作来解决计算问题。 In effect, this suggests constructing a data center by lining up a large number of low-end servers instead of deploying a smaller set of high-end servers. 实际上，这表明构建数据中心，由大量的服务器高。 With this rise of interest in clusters has come a proliferation of tools for programming them. 集群，他们的兴趣，。 One of the earliest a