谷歌大规模排序实验的历史[翻译].doc

WORD文档下载可编辑 专业技术资料分享 原文链接： HYPERLINK /blog/big-data/2016/02/history-of-massive-scale-sorting-experiments-at-google \t /cgi-bin/_blank /blog/big-data/2016/02/history-of-massive-scale-sorting-experiments-at-google 作者：Marian Dvorsky，软件工程师，谷歌云平台 HYPERLINK /blog/big-data/2016/02/history-of-massive-scale-sorting-experiments-at-google.html \t /cgi-bin/_blank History of massive-scale sorting experiments at Google 谷歌大规模排序实验的历史 Thursday, February 18, 2016 星期四，2016年2月18日 We’ve tested MapReduce by sorting large amounts of random data ever since we created the tool. We like sorting, because it’s easy to generate an arbitrary amount of data, and it’s easy to validate that the output is correct. 我们发明了MapReduce这个工具之后，对它进行了大规模随机数据的排序测试。我们喜欢排序，因为很容易产生任意规模的数据，也很容易验证排序的输出是否正确。 Even the? HYPERLINK /media//en//archive/mapreduce-osdi04.pdf \t /cgi-bin/_blank original MapReduce paper?reports a TeraSort result. Engineers run 1TB or 10TB sorts as regression tests on a regular basis, because obscure bugs tend to be more visible on a large scale. However, the real fun begins when we increase the scale even further. In this post I’ll talk about our experience with some petabyte-scale sorting experiments we did a few years ago, including what we believe to be the largest MapReduce job ever: a 50PB sort. 我们最初的MapReduce论文就报道了一个TeraSort排序的结果。工程师在一定的规则基础上对1TB或10TB的数据进行排序测试，因为细小的错误更容易在大规模数据运行的时候被发现。然而，真正有趣的事情在我们进一步扩大数据规模后才开始。在这篇文章中，我将讲一讲我们在几年之前所做的一些PB级别的排序实验，包括我们认为是目前最大的MapReduce工作：50PB排序。 These days, GraySort is the large scale sorting benchmark of choice. In GraySort, you must sort at least 100TB of data (as 100-byte records with the first 10 bytes being the key), lexicographically, as fast as possible. The site? HYPERLINK / \t /cgi-bin/_blank ?tracks official winners for this benchmark. We never entered the official competition. 那时候，GraySort是大型排序基准的选择。在GraySort基准下，你必须按照尽快对至少100TB的数据(每100B数据用最前面的10B数据作为键)进行字典序排序。S这个网站追踪报道了这个基准的官方优胜者。而我们从未正式参加过比赛。 MapReduce happens