OReilly.Learning.Spark.2015.1-CN-13-Chapter4.pdf

Learning Spark LIGHTNING-FAST DATAANALYSIS Holden Karau, Andy Konwinski, PatrickWendellMateiZaharia CHAPTER 4 Working with Key/Value Pairs This chapter covers how to work with RDDs of key/value pairs, which are a common data type required for many operations in Spark. Key/value RDDs are commonly used to perform aggregations, and often we will do some initial ETL (extract, trans‐ form, and load) to get our data into a key/value format. Key/value RDDs expose new operations (e.g., counting up reviews for each product, grouping together data with the same key, and grouping together two different RDDs). 本章介绍了如何使用key/value RDD，这是Spark 最常见的数据类型。key/valueRDD 常用于进行数据聚合，而且往往我们会做一些初步的ETL （提取，传输，加载）， 以把我们的数据转成key/value 格式。key/value RDD 提供更多操作（例如，统计每 个产品的评论数，统计相同键的数据，把两个不同的RDD 组合在一起）。 We also discuss an advanced feature that lets users control the layout of pair RDDs across nodes: partitioning. Using controllable partitioning, applications can some‐ times greatly reduce communication costs by ensuring that data will be accessed together and will be on the same node. This can provide significant speedups. We illustrate partitioning using the PageRank algorithm as an example. Choosing the right partitioning for a distributed dataset is similar to choosing the right data struc‐ ture for a local one—in both cases, data layout can greatly affect performance. 我们会进一步讨论让用户控制各节点之间RDD 的布局，即分区策略。使用可控的分 区，把常常同时访问的数据放在同一节点，将大大降低通信费用，提升访问速率。 我们将以PageRank 算法为例介绍分区策略。为分布式数据集选择正确的分区策略的 重要性类似于为本地数据选择正确的数据结构。在这两种环境下，分区策略和数据 结构都会极大地影响系统性能。 Motivation Spark provides special operations on RDDs containing key/value pairs. These RDDs are called pair RDDs. Pair RDDs are a useful building block in many programs, as 50 | Chapte