Spark Core技术平台详解.pdf

技术创新，变革未来 Spark Core技术平台详解 内容 Spark RDD Spark Operation RDD容错与持久化 Spark优化 Spark例子 Spark RDD Resilient Distributed Dataset (RDD): 分布式数据集 （分布在集群、已分区的） 存储在内存或磁盘上 只读的，不可变 自动重建 （容错） 有两类操作 transformations actions Spark Transformations and Actions transformations 通过其他RDD构建新的RDD map, filter, join lazy operation actions 返回计算结果或者保存到文件系统 count, collect, save triggers execution Spark Transformations and Actions Spark RDD Transformations Spark RDD Transformations Spark RDD Actions Spark Pair RDD Transformations Spark Pair RDD Transformations Spark RDD Lineage val inputRDD = sc.textFile(log.txt) val errorsRDD = inputRDD.filter(line = line.contains(error)) val warningsRDD = inputRDD.filter(line = line.contains(“warning)) badLinesRDD = errorsRDD.union(warningsRDD) println(Input had + badLinesRDD.count() + concerning lines) An RDD has enough information about how it was derived from other datasets(its lineage) RDD容错 通过RDD Lineage，重新计算丢失的数据，如 messages = textFile(...).filter(_.contains(“error”)) .map(_.split(‘\t’)) HadoopRDD FilteredRDD MappedRDD path = hdfs://… func = _.contains(...) func = _.split(…) RDD Persistence 缓存级别 memory_only (default), memory_and_disk等 API