State管理之updateStateByKey和mapWithState解密.doc

State管理之updateStateByKey和mapWithState解密 背景： 整个Spark Streaming是按照Batch Duractions划分Job的。但是很多时候我们需要算过去的一天甚至一周的数据，这个时候不可避免的要进行状态管理，而Spark Streaming每个Batch Duractions都会产生一个Job，Job里面都是RDD，所以此时面临的问题就是怎么对状态进行维护？这个时候就需要借助updateStateByKey和mapWithState方法完成核心的步骤。 源码分析： 1. 无论是updateStateByKey还是mapWithState方法在DStream中均没有，但是是通过隐身转换函数实现其功能。 object DStream { // `toPairDStreamFunctions` was in SparkContext before 1.3 and users had to // `import StreamingContext._` to enable it. Now we move it here to make the compiler find // it automatically. However, we still keep the old function in StreamingContext for backward // compatibility and forward to the following function directly. implicit def toPairDStreamFunctions[K, V](stream: DStream[(K, V)]) (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null): PairDStreamFunctions[K, V] = { new PairDStreamFunctions[K, V](stream) } updateStateByKey： 1. 在PairDStreamFunctions中updateStateByKey具体实现如下： 在已有的历史基础上，updateFunc对历史数据进行更新。该函数的返回值是DStream类型的。 /** * Return a new state DStream where the state for each key is updated by applying * the given function on the previous state of the key and the new values of each key. * Hash partitioning is used to generate the RDDs with Sparks default number of partitions. * @param updateFunc State update function. If `this` function returns None, then * corresponding state key-value pair will be eliminated. * @tparam S State type */ def updateStateByKey[S: ClassTag]( updateFunc: (Seq[V], Option[S]) = Option[S] ): DStream[(K, S)] = ssc.withScope { // defaultPartitioner updateStateByKey(updateFunc, defaultPartitioner()) } 2. defaultPartitioner: private[streaming] def defaultPartitioner(numPartitions: Int = self.ssc.sc.defaultParallelism) = { new HashPartitioner(numPartitions) 3. partitioner就是控制RDD的每个patition /** * Return a new state DStream where the state for each key is updated by applying * the given function on the previ