Spark Streaming源码解读之Driver中的ReceiverTracker详解.doc

Spark Streaming源码解读之Driver中的ReceiverTracker详解 本篇博文的目标是： Driver的ReceiverTracker接收到数据之后，下一步对数据是如何进行管理 一：ReceiverTracker的架构设计 1. Driver在Executor启动Receiver方式，每个Receiver都封装成一个Task，此时一个Job中就一个Task,而Task中就一条数据，也就是Receiver数据。由此，多少个Job也就可以启动多少个Receiver. 2. ReceiverTracker在启动Receiver的时候他有ReceiverSupervisor,其实现是ReceiverSupervisorImpl, ReceiverSupervisor本身启动的时候会启动Receiver，Receiver不断的接收数据，通过BlockGenerator将数据转换成Block。定时器会不断的把Block数据通过BlockManager或者WAL进行存储，数据存储之后ReceiverSupervisorImpl会把存储后的数据的元数据Metadate汇报给ReceiverTracker,其实是汇报给ReceiverTracker中的RPC实体ReceiverTrackerEndpoint. ReceiverSupervisorImpl会将元数据汇报给ReceiverTracker,那么接收到之后,下一步就对数据进行管理。 通过receivedBlockHandler写数据 private val receivedBlockHandler: ReceivedBlockHandler = { if (WriteAheadLogUtils.enableReceiverLog(env.conf)) { if (checkpointDirOption.isEmpty) { throw new SparkException( Cannot enable receiver write-ahead log without checkpoint directory set. + Please use streamingContext.checkpoint() to set the checkpoint directory. + See documentation for more details.) } //WAL new WriteAheadLogBasedBlockHandler(env.blockManager, receiver.streamId, receiver.storageLevel, env.conf, hadoopConf, checkpointDirOption.get) } else { //BlockManager new BlockManagerBasedBlockHandler(env.blockManager, receiver.storageLevel) } } 2. PushAndReportBlock存储Block数据，且把信息汇报给Driver。 /** Store block and report it to driver */ def pushAndReportBlock( receivedBlock: ReceivedBlock, metadataOption: Option[Any], blockIdOption: Option[StreamBlockId] ) { val blockId = blockIdOption.getOrElse(nextBlockId) val time = System.currentTimeMillis val blockStoreResult = receivedBlockHandler.storeBlock(blockId, receivedBlock) logDebug(sPushed block $blockId in ${(System.currentTimeMillis - time)} ms) val numRecords = blockStoreResult.numRecords val blockInfo = ReceivedBlockInfo(streamId, numRecords, metadataOption, blockStoreResult) trackerEndpoint.askWithRetry[Boolean](A