Hadoop MapReduce执行全流程关键代码.docVIP

Hadoop?Map/Reduce?执行流程关键代码?? ?? JobClient.runJob(conf)?|?运行job?? |--JobClient?jc?=?new?JobClient(job);?? |--RunningJob?rj?=?jc.submitJob(job);?? ????|--submitJobInternal(job);?? ????????|--int?reduces?=?job.getNumReduceTasks();?? ????????|--JobContext?context?=?new?JobContext(job,?jobId);?? ????????|--maps?=?writeOldSplits(job,?submitSplitFile);?? ????????|--job.setNumMapTasks(maps);?? ????????|--job.writeXml(out);?? ????????|--JobStatus?status?=?jobSubmitClient.submitJob(jobId);?? ---------------------------------------------------------------------------------------------- ?? JobTracker.submitJob(JobId)?|提交job?? |--JobInProgress?job?=?new?JobInProgress(jobId,?this,?this.conf);?? |--checkAccess(job,?QueueManager.QueueOperation.SUBMIT_JOB);??|检查权限?? |--checkMemoryRequirements(job);??|检查内存需求?? |--addJob(jobId,?job);??|添加至job队列?? ????|--jobs.put(job.getProfile().getJobID(),?job);?? ????|--?for?(JobInProgressListener?listener?:?jobInProgressListeners)?｜添加至监听器，供调度使用?? ????????|--listener.jobAdded(job);?? ?? ------------------------------------------------------------------------------------------------------ JobTracker.heartbeat()??|JobTracker启动后供TaskTracker以RPC方式来调用，返回Response集合?? |--ListTaskTrackerAction?actions?=?new?ArrayListTaskTrackerAction();?? |--tasks?=?taskScheduler.assignTasks(taskTrackerStatus);??|通过调度器选择合适的tasks?? |--for?(Task?task?:?tasks)?? ????|--expireLaunchingTasks.addNewTask(task.getTaskID());?? ????|--actions.add(new?LaunchTaskAction(task));??|实际actions还会添加commmitTask等?? |--response.setHeartbeatInterval(nextInterval);?? |--response.setActions(actions.toArray(new?TaskTrackerAction[actions.size()]));?? |--return?response;?? ?? ? --------------------------------------------------------------------------------------------------------- ? TaskTracker.offerService?|TaskTracker启动后通过offerservice()不断发心跳至JobTracker中?? |--transmitHeartBeat()?? ????|--HeartbeatResponse?heartbeatResponse?=?jobClient.heartbeat(status,?justStarted,?justInited,askForNewTask,?heartbea