Hadoop之JobTrack分析.docVIP

? Hadoop之JobTrack分析 1.client端指定Job的各种参数配置之后调用job.waitForCompletion(true)?方法提交Job给JobTracker，等待Job 完成。 [java]?view plaincopyprint? public?void?submit()?throws?IOException,?InterruptedException,??? ?????????????????????????????ClassNotFoundException?{?? ???ensureState(JobState.DEFINE);//检查JobState状态?? ???setUseNewAPI();//检查及设置是否使用新的MapReduce?API?? ????? ???//?Connect?to?the?JobTracker?and?submit?the?job?? ???connect();//链接JobTracker?? ???info?=?jobClient.submitJobInternal(conf);//将job信息提交?? ???super.setJobID(info.getID());?? ???state?=?JobState.RUNNING;//更改job状态?? ??}?? 以上代码主要有两步骤,连接JobTracker并提交Job信息。connect方法主要是实例化JobClient对象，包括设置JobConf和init工作： [java]?view plaincopyprint? public?void?init(JobConf?conf)?throws?IOException?{?? ????String?tracker?=?conf.get(mapred.job.tracker,?local);//读取配置文件信息用于判断该Job是运行于本地单机模式还是分布式模式?? ????tasklogtimeout?=?conf.getInt(?? ??????TASKLOG_PULL_TIMEOUT_KEY,?DEFAULT_TASKLOG_TIMEOUT);?? ????this.ugi?=?UserGroupInformation.getCurrentUser();?? ????if?(local.equals(tracker))?{//如果是单机模式，new?LocalJobRunner??? ??????conf.setNumMapTasks(1);?? ??????this.jobSubmitClient?=?new?LocalJobRunner(conf);?? ????}?else?{?? ??????this.jobSubmitClient?=?createRPCProxy(JobTracker.getAddress(conf),?conf);?? ????}?????????? ??}?? 分布式模式下就会创建一个RPC代理链接： [java]?view plaincopyprint? public?static?VersionedProtocol?getProxy(?? ??????Class??extends?VersionedProtocol?protocol,?? ??????long?clientVersion,?InetSocketAddress?addr,?UserGroupInformation?ticket,?? ??????Configuration?conf,?SocketFactory?factory,?int?rpcTimeout)?throws?IOException?{?? ?? ????if?(UserGroupInformation.isSecurityEnabled())?{?? ??????SaslRpcServer.init(conf);?? ????}?? ????VersionedProtocol?proxy?=?? ????????(VersionedProtocol)?Proxy.newProxyInstance(?? ????????????protocol.getClassLoader(),?new?Class[]?{?protocol?},?? ????????????new?Invoker(protocol,?addr,?ticket,?conf,?factory,?rpcTimeout));?? ????long?serverVersion?=?proxy.getProtocolVersion(protocol.getName(),??? ???????????????????????????