Hadoop2.7集群安装.docxVIP

Hadoop2.7集群安装修改记录编号日期描述 版本作者审核发布日期12016/7/13初稿1.0夏裕太1目 录1准备12Host13SSH14文件目录25Hadoop的安装与配置25.1解压缩25.2环境变量35.3Hadoop的配置35.3.1core-site.xml35.3.2hdfs-site.xml45.3.3mapred-site.xml (新增)55.3.4yarn-site.xml55.3.5配置hadoop-env.sh65.3.6配置slaves65.3.7scp远程拷贝75.4运行Hadoop75.4.1运行HDFS格式化NameNode启动NameNode启动DataNode85.4.2SLAVE2上执行命令85.4.3SLAVE3上执行命令85.4.4start-dfs.sh 运行HDFS95.4.5运行YARN95.4.6start-yarn.sh 运行YARN105.5快捷启动整个hadoop115.6测试Hadoop125.6.1测试HDFS125.6.2测试YARN125.6.3测试mapreduce12准备1、JDK7，详情请参考《CentOS7下安装JDK7.docx》2、安装包hadoop-2.7.2.tar.gz三台机器41 MASTER42 SLAVE43 SLAVE3Host由于我搭建Hadoop集群包含三台机器，所以需要修改调整各台机器的hosts文件配置，命令如下三台服务器都要增加如下配置[root@MASTER1 bin]# vi /etc/hosts41 MASTER42 SLAVE43 SLAVE3SSH由于NameNode与DataNode之间通信，使用了SSH，所以需要配置免登录。首先登录Master机器，生成SSH的公钥，命令如下：[root@MASTER1 bin]# ssh-keygen -t rsa Generating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/id_rsa already exists.Overwrite (y/n)? yEnter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa.Your public key has been saved in /root/.ssh/id_rsa.pub.The key fingerprint is:ba:78:58:05:bf:36:a6:68:6a:b1:36:18:a1:b2:35:f8 root@MASTER1The keys randomart image is:+--[ RSA 2048]----+| || . || o ||. o ||.o .S. ||= + ..= ||.* + +.+ . ||o E +.o. || o.+... |执行命令后会在当前用户目录下生成.ssh目录，然后进入此目录将id_rsa.pub追加到authorized_keys文件中，命令如下：[root@MASTER1 bin]# cd ~/.ssh[root@MASTER1 .ssh]# cat id_rsa.pub authorized_keys最后将authorized_keys文件复制到其它机器节点,命令如下[root@MASTER1 .ssh]# scp authorized_keys root@SLAVE2:/root/.ssh/[root@MASTER1 .ssh]# scp authorized_keys root@SLAVE3:/root/.ssh/The authenticity of host slave3 (43) cant be established.ECDSA key fingerprint is 3e:77:b7:27:eb:c7:6c:d8:50:b1:1d:d2:8f:78:ee:2e.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added slave3,43 (ECDSA) to the list of known