hadoop集群环境安装及测试.docx

三台Ubuntu16.04虚拟机，主机名分别为master-hadoop和slaver0，slaver1配置ssh无密码登录：三台虚拟机(master-hadoop，slaver0，slaver1)安装ssh协议：apt-get install sshapt-get install rsync2、master-hadoop执行(ssh-keygen –trsa-P “”),点击回车,生成密钥。3、在master-hadoop上执行:?scp ~/.ssh/id_rsa.pub mailto:kong@slaver0:~/.sshkong@slaver0:~/.ssh?scp ~/.ssh/id_rsa.pub mailto:kong@slaver1:~/.sshkong@slaver1:~/.ssh 把master-hadoop的公钥分别复制到slaver0、slaver1中4、在虚拟机上执行?cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys 进行密钥授权5、虚拟机执行service ssh restart 重启ssh服务，然后进行ssh测试无密码链接。安装JDK（master-hadoop、slaver0、slaver1）：下载JDK1.7（不要用1.8，hadoop不能有效的支持jdk1.8），解压到/usr/local/java/jdk中配置环境变量：执行vim /etc/profile在profile最后添加上JAVA_HOME=/usr/local/java/jdkCLASSPATH=${JAVA_HOME}/lib:${JAVA_HOME}/jre/libPATH=${JAVA_HOME}/bin:$PATHexport JAVA_HOME PATH 执行 source /etc/profile 执行java –version验证jdk环境是否成功搭建安装hadoop(master-hadoop)：下载Hadoop2.7.3，解压到/home/kong/hadoop中设置环境变量：执行vim /etc/profile末尾添加HADOOP_HOME=/home/kong/HadoopPATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin设置文件（以下文件都在/home/kong/hadoop/etc/hadoop/下）:1、在/home/kong/hadoop中新建tmp文件夹；在/home/kong/hadoop/share/hadoop/hdfs中新建data和name文件夹2、在slaves文件中添加slaver0、slaver13、在hadoop-env.sh文件最后添加export JAVA_HOME=/usr/local/java/jdk/4、在core-site.xml中添加configuration property //主机名和端口号namefs.defaultFS/name valuehdfs://master-hadoop:9000/value //上传下载文件所用，rpc协议，不能使用http协议访问/property property nameio.file.buffer.size/name value131072/value /property property //存储路径namehadoop.tmp.dir/name //临时文件valuefile:/home/kong/hadoop/tmp/value /property /configuration 5、在hdfs-site.xml中添加configuration property name.dir/name valuefile:/home/kong/hadoop/share/hadoop/hdfs/name/value /property property namenode.data.dir/name valuefile:/home/kong/hadoop/share/hadoop/hdfs/data/value /property property namedfs.replication/name !--数据副本数量，默认3，设置为1-- value1/value /property propertynamedfs.permissions/namevaluefalse/value/property/configuration 6、yarn-site.xmlconfiguration property nameyarn.resourcemanager.address/name valuemaster-hadoop:8032/value