Hadoop大数据处理技术基础与实践第三章 HDFS 技术.ppt

配置文件加载顺序 先加载缺省配置文件，再加载用户定义的配置文件 对于每一个文件只加载一次：第一个在classpath出现的 classpath的目录顺序 $HADOOP_CONF_DIR $JAVA_HOME/lib/tools.jar 如果$HADOOP_HOME目录下有build目录，则添加build下各子目录 $HADOOP_HOME/hadoop-core-*.jar $HADOOP_HOME/lib/*.jar 用户在hadoop-env.sh中定义的$HADOOP_CLASS_PATH 当前作为hadoop jar …参数提交的JAR包 * 第3章 * get/copyToLocal 用途 将HDFS中的一个或多个文件导出到本地文件系统 使用方法 hadoop fs -get/copyToLocal [-ignorecrc] [-crc] URI localsrc 例子: hadoop fs -get /user/hadoop/hadoopfile localfile hadoop fs -get hdfs://host:port/user/hadoop/file localfile * 第3章 * ls/lsr 用途 列出文件夹目录信息 lsr递归显示文件子目录 使用方法 hadoop fs -ls/lsr URI 例子: [hadoop@master]$ hadoop fs -ls /user/hadoop drwxr-xr-x - hadoop supergroup 0 2012-05-14 17:24 /user/hadoop/a [hadoop@master]$ hadoop fs -lsr /user/hadoop drwxr-xr-x - hadoop supergroup 0 2012-05-14 17:24 /user/hadoop/a -rw-r--r-- 2 hadoop supergroup 47 2012-05-14 17:24 /user/hadoop/a/oct.sh -rw-r--r-- 2 hadoop supergroup 6 2012-05-14 17:24 /user/hadoop/a/x.sh * 第3章 * fsck 用法 hadoop fsck [GENERIC_OPTIONS] path [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]] 检查dfs的文件的健康状况 只能运行在master上 [hadoop@master]$ hadoop fsck /user/hadoop FSCK started by hadoop from /7 for path /user/hadoop at Fri Jun 15 17:11:50 CST 2012 . /user/hadoop/a/oct.sh: Under replicated blk_-3537883739360629007_1303. Target Replicas is 2 but found 1 replica(s). . /user/hadoop/a/x.sh: Under replicated blk_8664072905926163771_1304. Target Replicas is 2 but found 1 replica(s). * 第3章 * fsck Status: HEALTHY Total size: 57697 B Total dirs: 5 Total files: 6 Total blocks (validated): 5 (avg. block size 11539 B) Minimally replicated blocks: 5 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 5 (100.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 2 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 2 (100.0 %) Number of data-nodes: