hadoop学习心得.docVIP

FileInputFormat splits only large files. Here “large” means larger than an HDFS block. The split size is normally the size of an HDFS block, which is appropriate for most applications; however,it is possible to control this value by setting various Hadoop properties. 2.So the split size is blockSize. 3.Making the minimum split size greater than the block size increases the split size, but at the cost of locality. One reason for this is that FileInputFormat generates splits in such a way that each split is all or part of a single file. If the file is very small (“small” means significantly smaller than an HDFS block) and there are a lot of them, then each map task will process very little input, and there will be a lot of them (one per file), each of which imposes extra bookkeeping overhead. hadoop处理大量小数据文件效果不好hadoop对数据的处理是分块处理的，默认是64M分为一个数据块，如果存在大量小数据文件（例如：2-3M一个的文件）这样的小数据文件远远不到一个数据块的大小就要按一个数据块来进行处理。这样处理带来的后果由两个：1.存储大量小文件占据存储空间，致使存储效率不高检索速度也比大文件慢。2.在进行MapReduce运算的时候这样的小文件消费计算能力，默认是按块来分配Map任务的（这个应该是使用小文件的主要缺点）那么如何解决这个问题呢？1.使用Hadoop提供的Har文件，Hadoop命令手册中有可以对小文件进行归档。2.自己对数据进行处理，把若干小文件存储成超过64M的大文件。 is the base class for all implementations of InputFormat that use files as their data source (see Figure 7-2). It provides two things: a place to define which files are included as the input to a job, and an implementation for generating splits for the input files. The job of dividing splits into records is performed by subclasses. An InputSplit has a length in bytes, and a set of storage locations, which are just hostname strings. Notice that a split doesn’t contain the input data; it is just a reference to the data. As a MapReduce application writer, you don’t need to deal with InputSplits directly, as they are created by an InputFormat. An InputFormat is responsible for creating the input splits, and dividing them into records. Before we see some concrete examples of InputFormat, let’s briefly examine how it is used in MapReduce. Here’s the interface: public interfac