Apache官方文档 --- Hadoop Streaming.pdfVIP

Hadoop Streaming Content-Type text/html; utf-8 Table of contents 1 Hadoop Streaming3 2 How Does Streaming Work 3 3 Package Files With Job Submissions4 4 Streaming Options and Usage 4 4.1 Mapper-Only Jobs 4 4.2 Specifying Other Plugins for Jobs 4 4.3 Large files and archives in Hadoop Streaming 5 4.4 Specifying Additional Configuration Variables for Jobs 6 4.5 Other Supported Options 7 5 More usage examples 8 5.1 Customizing the Way to Split Lines into Key/Value Pairs 8 5.2 A Useful Partitioner Class (secondary sort, the -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner option) 9 5.3 Working with the Hadoop Aggregate Package (the -reduce aggregate option) 10 5.4 Field Selection ( similar to unix cut command) 11 6 Frequently Asked Questions 11 6.1 How do I use Hadoop Streaming to run an arbitrary set of (semi-)independent tasks? 12 6.2 How do I process files, one per map? 12 6.3 How many reducers should I use? 12 6.4 If I set up an alias in my shell script, will that work after -mapper, i.e. say I do: alias c1=cut -f1. Will -mapper c1 work? 12 6.5 Can I use UNIX pipes? For example, will -mapper cut -f1 | sed s/foo/bar/g work?13 6.6 When I run a streaming job by distributing large executables (for example, 3.6G) Copyright © 2007 The Apache Software Foundation. All rights reserved. Hadoop Streaming through the -file option, I get a No space left on device error. What do I do? 13 6.7 How do I specify multiple input directories? 13 6.8 How do I generate output files with gzip format? 13 6.9 How do I provide my own input/output format with streaming? 14 6.10 How do I parse XML documents using streaming? 14