《Hadoop Job Optimization (Microsoft IT white paper)》.docx

Hadoop Job Optimization Microsoft IT SES Enterprise Data Architect Team  SAVEDATE \@ yyyy-MM-dd \* MERGEFORMAT 2013-04-17 In this document, we describe specific types of Hadoop job configurations that can act as bottlenecks and hurt overall performance. We provide techniques that you can use to remediate each type of bottleneck, and provide suggestions for general optimization of your Hadoop jobs.  HYPERLINK \l _Overview_of_Hadoop Section 1 provides a short introduction to how MapReduce works internally.  HYPERLINK \l _Identifying_Bottlenecks Section 2 defines a common performance tuning framework that you can use to guide repeatable processes. We explain how to identify resource bottlenecks based on performance indicators.  HYPERLINK \l _Removing_Bottlenecks Section 3 describes tuning techniques in detail, to help you to address and remove performance bottlenecks. We also provide a matrix to guide selection of which technique to apply to each performance problem. Table of Contents  TOC \o 1-2 \h \z \u  HYPERLINK \l _Toc353902897 Introduction  PAGEREF _Toc353902897 \h 3  HYPERLINK \l _Toc353902898 Overview of Hadoop Performance Tuning  PAGEREF _Toc353902898 \h 5  HYPERLINK \l _Toc353902899 Common Resource Bottlenecks  PAGEREF _Toc353902899 \h 9  HYPERLINK \l _Toc353902900 Identifying Bottlenecks  PAGEREF _Toc353902900 \h 11  HYPERLINK \l _Toc353902901 Choosing a Solution  PAGEREF _Toc353902901 \h 24  HYPERLINK \l _Toc353902902 Resolving Bottlenecks  PAGEREF _Toc353902902 \h 26  HYPERLINK \l _Toc353902903 Conclusion  PAGEREF _Toc353902903 \h 49  HYPERLINK \l _Toc353902904 Appendix A: Join Optimization in Hive  PAGEREF _Toc353902904 \h 51  HYPERLINK \l _Toc353902905 Appendix B: List of Tuning Properties in Shuffle  PAGEREF _Toc353902905 \h 53  HYPERLINK \l _Toc353902906 Appendix C: Miscellaneous Tuning Properties  PAGEREF _Toc353902906 \h 56  HYPERLINK \l _Toc353902907 References and