最优化 Spark 应用的性能使用低成本的层次化方案加速大数据处理才.pdf

最优化Spark应用的性能 -- 使用高性价比的层次化方案加速大数据处理 Yucai, Yu ( BDT/STO/SSG April, 2016 About me/us • Me: Spark contributor, previous on virtualization, storage, OS etc. • Intel Spark team, working on Spark upstream development and x86 optimization, including: core, Spark SQL, Spark R, GraphX, machine learning etc. • Top 3 contribution in 2015, 3 committers. • Two publication: Intel Confidential 4/23/2016 2 Agenda • General software tuning • Bring up 3x performance with NVMe SSD • NVMe SSD Overview • Use NVMe SSD to accelerate computing • Why SSD is important to Spark Intel Confidential 4/23/2016 3 General software tuning • Resource allocation • Serialization • Partition • IO • MISC Intel Confidential 4/23/2016 4 Resource Allocation - CPU • spark.executor.cores – recommend 5 cores per executor* - Less core number (like single-core per executor) introduces JVM overhead. e.g., multiple broadcast copies - More cores number may hard to apply big resource - To achieve full write throughput onto HDFS • Number of executors per node – cores per node / 5 * (1 ~ 0.9) * /blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ Intel Confidential 4/23/2016 5 Resource Allocation - Memory • spark.executor.memory – memory size per executor - Leave 10-15% total memory for OS cache: dcache, page cache etc. - memory per node * (85-90)% / executors per node - 2-5 GB per core: 2-5GB * spark.executor.cores • spark.yarn.executor.memoryOverhead – indicate for offhe