演讲三 Spark在互联网广告监测行业的应用（刘喆）.pdfVIP

Spark 在互联网广告监测行业的应用 AdMaster ? Who are we • Monitor Ads on network • Data Solution Provider Why Spark • Fast • simple and direct(at least looks like) difficulties • Scala – Syntax – FP • RDD – Looks simple, just look How we use Spark • Spark on YARN • YARN : 200+ node * 64G Problem • many kind of jobs on YARN : MR, Storm, Spark • Memory Control Optimization • Do as little shuffle as possible • Memory statistic, then adjust configuration Where Spark helps us • Human grouping • IRGP weakness • Configure is compl