spark-1-151203050842-lva1-app6891.pdfVIP

Apache Spark Release 1.6 Patrick Wendell About Me @pwendell U.C. Berkeley PhD, left to co-found Databricks Coordinate community roadmap Frequent release manager for Spark About Databricks Founded by Spark team, donated Spark to Apache in 2013 and lead development today. Collaborative, cloud-hosted data platform powered by Spark Free trial to check it out / We’re hiring! Apache Spark Engine Scale out, fault tolerant Spark Spark SQL MLlib GraphX Streaming Python, Java, Scala, and R APIs Spark Core Standard libraries … Unified engine across diverse workloads environments Users Distributors Apps Spark’s 3 Month Release Cycle For production jobs, use the latest release V1.6.0 V1.6.1 To try out unreleased features or fixes, use nightly builds /~ pwendell/spark-nightly/ branch-1.6 master Spark 1.6 Spark 1.6 Release Will ship upstream through Apache foundation in December (likely) Key themes Out of the box performance Previews of key new API’s Follow along with me at http://bit.ly/1OBkjMM Follow along http://bit.ly/1lrvdLc Memory Management in Spark: = 1.5 • Two separate memory managers: • Execution memory: computation of shuffles, joins, sorts, aggregations • Storage memory: caching and propagating internal data sources across cluster • Challenges with this: • Manual interv