基于Spark软件栈的下一代大数据分析.pdfVIP

Next-Gen Big Data Analytics using the Spark stack Jason Dai Chief Architect of Big Data Technologies Software and Services Group, Intel Agenda Overview • Apache Spark stack • Next-gen big data analytics • Our vision and efforts Highlights of our work on Spark stack • Reliability of Spark Streaming • SQL processing on Spark • Spark Stream-SQL • Tachyon hierarchical storage • Analytics SparkR • … 2 Next-Gen Big Data Analytics Volume • Massive scale exponential growth Variety • Multi-structured, diverse sources inconsistent schemas Value • Simple (SQL) – descriptive analytics • Complex (non-SQL) – predictive analytics Velocity The Spark stack • Interactive – the speed of thought • Streaming/online – drinking from the firehose 3 Project Overview Research open source projects initiated by BDAS: Berkeley Data Analytics Stack (Ref: /software/) AMPLab in UC Berkeley Intel closely collaborating with AMPLab community on open source development • Started in 2012 when still a research project • Intel among top 3 Spark contributors – Multiple Intel committers since the start of the project – Many critical contributions – Netty based shuffle, FairScheduler, metrics system, “yarn-client” mode, … Continuous collaborations with AMPLab on new innovations in BDAS • Major industry partners for Tachyon, GraphX other machine learning efforts 4 Next Generations of Big Data Analytics Next-gen big data analytics paradigm • Data captured processed in a (semi-) real-time / streaming fashion • Data mined using SQL queries as well as large-scale advanced analytics (e.g.,