张晓东解析并行处理在大数据分析中所面对的挑战.pdf

并行处理在大数据分析中所面对的挑战 张晓东 美国俄亥俄州立大学 合作单位 美国俄亥俄州立大学计算机科学与工程系 Facebook 数据管理系统团队 美国埃默里大学医学院 （Emory University ） 中科院计算所 Broad Challenges of Big Data Analytics  Existing DB technology is not prepared for the huge volume • Needs new system infrastructure for structured and non-structured data • Must be hardware independent, HP, HT, scalable, and fault-tolerant Truly multidisciplinary collaborations across all the disciplines • Although data volumes in all fields are high, the formats of data and natures of analytics are very different.  Big data analytics must be cost effective to the society • Conventional database model is not affordable • Low cost clusters and open source software are foundations  Open many new research challenges • In algorithms, systems design/implementations, and in many applications 2 Computing Paradigm Shift for Big Data Analytics  Conventional parallel processing model is “scale-up” based • BSP model, CACM, 1990: optimizations in both hardware and software • Hardware: low ratio of comp/comm, fast locks, large cache and memory • Software: overlapping comp/comm, exploiting locality, co-scheduling …  Big data processing model is “scale-out” based • DOT model, SOCC’11: hardware independent software design • Scalability: maintain a sustained throughput growth by continuously adding low cost computing and storage nodes in distributed systems • Constraints in computing patterns: communication- and data-sharing-free 3 BSP is a Scale-Up Model for HPC