第二讲提高数据处理能力意识总动员.ppt

提高现代数据处理意识与处理能力总动员 大数据时代谈数据处理重要性 源自维基百科的大数据（big data）介绍（1） Big data[1][2] 是一系列的数据集的集合其之大之复杂导致传统数据处理方法和现场的数据管理工来处理变得很具有挑战性。 有以下三方面的挑战： capture, curation, storage,[3] search, sharing, transfer, analysis,[4] visualization. 研究大数据的动机在于要获得比分析单个数据集多得多的额外信息。 与分散研究相比，同样的数据协同研究会得到潜在的相关性，有可能发现 商业网点的倾向。从而提高研究的品质。 prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions.“[5][6][7] 都是大数据时代的新问题。 源自维基百科的大数据（big data）介绍（2） 到了 2012, 对于数据规模的限制（limits on the size of data sets ）的要求已经提上了议事日程。[8][9] 如下领域：气象学、染色体、通讯、复杂物理模拟，生物和环境工程（meteorology, genomics,[10] connectomics, complex physics simulations,[11] and biological and environmental research.[12]）互联网、金融、商业（Internet search, finance and business informatics）都不得不对数据规模限制。 数据大规模增长主要因为无处不在的信息传感器空中遥感技术（ubiquitous information-sensing mobile devices, aerial sensory technologies), 还有software logs, cameras, microphones, radio-frequency identification readers, and wireless sensor networks.[13][14] 从 1980s [15] 起，平均每40个月涨一倍，而到了 2012, every day 2.5 quintillion (2.5×1018) bytes of data were created.[16] The challenge for large enterprises is determining who should own big data initiatives that straddle the entire organization.[17] 大数据（big data）介绍（3） Big data is difficult to work with using most relational database management systems and desktop statistics and visualization packages, requiring instead massively parallel software running on tens, hundreds, or even thousands of servers.[18] What is considered big data varies depending on the capabilities of the organization managing the set, and on the capabilities of the applications that are traditionally used to process and analyze the data set in its domain. For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration.[19] 大数据的定义 Big data usually includes