数据分析之道概要1.pptx

大数据分析师的卓越之道 2 Infrastructure Data Value Knowledge Discovery 数据分析的典型场景 Data Value 改变思维方式 提高技术素养 丰富分析能力 3 基础设施已经改朝换代分析师也需要与时俱进 新的世界观：不确定的世界 4 大数据的测不准 5 《科学》：大数据傲慢 《自然》：测不准 6 Collection Analytics Interpretation Evaluation Preparation Hypotheses 数据分析方法论的升级 7 Collection Analytics Interpretation Evaluation Preparation Hypotheses 数据分析方法论的升级 8 Hypotheses 机械地发掘相关性和假设 直觉，拿侦探小说练手 阅读广泛涉猎 跨界思维碰撞 融入业务部门 防止数据采集与分析、业务与数据分析的脱节 9 Collection Analytics Interpretation Evaluation Preparation Hypotheses 数据分析方法论的升级 10 数据！数据！数据！ n=All ! Enterprise Data Warehouse  Enterprise Data Hub/Data Lake External data sources Structured  semi-structured  unstructured Log analysis Text analysis Image/video Data with geo and temporal tags Networks and graphs 11 数据？数据？数据？ n=All ? More data vs. sampling “Raw data” is an oxymoron Signals and noises Sampling bias Data exchange and sharing Data rights, data pricing Data lifecycle management Provenance capture, representation, and querying Sometimes data are not assets, but costs 12 Collection Analytics Interpretation Evaluation Preparation Hypotheses 数据分析方法论的升级 13 数据质量：重中之重 Noisy, biased and polluted data are unavoidable Goal: models = components for noise + relatively complex models for signal Cleansing, validation, … Can it start with a small subset? Can the process be automated? Work together with visualization, machine learning Curation, Wrangling, … Automated learning to discover structure, resolve entities, and transform data 14 数据表示 Reduce compute and communication complexity Sparse, compressed data structure Approximate computation Reduce statistical complexity Dimensionality reduction, clustering Sampling Non-random sampling, compressive sensing, … … Choose best representation for specific computational methods E.g. tables for data parallelism, networks/graphs for graph parallelism UIMA: Unstructured Information Management Architecture 15 Collection Analytics Interpretation Evaluation Preparation Hypotheses 数据分析方法论的升级 16 Source: Computational Sc