培乐园-海量数据之架构和处理5.pdfVIP

2. Infrastructure: MapReduce 2. Infrastructure: HBASE 2. Infrastucture: HBASE 2. Infrastructure: design Scalability, Reliability, Performance, Throughput, Latency • Scalability, Reliability, Performance, Throughput, Latency SSccaallaabbiilliittyy,, RReelliiaabbiilliittyy,, PPeerrffoorrmmaannccee,, TThhrroouugghhppuutt,, LLaatteennccyy Design: • Design: DDeessiiggnn:: – Partitioning/Sharding – Consistent hash – Consistency model – Data Models – Storage layouts – Log Structured Merge Tree(bigtable) Notes: • Notes: NNootteess:: – Strict Consistency – Eventual Consistency – Timestamp and Vector Clocks – Gossip – Primary key - value/blob/structure/semi-strucure – Secondary Indexes – Tables/Namespaces – Multi-version Storage – Row-based, Column-based storage – BloomFilters 3. Big Data 3. Big Data: hypergrowth • Reuters-21578: about 10K docs (ModApte) Bekkerman et al, SIGIR 2001 Bekkerman et al, SIGIR 2001 BBeekkkkeerrmmaann eett aall,, SSIIGGIIRR • RCV1: about 807K docs Bekkerman Scholz, CIKM 2008 Bekkerman Scholz, CIKM 2008 BBeekkkkeerrmmaann SScchhoollzz,, CCIIKKMM • LinkedIn job title data: about 100M docs Bekkerman Gavish, KDD 2011 Bekkerman Gavish, KDD 2011 BBeekkkkeerrmmaann GGaavviisshh,, KKDDDD From KDD2011 3. Big Data: hypergrowth hours days months years 3. Big Data: hypergrowth • Bigness – Volume, Velocity, Size • Structure – Variety, Variability, Complexity 3. Big Data : Machine Learning • Thousand instances – Manually • Million instances – preprocessin