云端的小飞象系列报告之二课件.pptVIP

Nova: Continuous Pig/Hadoop Work?ows By Yahoo！ 精品文档 Nova Overview Scenarios Ingesting and analyzing user behavior logs Building and updating a search index from a stream of crawled web pages Processing semi-structured data Two-layer programming model (Nova over Pig) Continuous processing Independent scheduling Cross-module optimization Manageability features 精品文档 Workflow Model Workflow Two kinds of vertices: tasks (processing steps) and channels (data containers) Edges connect tasks to channels and channels to tasks Four common patterns of processing Non-incremental (template detection) Stateless incremental (shingling) Stateless incremental with lookup table (template tagging) Stateful incremental (de-duping) 精品文档 Workflow Model (Cont.) Data and Update Model Blocks: A channel’s data is divided into blocks Contains a complete snapshot of data on a channel as of some point in time Base blocks are assigned increasing sequence numbers(B0,B1,B2……Bn) Base block Used in conjunction with incremental processing Contains instructions for transforming a base block into a new base block( ) Delta block 精品文档 Workflow Model (Cont.) Task/Data Interface Consumption mode: all or new Production mode: B or Δ 精品文档 Workflow Model (Cont.) Workflow Programming and Scheduling Data-based trigger. Time-based trigger Cascade trigger. Data Compaction and Garbage Collection If a channel has blocks B0， ， , ，the compaction operation computes and adds B3 to the channel After compaction is used to add B3 to the channel，and current cursor is at sequence number 2， then B0， ， can be garbage-collected. 精品文档 Nova System Architecture 精品文档 Apache Hadoop Goes Realtime at Facebook By Facebook 精品文档 Workload Types Facebook Messaging High Write Throughput Large Tables Data Migration Facebook Insights Realtime Analytics High Throughput Increments Facebook Metrics System (ODS) Automatic