综合论文：DiscretizedStreams.FaultTolerantStreamingComputationatScale（英文）.pdfVIP

Discretized Streams: Fault-Tolerant Streaming Computation at Scale Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, Ion Stoica University of California, Berkeley Abstract faults and stragglers (slow nodes). Both problems are in- Many “big data” applications must act on data in real evitable in large clusters [12], so streaming applications time. Running these applications at ever-larger scales re- must recover from them quickly. Fast recovery is even quires parallel platforms that automatically handle faults more important in streaming than it was in batch jobs: and stragglers. Unfortunately, current distributed stream while a 30 second delay to recover from a fault or strag- processing models provide fault recovery in an expen- gler is a nuisance in a batch setting, it can mean losing sive manner, requiring hot replication or long recovery the chance to make a key decision in a streaming setting. times, and do not handle stragglers. We propose a new Unfortunately, existing streaming systems have processing model, discretized streams (D-Streams), that limited fault and straggler tolerance. Most dis- overcomes these challenges. D-Streams enable a par- tributed streaming systems, including Storm [37], allel recovery mechanism that improves efficiency over TimeStream [33], MapReduce Online [11], and stream- traditional replication and backup schemes, and tolerates ing databases [5, 9, 10], are based on a continuous op- stragglers. We show that they support a rich set of oper- erator model, in which long-running, stateful operators ators