Ryan Weald：Spark流媒体产品化.pdf

Productionalizing Spark Streaming Spark Summit 2013 Ryan Weald @rweald @rweald What We’re Going to Cover •What we do and Why we choose Spark •Fault tolerance for long lived streaming jobs •Common patterns and functional abstractions •Testing before we “do it live” @rweald Special focus on common patterns and their solutions @rweald What is Sharethrough? Advertising for the Modern Internet Form Function @rweald What is Sharethrough? @rweald Why Spark Streaming? @rweald Why Spark Streaming •Liked theoretical foundation of mini-batch •Scala codebase + functional API •Young project with opportunities to contribute •Batch model for iterative ML algorithms @rweald Great... Now productionalize it @rweald Fault Tolerance @rweald Keys to Fault Tolerance 1.Receiver fault tolerance 2.Monitoring job progress @rweald Receiver Fault Tolerance •Use Actors with supervisors •Use self healing connection pools @rweald Use Actors class RabbitMQStreamReceiver (uri :String, exchangeName: String, routingKey: String) extends Actor with Receiver with Logging { implicit val system = ActorSystem() override def preStart() = { //Your code to setup connections and actors //Include inner class to process messages } def receive: Receive = { case _ = logInfo(unknown message) } } @rweald Track All Outputs •Low watermarks - Google MillWheel •Database updated_at •Expected output file size alerting @rweald Common Patterns