LinkedIn互联网可伸缩数据流处理的经验教训.pptx

LinkedIn互联网可伸缩数据流处理的经验教训?????IntroductionTypical Architecture for Data ProcessingData IngestionStream processingFutureAGENDA Processing latencyMilliseconds to minutesMuch later.0 secondsFEED RELEVANCE(E.G. NEWSFEED)CYBER-SECURITYINTERNET OF THINGS?????IntroductionTypical Architecture for Data ProcessingData IngestionStream processingFutureAGENDAClients(browser,devices,sensors….)ARCHITECTUREServicesTierIngestionacleKafkaBrooklinEspressoServing/OrPinotVoldemort/VeniceBulkKafkuploadaProcessingBatchProcessingRealTimeProcessing(MapReduce/Spark)HDFSGobblin(ApacheSamza)…..?????IntroductionTypical Architecture for Data ProcessingData IngestionStream processingFutureAGENDAAPACHEKAFKAAZUREEVENTHUBAWSKINESISGOOGLEPUB-SUBPartitionAwareYesYesYesNoAbilitytoReplayYesYesYesNoPortableacrossCloudsYesNoNoNoDATAINGESTIONOPTIONS? Running at Small Scale ( 1TB a day ) ? Pay as yougo Cloudmanagedservices (Kinesis/EventHub)arecheaper? Running at Medium Scale 100 Billion messages/day, 5000 partitions ? Pay as yougo Cloudmanagedservices(Kinesis/EventHub)are cheaper (incl.operationalcost)? Running at Large Scale 100 Billion messages/day, 5000 partitions ? Managing your own customApache Kafka Cluster is cheaperDATA INGESTION : APPROX. COST BASED COMPARISON– Ingested Data/Day ? ~ 1.2 TrillionMessages ? 350 TB ? Peak loadof 16 MillionMessages/second– Data consumed/day : 1.3 PB– ~1800+ Kafka Broker MachinesKAFKA @ LINKEDIN– Machines/disks will die almost daily ? Need auto-healing( Kafka Cruise Control)– Take good care of Zookeeper ? 5 node Zookeeper clusters on SSD.– Monitoring will always reveal problems. ? KafkaMonitor(link),Burrow /linkedin/Burrow(/linkedin/Burrowlink/linkedin/Burrow)– Dealing with Multi-tenancy ? Quotas/RateLimitingis criticalto avoidavailabilitydipsLESSONS FROM RUNNING KAFKA @ SCALE– Bigger machines can be cheaper ? But..Machinefailurestakevery longto recover– More Data Storage = Bigger Clusters ? Bettersupportfor regulardisks insteadof using RAID-10 ? OptimizedData Rete