2015 Container技术峰会-Docker与Mesos的结合应用.docVIP

2015 Container技术峰会-Docker与Mesos的结合应用 王璞@数人科技 2015/04/18 Why Distributed System? ?? Cost Distributed System Design Principles ?? Scalability ?? more mediocre machines than a few powerful ones ?? No SINGLE POINT FAILURE (fault tolerance) ?? Service duplication / multiple instances ?? Data replica ?? High Availability ?? Service requests: Load balancing / Service discovery ?? Data requests: Sharding / Consistent hashing ?? Data locality (less communication) Distributed Infrastructure @ DataMan ?? Automatically manage computing resources ?? scale out to hundreds, thousands, millions of servers ?? pseudo single server from the developer’s point of 6#00aa00view ?? Functionally similar to an OS ?? Far from mature ?? centralized OS is way more mature than distributed OS ?? not for end users, but for developers ?? just a running environment Linux OS v.s. Distributed “OS” ?? 3 Masters, synchronized by Zookeeper ?? ~100 Slaves, each 2 core 4G ?? Frameworks: Marathon, Chronos, Spark, Storm Mesos Mesos: Fine-Grained Sharing Framework 1 Framework 2 Framework 3 Coarse-Grained Sharing (HPC): Fine-Grained Sharing (Mesos): + Improved utilization, responsiveness, data locality Storage System (e.g. HDFS) Storage System (e.g. HDFS) Fw. 1 Fw. 1Fw. 3 Fw. 3 Fw. 2Fw. 2 Fw. 2 Fw. 1 Fw. 3 Fw. 2Fw. 3 Fw. 1 Fw. 1 Fw. 2Fw. 2 Fw. 1 Fw. 3 Fw. 3 Fw. 3 Fw. 2 Fw. 2 ?? Offer available resources to frameworks, let them pick which resources to use and which tasks to launch + Keeps Mesos simple, lets it support future frameworks - Decentralized decisions might not be optimal Mesos: Resource Offers Spark job Spark scheduler Hadoop job Hadoop scheduler Allocation module Mesos master Mesos slave Spark executor Mesos slave Spark executor tasktask Resource offer Pick framework to offer resources to Mesos Architecture Mesos Architecture Spark job Spark scheduler Hadoop job Hadoop scheduler Allocation module Mesos master Mesos slave Spark executor Mesos slave Spark executor tasktask Pick f