系统管理与监控.pptx

大数据系统的部署、调度与监控;本次课的目标;怎么听本节课;一个系统管理员（我）的血泪;我维护的200节点集群;Everything managed by …;HPC: The good old days (for sys admins);John Boyle. Biology must develop its own big-data systems. Nature (world view). July 2013 ;Motivation 2: Customers demand performance …;We have a variety of applications;Social “Big Data”;Lots of dependencies …;Resource hungry too …;Customer’s needs change …;Customer’s needs change …;Customer’s needs change …;Customer’s needs change …;系统的部署：从裸机到大数据系统;;基本想法：安装一台机器，自动安装所有其他机器;网络和硬件的配置;解决方案：定制化服务器+整机架交付;* /content/www/us/en/architecture-and-technology/intel-rack-scale-architecture.html;集装箱规模的交付和部署;硬件支持;操作系统和基础架构;解决方案：配置管理;流行的配置管理工具;配置管理：可视化;项目要求和截止期;课程项目comments;课程项目特别提醒;Hadoop作业;系统全局状态的维护与管理;系统的全局状态：挑战;Chubby的解决方案：;Primary Election;A model: two general problem;Fischer-Lynch-Paterson (FLP);Paxos Introduction;Paxos: the name;Paxos Consensus Model;Leslie Lamport;A Paxos Round;Replicated State Machine;Using Paxos;Using Paxos to replicate states;;Example 2: Read;;Chubby Design: System Structure;Design: Files, Dirs, Handles;Lock Leases;ZooKeeper Service ;Example use of Zookeeper;任务调度：问题和挑战;Problem: Resource Sharing in Data Centers;Solution: Resource Scheduler;What are the “demands”?;What are the “resources”?;Goals for the scheduler (1);Goals for the scheduler (2);Goals for the scheduler (3);Other goals;任务调度举例：Borg;Borg;Resources managed by Borg;Demands;Borg Architecture;How Borg achieved the goals;Borg Scheduling Policy;Naming;Other things Borg handles;Lessons;Open source: kubernetes;任务调度：Mesos;Mesos Demo;Mesos Architecture;Resource Offering;任务调度：sparrow;问题：scheduler 太慢怎么办？;10 min.;多个scheduler的问题？;Per-task sampling;Per-task sampling;Per-task Sampling;Simulated Results;调度器状态同步结论;系统的监控;系统监控的目标;最普遍的监控工具：日志;Splunk与日志的处理;Monitoring as a Service;其他监控手段;Tracing Technology：Dapper;基于大数据的系统的自动??维;本节课总结