GFS介绍.pptVIP

GFS: The Google File System Brad Karp UCL Computer Science Motivating Application: Google Crawl the whole web Store it all on “one big disk” Process users’ searches on “one big CPU” More storage, CPU required than one PC can offer Custom parallel supercomputer: expensive (so much so not really available today) Cluster of PCs as Supercomputer Lots of cheap PCs, each with disk and CPU High aggregate storage capacity Spread search processing across many CPUs How to share data among PCs? Ivy: shared virtual memory Fine-grained, relatively strong consistency at load/store level Fault tolerance? NFS: share fs from one server, many clients Goal: mimic original UNIX local fs semantics Compromise: close-to-open consistency (performance) Fault tolerance? Google Platform Characteristics 100s to 1000s of PCs in cluster Cheap, commodity parts in PCs Many modes of failure for each PC: App bugs, OS bugs Human error Disk failure, memory failure, net failure, power supply failure Connector failure Monitoring, fault tolerance, auto-recovery essential Google File System: Design Criteria Detect, tolerate, recover from failures automatically Large files, = 100 MB in size Large, streaming reads (= 1 MB in size) Read once Large, sequential writes that append Write once Concurrent appends by multiple clients (e.g., producer-consumer queues) Want atomicity for appends without synchronization overhead among clients GFS: Architecture One master server (state replicated on backups) Many chunk servers (100s – 1000s) Spread across racks; intra-rack b/w greater than inter-rack Chunk: 64 MB portion of file, identified by 64-bit, globally unique ID Many clients accessing same and different files stored on same cluster GFS: Architecture (2) Master Server Holds all metadata: Namespace (directory hierarchy) Access control information (per-file) Mapping from files to chunks Current locations of chunks (chunkservers) Manages chunk leases to chunkservers Garbage collects orphaned chunks Migrates chunks b