hadoop多数据中心灾备.pdfVIP

Disaster Recovery Solution for Apache Hadoop Contributors: Chen Haifeng (haifeng.chen@), Gangumalla Uma (uma.gangumalla@), Dey Avik (avik.dey@), Li Tianyou (tianyou.li@), Purtell, Andrew (andrew.k.purtell@) Contents Introduction 2 Targets 2 Approach 2 Design 5 Concept of Region 5 Cluster Joining 5 Synchronous Writing 5 Synchronous Data Writing 6 Synchronous Namespace Journaling 7 Handling Journaling Failures 8 Asynchronous Replication8 Asynchronous Data Replication 9 Asynchronous Namespace Journaling 10 Mirror Cluster Unavailable 11 Failover 11 Data Block Replication 13 Performance Considerations 13 1 Introduction Apache Hadoop is architected to operate efficiently at scale for normal hardware failures within a datacenter. It is not designed today to handle datacenter failures. Although HDFS is not designed for nor deployed in configurations spanning multiple datacenters, replicating data from one location to another is common practice for disaster recovery and global service availability. There are current solutions available for batch replication using data copy/export tools. However, while providing some backup capability for HDFS data, they do not provide the capability to recover all your HDFS data from a datacenter failure and be up and running again with a fully operational Hadoop cluster in another datacenter in a matter of minutes. For disaster recovery from a datacenter failure, we should provide a fully distributed, zero data loss, low latency, high throughput and secure HDFS data replication solution for multiple datacenter setup. Targets The following list the targets of this design: 1. Support both synchronous writ