vSphere Big Data Extensions之Hadoop参考架构和性能最佳实践.pptVIP

Virtualization Host VMDK OS Image – VMDK Hadoop Virtual Node 1 Datanode Ext4 Task- tracker Ext4 Ext4 Ext4 Local disks OS Image – VMDK VMDK VMDK VMDK VMDK VMDK VMDK VMDK Hadoop Virtual Node 2 Datanode Ext4 Task- tracker Ext4 Ext4 Ext4 mapred.local.dir Standard Deployment Configuration Virtualization Host OS Image – VMDK Hadoop Virtual Node 1 Task- tracker Shared storage SAN/NAS Local disks OS Image – VMDK VMDK VMDK VMDK VMDK VMDK VMDK VMDK Hadoop Virtual Node 2 Datanode Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 VMDK VMDK VMDK VMDK VMDK VMDK VMDK VMDK VMDK … … Standard Deployment Configuration for D/C Separation Data Path for Combined vs. Data/Compute Separation Virtualization Host Virtualization Host Hadoop Virtual Node 1 Hadoop Virtual Node 2 TaskTracker Virtual Switch Hadoop Virtual Node Virtual Switch TaskTracker Serengeti provide local storage based temp for D/C separation. Each compute VM needs its own temp space Required temp space is different from an application to another Can result in wasted space Recommended Topology of Data/Compute Separation Virtualization Host VMDK OS Image – VMDK Hadoop Virtual Node 1 Ext4 Task- tracker Shared storage SAN/NAS Local disks OS Image – VMDK VMDK VMDK VMDK VMDK VMDK VMDK VMDK Hadoop Virtual Node 2 Datanode VMDK Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 Ext4 … Virtualization Host Hadoop Virtual Node 1 Hadoop Virtual Node 2 TaskTracker Virtual Switch Virtualization Host Hadoop Virtual Node 1 Hadoop Virtual Node 2 TaskTracker Virtual Switch Data Path for Local TT Storage vs. NFS Temp Serengeti provide NFS based temp for D/C separation Improve local storage space utilization. Trade-off between bandwidth efficiency vs. overhead of NFS. Consolidated Storage on Single DN VM Virtualization Host OS Image – VMDK Hadoop Virtual Node 1 Task- tracker Shared storage SAN/NAS Local disks OS Image – VMDK VMDK VMDK VMDK VMDK VMDK VMDK VMDK Hadoop Virtual Node 2 Datanode Ext4 Ext4