基于Hadoop原生HDFS的大规模并行SQL.pptx

HAWQMPP SQL for HDFS of Hadoop基于Hadoop原生HDFS的大规模并行SQLHAWQ Is The… Enterprise platform that provides the fewest barriers, lowest risk, most costeffective and fastest way to enter in to big data analytics on HDFS of HadoopHAWQ 简述CPUMemSQL Engine Cost-Based Query OptimizationANSI SQL 2003/2011 SupportMulti-User PlatformDiskResource QueuesRobust Query OptimizerUsers Concurrency DataEncryption Role-Based SecurityAccessibility ODBC/JDBC Driver L3,4 Complex DataManagement Sub-Partitioning Distributions PartitioningStorage Options Polymorphic StorageGreenplum databasere-platformed on Hadoop/HDFS txt Avro SeqHBase HiveParallel Loading/Unloading HDFS Native Formats Extendable…Row/ColumnarStorage Built-in Compression HDFS Native FormatsMapReduce IntegrationHAWQ的优点…? 支持Apache Hadoop原生HDFS的SQL大规模并行引擎（MPP SQL）? GPFX External Tables 接口，使用SQL透明访问Hadoop上各类数据 – HDFS, HBase, Hive，Parquet格式等等? 还支持SQL透明访问NFS，HTTP其他格式的数据（可自定义）? Performance and Scalability – Parallel Everything – Dynamic Pipelining – High Speed Interconnect（基于UDP） – HDFS access with C++ libhdfs3– Co-Located Joins Data Locality– Partition Elimination（支持静态动 态表分区）– Higher Cluster Utilization– Concurrency Control（资源作业优 先级调度）HAWQ 及Hadoop软件栈 HAWQ– MPP SQLANSI SQL + Analytics XtensionFramework QueryOptimizer Resource Management Workflow YarnZookeeper Catalog ServicesDynamic Pipelining HDFSData LoaderFlumeApacheHAWQ Added ValueHAWQ 与 Hadoop HDFSHAWQMasterMaster…SegmentSegmentSegmentSegmentSegment…DataDataDataDataDataNameNameHAWQ 与 Hadoop HDFS数据访问流 Master host HAWQ Interconnect Segment hostSegment SegmentSegment host SegmentSegment Segment SegmentSegment host Segment SegmentSegment hostSegmentSegmentSegmentSegmentRead/WriteBreplicationDatanodeDatanodeDatanodeDatanodeRack1Rack2Meta OpsNamenodeHDFSHAWQ 对比 Greenplum DB 基本架构SQL 大规模并行处理SQL MPP (Massively Parallel Processing)无共享架构 Shared-Nothing Architecture MasterSQLMapReduce...... 节点 生成查询计划并派发 汇总执行结果 NetworkInterconnect Segment 节点执行查询计划及数据 存储管理 外部数据源 并行装载或导出... ... 数据库存储层Sharding+Re