Apache Flink与Apache Hive的集成介绍.pdf

Integrating Apache Flink with Apache Hive 01 Design Architecture 02 Contents Project Status 03 Performance Benchmark 04 Demo Design Architecture 01 Background Flink Enable Flink to run batch jobs and enrich the Flink ecosystem SQL SQL is the most prevalent tool for big data analysis FlinkSQL: FlinkSQL is incomplete and misses important features: Doesn’t have a metadata management mechanism DDL Lack of DDL support Unable to integrate with external systems conveniently Goals Catalog Unified Catalog API Catalog Provide both in-memory and persistent catalog implementations Hive: Interoperability with Hive: Hive Access to Hive metadata Hive Read/write Hive tables FlinkHive Add Flink as a Hive execution engine (long -term plan)