基于RocketMQ Connect构建数据流转处理平台.pdf

基于RocketMQ Connect构建数据流转处理平台 技术创新，变革未来 目录 • OpenMessaging Connect Overview • OpenMessaging Connect Architecture • OpenMessaging Connect in practice • OpenMessaging Connect Now and Future Concept RocketMQ Connect ：A simple understanding of RocketMQ Connect is the middleware that uses RocketMQ to copy data from a source data storage system to a target data store. Mainly composed of source connector, sink connector, runtime. Source Connector：Responsible for ingesting the data from source data and sending it to RocketMQ Sink Connector ：Responsible for consuming messages from RocketMQ and writing the data to target storage Runtime： The runtime is the source, sink connector runtime environment, responsible for loading the Connector, providing the RESTful interface, starting the connector task, service discovery between cluster nodes, configuration synchronization, consumption schedule saving, failover, load balancing, and so on. File Connect Demo RocketMQ Connect Overview Traditional stream data process model OLAP Producer Consumer Service Service Producer Consumer Service Service Sourcing Data Sink Data OpenMessaging Connect Overview Challenge • How to copies data between so many different sources • Efficiency • High Availability, failover • Scalability OpenMessaging Connect Overview OLAP OpenMessaging Connect Overview OLAP OpenMessaging Connect Overview Benefits of Messaging Connect • Focus on dat