《如何使用开源软件为》课件.ppt

Spring Hadoop – Batch & Integration * Reuse same Batch infrastructure and knowledge to manage Hadoop workflows Step can be any Hadoop job type or HDFS script Hadoop Workflows managed by Spring Batch * Spring Batch for File/DB/NoSQL driven applications Collect: Process local files Transform: Scripting or Java code to transform and enrich RT Analysis: N/A Ingest: (batch/aggregate) write to HDFS or split/filtering Batch Analysis: Orchestrate Hadoop steps in a workflow Distribute: Copy data out of HDFS to structured storage JMX enabled along with REST interface for job control Capabilities: Spring + Hadoop + Batch * Collect Transform RT Analysis Ingest Batch Analysis Distribute Use Spring Batch Configuration for Hadoop * <job id="job1"> <step id="import" next="wordcount"> <tasklet ref=“import-tasklet"/> </step> <step id=“wc" next="pig"> <tasklet ref="wordcount-tasklet"/> </step> <step id="pig"> <tasklet ref="pig-tasklet“></step> <split id="parallel" next="hdfs"> <flow><step id="mrStep"> <tasklet ref="mr-tasklet"/> </step></flow> <flow><step id="hive"> <tasklet ref="hive-tasklet"/> </step></flow> </split> <step id="hdfs"> <tasklet ref="hdfs-tasklet"/></step> </job> Reuse previous Hadoop job definitions Spring Batch Configuration for Hadoop * Spring Integration for Event driven applications Collect: Single node or distributed data collection (tcp/JMS/Rabbit) Transform: Scripting or Java code to transform and enrich RT Analysis: Connectivity to multiple analysis techniques Ingest: Write to HDFS, Split/Filter data stream to other stores JMX enabled + control bus for starting/stopping individual components Capabilities: Spring + Hadoop + SI * Collect Transform RT Analysis Ingest Batch Analysis Distribute Use Poll a local directory for files, files are rolled over every 10 min Copy files to staging area and then to HDFS Use an aggregator to wait to “process all files available every hour” to launch