如何使用开源软件为 Hadoop 构建大数据管道.pptVIP

Spring Hadoop – Batch Integration * Reuse same Batch infrastructure and knowledge to manage Hadoop workflows Step can be any Hadoop job type or HDFS script Hadoop Workflows managed by Spring Batch * Spring Batch for File/DB/NoSQL driven applications Collect: Process local files Transform: Scripting or Java code to transform and enrich RT Analysis: N/A Ingest: (batch/aggregate) write to HDFS or split/filtering Batch Analysis: Orchestrate Hadoop steps in a workflow Distribute: Copy data out of HDFS to structured storage JMX enabled along with REST interface for job control Capabilities: Spring + Hadoop + Batch * Collect Transform RT Analysis Ingest Batch Analysis Distribute Use Spring Batch Configuration for Hadoop * job id=job1 step id=import next=wordcount tasklet ref=“import-tasklet/ /step step id=“wc next=pig tasklet ref=wordcount-tasklet/ /step step id=pig tasklet ref=pig-tasklet“/step split id=parallel next=hdfs flowstep id=mrStep tasklet ref=mr-tasklet/ /step/flow flowstep id=hive tasklet ref=hive-tasklet/ /step/flow /split step id=hdfs tasklet ref=hdfs-tasklet//step /job Reuse previous Hadoop job definitions Spring Batch Configuration for Hadoop * Spring Integration for Event driven applications Collect: Single node or distributed data collection (tcp/JMS/Rabbit) Transform: Scripting or Java code to transform and enrich RT Analysis: Connectivity to multiple analysis techniques Ingest: Write to HDFS, Split/Filter data stream to other stores JMX enabled + control bus for starting/stopping individual components Capabilities: Spring + Hadoop + SI * Collect Transform RT Analysis Ingest Batch Analysis Distribute Use Poll a local directory for files, files are rolled over every 10 min Copy files to staging area and then to HDFS Use an aggregator to wait to “process all files available every hour” to launch MR job Ingesting Copying Local Log Files into HDFS * Use syslog adapter Transforme