火花流量flume-ng 8公园溪流水流spark stream整合.pdfVIP

【

sparkstream特点】

SparkStreaming流式处理系统特点有：

将流式计算分解成一系列短小的批处理作业

将失败或者执行较慢的任务在其它节点上并行执行

较强的容错能力(基于RDD继承关系Lineage)

使用和RDD一样的语义

【配置】

建立maven工程，写好pom.xml

需要sparkstreaming的flume插件包，jar的maven地址如下，填入pom.xml中

dependency

groupIdorg.apache.spark/groupId

artifactIdspark-streaming-flume_2.10/artifactId

version1.1.0/version

/dependency

【实例】

/**

*Createdbymcon2015/6/12.

*/

importorg.apache.spark.SparkConf;

importorg.apache.spark.api.java.function.Function;

importorg.apache.spark.streaming.*;

importorg.apache.spark.streaming.api.java.*;

importorg.apache.spark.streaming.flume.FlumeUtils;

importorg.apache.spark.streaming.flume.SparkFlumeEvent;

publicfinalclassJavaFlumeEventCount{

privateJavaFlumeEventCount(){

}

publicstaticvoidmain(String[]args){

Stringhost=args[0];

intport=Integer.parseInt(args[1]);

DurationbatchInterval=newDuration(Integer.parseInt(args[2]));

SparkConfsparkConf=newSparkConf().setAppName(JavaFlumeEventCount);

JavaStreamingContextssc=newJavaStreamingContext(sparkConf,

batchInterval);

JavaReceiverInputDStreamSparkFlumeEventflumeStream=FlumeUtils

.createStream(ssc,host,port);

flumeStream.count();

flumeStream.count().map(newFunctionLong,String(){

privatestaticfinallongserialVersionUID=-L;

publicStringcall(Longin){

returnReceived+in+flumeevents.;

}