R and H adoop I ntegrated P rocessing E nvironment课件.ppt

R and Hadoop Integrated Processing Environment;R and Large Data;R and HPC;Computing With Data;Using Hadoop DFS to Store;Namenode;Mapreduce;Mapreduce in R;File;R and Hadoop ;Everything is a key-value pair Keys need not be unique ;Airline Data;Save Airline as R Data Frames;Save Airline as R Data Frames;Save Airline as R Data Frames;Save Airline as R Data Frames;Quantile Plot of Delay;Quantile Plot of Delay;Quantile Plot of Delay;坍声机氧秩撵檄蛾缨槛袒芋骤劲反枷饮鸡细阑噶迫禹轻糖渍湍噬虹紫穷嚏R and H adoop I ntegrated P rocessing E nvironment课件R and H adoop I ntegrated P rocessing E nvironment课件;Conditioning;Conditioning;Running a FF Design;Running an FF Design;Chunk == 1, send data to reducers m2 - expression({ lapply(seq_along(map.keys),function(r){ key - map.keys[[r]] value - map.values[[r]] apply(para3.r,1,function(j){ rhcollect(list(k=key,p=j), value) }) }) }) map.values is a list of connection data map.keys are connection identifiers para3.r is list of 243 parameter sets;Reduce: apply algorithm r2 - expression( reduce={ value - reduce.values[[1]]; params - as.list(reduce.key$p) tt=system.time(v - ks.detect(value,debug=F,params=params ,dorules=FALSE)) rhcounter(param,_all_,1) rhcollect(unlist(params) ,list(hash=reduce.key$k,numks=v$numks, time=tt)) }) rhcounter updates “counters” visible on Jobtracker website and returned to R as a list;FF Design … cont’d;FF Design … cont’d;Large Data;Large Data ;Large Data;Large Data;Hadoop as Key-Value DB;Hadoop as a Key-Value DB;Hadoop as a Key-Value DB;EC2;EC2 - Example;EC2 - Example;Log of ‘Time to complete’ vs. log of ‘Number of computers’ , the solid line is the least square fit to the data. The linear fit is what we expect in an ideal non preemptive world with constant time per task.;Todo