Hadoop中MapReduce例子.docxVIP

在下面的例子中我们分析Apache的log并把这些log进行分析并把分析完的结果按用户IP为ROW，把log中用户的访问时间，请求方法，用户请求的协议，用户的浏览器，服务状态等写到HBase的表中。首先我们要在HBase中建立我们的一个表来存储数据。 [ol] ? public static void creatTable(String table) throws IOException{?HConnection conn = HConnectionManager.getConnection(conf);?HBaseAdmin admin = new HBaseAdmin(conf);? if(!admin.tableExists(new Text(table))){?System.out.println(1. + table + table creating ... please wait);?HTableDescriptortableDesc = new HTableDescriptor(table);?tableDesc.addFamily(new HColumnDescriptor(http:));?tableDesc.addFamily(new HColumnDescriptor(url:));?tableDesc.addFamily(new HColumnDescriptor(referrer:));?admin.createTable(tableDesc);? } else {?System.out.println(1. + table + table already exists.);? }?System.out.println(2. access_log files fetching using map/reduce);? }[/ol]复制代码然后我们运行一个MapReduce任务来取得log中的每一行数据。因为我们只要取得数据而不需要对结果进行规约，我们只要编写一个Map程序即可。[ol] ? public static class MapClass extends MapReduceBase implements? Mapper {??@Override? public void configure(JobConf job) {?tableName = job.get(TABLE, );? }??public void map(WritableComparable key, Text value,?OutputCollector output, Reporter reporter)? throws IOException {? try {?AccessLogParser log = new AccessLogParser(value.toString());? if(table==null)? table = new HTable(conf, new Text(tableName));? long lockId = table.startUpdate(new Text(log.getIp()));?table.put(lockId, new Text(http:protocol), log.getProtocol().getBytes());?table.put(lockId, new Text(http:method), log.getMethod().getBytes());?table.put(lockId, new Text(http:code), log.getCode().getBytes());?table.put(lockId, new Text(http:bytesize), log.getByteSize().getBytes());?table.put(lockId, new Text(http:agent), log.getAgent().getBytes());?table.put(lockId, new Text(url: + log.getUrl()), log.getReferrer().getBytes());?table.put(lockId, new Text(referrer: + log.getReferrer()), log.getUrl().getBytes());??mit(lockId, log.getTimestamp());? } catch (ParseException e) {?e.printStackTrace();? } catch (Exception e) {?e.printStackTrace()