advanced software engineering project 1 mapreduce join (2人).ppt

Advanced Software Engineering 1. MapReduce Join (2人) Focused on performance analysis on different implementation of join processors in MapReduce. Homogenization: add additional information about the source of the data in the map phase, then do the JOIN in the reduce phase. Map-Reduce-Merge: a new primitive called merge is added to process the join separately. Other implementation: the map-reduce execution plan for joins generated by Hive. 进行性能分析与比较（生成至少10张以上图表） 2. 大型社交网络的结构分析(3-4人) 学习分类、聚类算法 使用Google+和Twitter社交圈数据 /data/egonets-Gplus.html /data/egonets-Twitter.html 在M/R或Spark上搭建分布式计算系统 通过Mahout/Mllib等开源工具进行数据分析、发现两种社交网络的“特性” 进行性能分析与比较（生成至少10张以上图表） Bonus：比较M/R和Spark的性能 Never use off-the-self softwares!!! 3. 分布式排序学习系统的搭建(3-4人) 学习Pointwise, Pairewise, Listwise三大类算法 使用Microsoft Learning-to-Rank Datasets /en-us/projects/mslr/ 在M/R、Storm、Spark其中一种架构上搭建分布式计算系统 至少实现上述三大类算法中的三种算法 进行性能分析与比较（生成至少10张以上图表） Bonus：比较M/R和Spark的性能 Mechanism Working in group: 2, OR, 3-4 students, clear roles Email me (ase_bit@) by this Friday (Dec 19) Team leader, Team members Topic Deadline: 16 Jan 2015! Deliverable: project report in Chinese Introduction (motivation, WHY?) Your proposal (HOW?) Performance Evaluation Conclusion Presentation Suggested Arrangement Week-1: Define your roles and start literature research Week-2 and 3: Propose solutions Week-4 and 5: Implementation and obtain results Finally, spend a few days writing your report 希望大家关注的问题 这不仅仅是个工程项目 通过以此来训练研究性思维 别人做过些什么？有什么问题？ 在哪里改进？性能如何？ 性能： 自身性能：正确率，吞吐率，并发率，时延 比较性能：其他算法，其他系统 多用开源框架 打分充分考虑团队整体贡献和每个队员的贡献 IEEE Xplore: / Social Network Analysis Key Players How to identify key/central nodes in network Cohesion How to characterize a network’s structure Example Facebook: 5.8million users (2009), avr 5.73 degrees, max 12 degrees Twitter: 5.2 billion relationships, avr 4.67 degrees 50% users only 4 step away Almost everyone 5 steps For any 1,500 random users, 3.435 steps Erdos Number: Collaborative distance through paper co-authori