南京大学-顾荣-海量数据处理案例.ppt

Abstract IR Architecture A Simple Inverted Index Demo the system Target of Web Retrieval Interface Make an Interface which accpet user’s query and response search results. 1.Restrict the query string; 2.Sort search result dynamically; 3.Response results page by page. Web Retrieval Interface Sort Pages Term 0 Term 2 Term 1 Here is a demo. Doc1 10 Doc3 90 Doc7 20 Text in here Query String Word Segement Doc2 20 Doc7 80 Doc5 15 Doc3 05 Doc2 40 Doc6 40 Merge Rank Again Doc7 100 Doc3 95 Doc6 40 Doc2 40 Doc5 15 Doc1 10 Merge the rank and Rank again~ 3.5 Optimization a)For each term only top 1500 DID are reserved at most. b)Use TreeMap to sort.. Reduce Sort Time Reduce I/O operations …… Cache Strategy Optimization measures in different areas. a)Response Page is created dynamically. b)Each time return 10 records. Put some hot Inverted Index file in the memory. Cache replacement LRU Contents Background Brief Intro to principle of Full Text Search Engine Implement of FTSE for BBS Lily Maybe GoogleBaidu has done these... Conclusion 4. Maybe GoogleBaidu has done... Parallelly Word Segement …… Users query Rank 3.A better rank strategy :To descirbe the relationship between a token and DID precisely 4.Record each users query string; a) feed back to Word Segement b) Provide remind function.(By input change event) 1. Search Stuff parallelly 2. An OutStanding Word Segement Algorithm ……. Contents Background Brief Intro to principle of Full Text Search Engine Implement of FTSE for BBS Lily Maybe GoogleBaidu has done these... Conclusion 5. Conclusion summary 5.1 5.2 5.3 Highlights About Map/Reduce… Summary Crawler Indexing process Web retrieval A Hard Coding Crawler and Miner ,aimed to get data for BBS lily The indexing process runs as a sequence of MapReduce operations. Set up a Web Interface for user to retrieve info. Crawler Web Interface Indexing process Related Work has three parts as blew: Highlights This stuff is COOL~ It can provide a friendly