Web Search and Text Mining.pptVIP

Web Search and Text Mining Lecture 3 Outline Distributed programming: MapReduce Distributed indexing Several other examples using MapReduce Zones in documents Simple scoring Term weighting Distributed Programming Many tasks: Process lots of data to produce other data Want to use hundreds or thousands of CPUs Easy to use MapReduce from Google provides: ? Automatic parallelization and distribution ? Fault-tolerance ? I/O scheduling Focusing on a special class of dist/parallel comput MapReduce: Basic Ideas Input Output: each a set of key/value pairs User functions: Map and Reduce Map: input