萨师煊国际大数据分析与研究中心16.pptVIP

* * * * * * * * * * * * * * * * * * Extract correct SRRs from returned response pages while discarding irrelevant information. The problem is to identify the rules (often called wrapper) that can extract the correct SRRs. Automatic SRR Extraction (2) General methodology Utilize the tag strings/DOM trees/visual information on one or more result pages from the same search engine to mine extraction patterns. Identify the minimal data-rich region/subtree that likely contains the SRRs. Identify separator(s) that separate different SRRs. More recent solutions use more visual information on result pages. Still cannot handle complex result pages well (javascript, multiple columns, multiple sections, multiple SRR formats) Automatic SRR Extraction (3) Result Merging (1) Problem: Merge returned documents from multiple sources into a single ranked list. Difficulties Full documents of search results are not available or too expensive to download and analyze on the fly. Local similarities (thus local ranks) are usually not comparable due to different similarity functions different term weighting schemes different statistical values, e.g., global idf vs. local idf Result Merging (2) A large number of solutions has been proposed to perform result merging. Some use local similarities associated with each result (modern search engines no longer provide the information). Some use local ranks of search results. Some analyze downloaded full documents. Some use the titles and snippets of the search results. Some consider the quality of the used search engine. Some consider whether a result is retrieved from multiple search engines. Some use a sample set of documents from each search engine Information that could be utilized for result merging: Local similarity or local rank of each result Title of each result Snippet of each result Publication time of each result Organization/person who published the result (from URL) Size of each result Number of search engines that returned the re