CRAWLING THE HIDDEN WEB.ppt

FORM ANALYSIS How does a crawler interact with a search form? Crawler builds an “Internal Form Representation” F = ({E1, E2, …, En}, S, M) Label(E1) is descriptive text describing the field e.g. Date Domain(E1) is set of possible values for the field which can be finite (select box) of infinite (text box) set of n form elements submission information e.g. submission URL meta-information e.g. URL of form page, web site hosting form, links to form FORM ANALYSIS Label(E1) = Make Domain(E1) = {Acura, Lexus…} Label(E5) = Your ZIP Domain(E5) = {s | s is a text string} TASK SPECIFIC DATABASE How does a crawler know what keywords to put into the form fields? Crawler has a “task-specific database” For instance, if the task is to search archives pertaining to the automobile industry, the database will contain lists of all car makes and models. Database has a Label Value Set (LVS) table Each row contains L – a label e.g. “Car Make” V = {v1, …, vn} – a graded set of values e,g, {‘Toyota’, ‘Honda’, ‘Mercedes-Benz’, …} Membership function Mv assigns weight to each member of the set V TASK SPECIFIC DATABASE LVS table can be populated through Explicit initialization by human intervention Built-in entries for commonly used categories e.g. dates Querying external data sources e.g. Open Directory Project Crawler’s encounter with forms that have finite domain fields Categories Regional: North America: United States no yes yes TASK SPECIFIC DATABASE Computing weights M(v1) Case 1: Precomputed Case 2: Computed by respective data source wrapper Case 3: Computed by crawling experience shown below Extract Label Find Label in LVS Table Replace (L, V) with (L, V U Domain(E) Add new entry to LVS Find entry that close resembles Domain(E) and add Domain(E) to set Extracted? no Found? MATCHING FUNCTION “Matching function” maps values from database to form field Step 1: Label matching Normalize form label and use string matching algorithm to compute minimum