概率记录链接方式：小教程一则.pptVIP

Probabilistic Record Linkage: A Short Tutorial William W. Cohen CALD 范文精选网/ 范文大全/ 范文 Record linkage: definition Record linkage: determine if pairs of data records describe the same entity I.e., find record pairs that are co-referent Entities: usually people (or organizations or…) Data records: names, addresses, job titles, birth dates, … Main applications: Joining two heterogeneous relations Removing duplicates from a single relation Record linkage: terminology The term “record linkage” is possibly co-referent with: For DB people: data matching, merge/purge, duplicate detection, data cleansing, ETL (extraction, transfer, and loading), de-duping For AI/ML people: reference matching, database hardening In NLP: co-reference/anaphora resolution Statistical matching, clustering, language modeling, … Record linkage: approaches Probabilistic linkage This tutorial Deterministic linkage Test equality of normalized version of record Normalization loses information Very fast when it works! Hand-coded rules for an “acceptable match” e.g. “same SSNs, or same zipcode, birthdate, and Soundex code for last name” difficult to tune, can be expensive to test Record linkage: goals/directions Toolboxes vs. black boxes: To what extent is record linkage an interactive, exploratory, data-driven process? To what extent is it done by a hands-off, turn-key, autonomous system? General-purpose vs. domain-specific: To what extent is the method specific to a particular domain? (e.g., Australian mailing addresses, scientific bibliography entries, …) Record linkage tutorial: outline Introduction: definition and terms, etc Overview of the Fellegi-Sunter model Classify pairs as link/nonlink Main issues in Felligi-Sunter model Some design decisions from original Felligi-Sunter paper other possibilities Felligini-Sunter: notation Two sets to link: A and B A x B = {(a,b) : a2A, b2B} = M [ U M = matched pairs, U= unmatched pairs Record for a2 A is a(a), for b2 B is b(b) Comparison vector, wri