台北医学大学医学资讯研究所.ppt

蔣以仁 Knowledge Discovery from Clustering Document 蔣以仁 臺北醫學大學醫學資訊研究所 ijchiang@.tw May 2, 2005 20 Hot Technologies – PC Magazine Carbon Nanotube Self-Driving Cars Biosensors OLED Fuel Cell E-Bomb Text Mining … Knowledge Sinks The business opportunity in text mining… Elements of Text Info Management Technologies Text Mining for Hypertext Creation Type of Links Drawbacks of Manual Taxonomies Access information are quickly becoming inadequate as the amount of on-line information growing at unprecedented rate Have too little information available for users to peruse manual classification of new documents within the existing taxonomy (immense bottleneck) Time Spent Per Visitor (minutes)by Search Engine, June 2002 Generality of Basic Techniques A Vector-Space Filtering Model Document Representation Vector Space each document is characterized by a Boolean or numerical vector Simple Document Computing is not about computers any more. It is about living. To live is to compute! Stemming Document - reduce words to some root form Comput i not about comput ani more. It i about live. To live i to comput! Vector Representation TFIDF Weighting IDF(t) = log( ) N is the total number of document in the collection nt is the number of documents in which t appears at once TFIDF(t,d) = e(t, d) / IDF(t) weight for a term t in a document d Boolean Vectors f(a) { Boolean model Probabilistic classification not consider frequency Eliminating Stop Words Stop words Little inherent topical content prepositions, conjunctions and pronouns Ex.: a, able, about, after, again, all, but, best, by, came, can, despite, did, during, enough, even, get, goes, etc.. Feature Selection Yang and Pedersen 1997 Comparison of different selection criteria DF – document frequency IG – information gain MI – mutual information (pointwise) CHI – chi square Common strategy Compute statistic for each term Keep n terms with highest value of this statistic ?2 statistic (CHI) ?2 is interested in (fo – fe)2/fe summ