文本挖掘教程.pptVIP

Text-Mining Tutorial Marko Grobelnik, Dunja Mladenic J. Stefan Institute, Slovenia What is Text-Mining? “…finding interesting regularities in large textual datasets…” (Usama Fayad, adapted) …where interesting means: non-trivial, hidden, previously unknown and potentially useful “…finding semantic and abstract information from the surface form of textual data…” Which areas are active in Text Processing? Tutorial Contents Why Text is Easy and Why Tough? Levels of Text Processing Word Level Sentence Level Document Level Document-Collection Level Linked-Document-Collection Level Application Level References to Conferences, Workshops, Books, Products Final Remarks Why Text is Tough? (M.Hearst 97) Abstract concepts are difficult to represent “Countless” combinations of subtle, abstract relationships among concepts Many ways to represent similar concepts E.g. space ship, flying saucer, UFO Concepts are difficult to visualize High dimensionality Tens or hundreds of thousands of features Why Text is Easy? (M.Hearst 97) Highly redundant data …most of the methods count on this property Just about any simple algorithm can get “good” results for simple tasks: Pull out “important” phrases Find “meaningfully” related words Create some sort of summary from documents Levels of Text Processing 1/6 Word Level Words Properties Stop-Words Stemming Frequent N-Grams Thesaurus (WordNet) Sentence Level Document Level Document-Collection Level Linked-Document-Collection Level Application Level Words Properties Relations among word surface forms and their senses: Homonomy: same form, but different meaning (e.g. bank: river bank, financial institution) Polysemy: same form, related meaning (e.g. bank: blood bank, financial institution) Synonymy: different form, same meaning (e.g. singer, vocalist) Hyponymy: one word denotes a subclass of an another (e.g. breakfast, meal) Word frequencies in texts have power distribution: …small number of very frequent words …big number of low frequency