面向互联网应用的中文浅层语言分析技术ResearchonChineseLexicalAnalysisbit.ppt

展望：Next Generation Web II From information to Message Interactive, dynamic Community Instant Messenger, Online chat: MSN Messenger, ICQ, QQ 展望：Next Generation Web III From one single dimension to multi-dimension Text content, multimedia data Timestamp Information structure Relationship network: such as sender receiver Sender time Receiver data Toward Next Generation Web Computing … P2P-centered Web platform Personalization Modeling and Personalized Web Services Message Full-Scale Mining: temporal reasoning, text mining, data mining, knowledge management, community generation … Customized Shallow Chinese Language Computing would be popular. Thanks! * Content Layouts * Content Layouts ICTCLAS：HHMM Architecture Corpus Character String Word graph Class-based WS model Role model Training NSP rough segment Unknown word recognition Optimized selection Lexical result Atom Segment ICTCLAS: Word Segmentation ci = wi iff wi is listed in the segmentation lexicon; PER, LOC, ORG, TIME or NUM iff wi is an unknown named entity; STR iff wi is an unknown symbol string; BEG iff beginning of a sentence END iff ending of a sentence OTHER otherwise. Word class definition Class-based segmentation model ICTCLAS: Unknown word recognition In unknown word recognition, we mainly deal with Named Entities, such as person names, location names, organization names, and transliterations of foreign names We use two-level HMM for unknown word recognition In 1st level HMM, we recognize person names，simple location names, transliterations of foreign names and other proper names In 2nd level HMM, we recognize complex location names and organization names, usually with some simple unknown words as its components. ICTCLAS : Unknown word recognition(Cont.) We divided the role tag set into an internal tag set and an external tag set The role tags in the internal tag set represent the component of the unknown words The role tags in the external tag set represent the context of the unknown