PowerPoint 프레젠테이션 - 欢迎来到清华大学自然语 ….pptVIP

清华大学计算机系 清华大学计算机系 乔维 A search-based Chinese Word Segmentation Method ——WWW 2007 Xin-Jing Wang: IBM China Wen Liu: Huazhong Univ. China Yong Qin: IBM China Introduction Challenges in CWS Ambiguous Unknown word Web and search technology Free from OOV problem Adaptive to different segmentation standards Entirely unsupervised The proposed approach Segments Collecting Query sentence = sub-sentence (by punctuation) Submit sub-sentence to a search engine Collect the highlights from returned snippets Query：“我明天要去止锚湾玩”： The proposed approach Segments Scoring Select a subset of segments as final segmentation Frequency-based: term frequency Segment occurrences : total number of occurrences SVM-based SVM classifier with RBF kernel and maps the outputs into probabilities as the scores Reconstruct the query using the segment way with highest score The proposed approach Segments Selecting Valid subset: if its member segments can reconstruct exactly the query Score of valid subset: the average score of its member segments. Greedy search to find valid subset For efficiency consideration Select the valid subset which has highest score as final segmentation Evaluations Experiment setting SVM-based score Training set: 3000 randomly selected sentences Feature space ——Three dimensional : TF DF LEN TF: term frequency DF: number of documents indexed by a segment Len: number of characters in a segment Frequency-based score need no training set Evaluations Comparison result SIGHAN’05 Evaluations Worse than reported results Why SVM is worse ? Feature space too simple Advantage: only 3000 or non training set Avoids OOV problem Better performance can be achieved with more search results provided (Google+Yahoo!) Evaluations Comparison to IBM full parser Conclusion It is good at discovering new words (no OOV problem) and adapting to different segmentation standards Entirely unsupervised which saves labors to labeling training data. F