分词程序（Word segmentation program）.docVIP

分词程序（Word segmentation program） A few months ago, found a Chinese thesaurus material on the Internet (hundreds of K), then we want to write a word. What I did not study on Chinese word segmentation, also with their own imagination and writing. If the relevant experts, please give advice. Thesaurus Thesaurus has about more than 50 thousand words (Google can be found, similar thesaurus can use), I summarized as follows: Area 82 Important 81 Xinhua News Agency 80 Technique 80 Conference 80 Myself 79 Cadre 78 Employee 78 Mass 77 No 77 Today 76 Comrade 76 Sector 75 Reinforce 75 Tissue 75 The first column is the word, and the second column is the weight. The word segmentation algorithm I wrote does not use weights at present Two, design ideas Algorithms are described briefly: On a string S, after scanning once for each word from the lexicon for scanning, the longest match. For example, I suppose S= is a citizen of Peoples Republic of China, Peoples Republic of China thesaurus, China, citizen, people, Republic of China and other words. When...... to scan in word, then from the word start back are 1,2,3,... Words... (in, China, Chinese people, Chinese people, peoples Republic of China, the peoples Republic of China, Peoples Republic of China, Peoples Republic of China). In the lexicon the longest match string is Peoples Republic of China, so cut apart, the scanner advancing to the word public. Data structure: Choose what kind of data structure has great influence on the performance. I use Hashtable _rootTable to record a thesaurus. Key (key, insert number). For each word, if the words are N words, the words 1,1~2,1~3, 1~N...... words as keys, and the same inserted into _rootTable. If a key repeat insertion, subsequent incremental value. Three, procedures Specific procedures are as follows: the program contains weights, the number of insertion and other factors, the current algorithm does not use these: Chinese video tutorialwordUnit.cs //struct-- (word, weight) pair 1 Publi