python 基础练习题.docxVIP

Advanced computation linguistics 1. Collect the most frequent words in 5 genres of Brown Corpus: news, adventure, hobbies, science_fiction, romance To collect most frequent words from the given genres we can follow the following steps: import nltk from nltk.corpus import brown brown.categories() [adventure, belles_lettres, editorial, fiction, government, hobbies, humor, learned, lore, mystery, news, religion, reviews, romance , science_fiction] news_text = brown.words(categories=[news,adventure,hobbies,science_fi ction,romance]) from bability import FreqDist fdist=FreqDist([w.lower() for w in news_text]) voca=fdist.keys() voca[:50] [the, ,, ., and, of, to, a, in, he, , ``, was, for, that, it, his, on, with, i, is, at, had, ?, as, be, you , ;, her, but, she, this, from, by, --, have, they, said, not, are, him, or, an, one, all, were, would, there, !, out, will] voca1=fdist.items() voca1[:50] [(the, 18635), (,, 17215), (., 16062), (and, 8269), (of, 8131), (to, 7125), (a, 7039), (in, 5549), (he, 3380), (, 3237), (``, 3237), (w as, 3100), (for, 2725), (that, 2631), (it, 2595), (his, 2237), (on, 2 162), (with, 2157), (i, 2034), (is, 2014), (at, 1817), (had, 1797), ( ?, 1776), (as, 1725), (be, 1610), (you, 1600), (;, 1394), (her, 1368) , (but, 1296), (she, 1270), (this, 1248), (from, 1174), (by, 1157), ( --, 1151), (have, 1099), (they, 1093), (said, 1081), (not, 1051), (are , 1019), (him, 955), (or, 950), (an, 911), (one, 903), (all, 894), ( were, 882), (would, 850), (there, 807), (!, 802), (out, 781), (will, 775)] This means that the frequency of word “the” is more than others. 2. Exclude or filter out all words that have a frequency lower than 15 occurrencies. (hint using conditional frequency distribution) By adding functionalities on the first task of collecting words based on their frequency of occurrences, we can filter words which has frequency occurrence of =15. filteredText= filter(lambda word: fdist[word]=15,fdist.keys()) voca=