6.6-实例10-文本词频统计.pptxVIP

Python语言程序设计实例10: 文本词频统计文本词频统计问题分析问题分析文本词频统计- 需求：一篇文章，出现了哪些词？哪些词出现得最多？- 该怎么做呢？英文文本中文文本问题分析文本词频统计- 英文文本：Hamet分析词频https://python123.io/resources/pye/hamlet.txt- 中文文本：《三国演义》 分析人物https://python123.io/resources/pye/threekingdoms.txtHamlet英文词频统计实例讲解#CalHamletV1.pydef getText():txt = open(hamlet.txt, r).read()txt = txt.lower()for ch in !#$%()*+,-./:;=?@[\\]^_‘{|}~:txt = txt.replace(ch, )return txthamletTxt = getText()words = hamletTxt.split()counts = {}- 文本去噪及归一化- 使用字典表达词频for word in words:counts[word] = counts.get(word,0) + 1items = list(counts.items())items.sort(key=lambda x:x[1], reverse=True)for i in range(10):word, count = items[i]print({0:10}{1:5}.format(word, count))theandto1138965754669550542542514ofyoui- 运行结果由大到小排序- 观察单词出现次数amyhamlet 462in 436准备好电脑，与老师一起编码吧！《三国演义》人物出场统计实例讲解(上)#CalThreeKingdomsV1.pyimport jiebatxt = open(threekingdoms.txt, r, encoding=utf-8).read()words = jieba.lcut(txt)counts = {}for word in words:if len(word) == 1:continueelse:counts[word] = counts.get(word,0) + 1- 中文文本分词- 使用字典表达词频items = list(counts.items())items.sort(key=lambda x:x[1], reverse=True)for i in range(15):word, count = items[i]print({0:10}{1:5}.format(word, count))曹操孔明将军却说玄德关公丞相二人不可荆州玄德曰孔明曰不能如此张飞953836772656585510491469440425- 中文文本分词390390384- 使用字典表达词频378358准备好电脑，与老师一起编码吧！《三国演义》人物出场统计实例讲解(下)《三国演义》人物出场统计将词频与人物相关联，面向问题词频统计人物统计#CalThreeKingdomsV2.pyimport jiebatxt = open(threekingdoms.txt, r, encoding=utf-8).read()excludes = {将军,却说,荆州,二人,不可,不能,如此}words = jieba.lcut(txt)counts = {}for word in words:if len(word) == 1:continueelif word == 诸葛亮 or word == 孔明曰:rword = 孔明elif word == 关公 or word == 云长:rword = 关羽elif word == 玄德 or word == 玄德曰:rword = 刘备elif word == 孟德 or word == 丞相:rword = 曹操- 中文文本分词else:- 使用字典表达词频- 扩展程序解决问题rword = wordcounts[rword] = counts.get(rword,0) + 1for word in excludes:del counts[word]items = list(counts.items())items.sort(key=lambda x:x[1], reverse=True)for i in range(10):word, count = items[i]print({0:10}{1:5}.format(word, count))曹操孔明刘备关羽张飞商议如何主公军士吕布145113831252784-