News and Blog Analysis with Lydia UCR新闻和丽迪雅分校博客分析.pptVIP

News and Blog Analysis with Lydia Steven Skiena Dept. of Computer Science SUNY Stony Brook /~skiena Opportunities in Text Analysis The increasing volume of online information coupled with decreasing costs of communications and computation creates exciting new opportunities in text mining. We can analyze all of the 1000+ online English-language newspapers daily on a single commodity computer! Our ultimate goal is to build a computational model of much of the world’s knowledge through analysis of news media, reference texts, and primary sources. Learning From Text Sources Knowledge extraction becomes easier when you start with reliable sources: Online newspapers (both domestic and foreign) what is happening in the world? Scientific abstracts, e.g. Medline/Pubmed what is known about disease and medicine? Blogs and chat rooms what do people think is happening in the world? Tracking research interest in diseases over time Related Systems IBMs WebFountain Framework for web-scale text analytics Columbias Newsblaster Provides summaries of online newspaper articles Google News Automatic aggregation and presentation of online news. String Algorithms and NLP Since both fields deal with text there should be real connections, but NLP people parse sentences, not documents, so n20. The weirdnesses of language don’t tend to result in well-defined problems. Problems related to synonym set identification and phonetic hashing may be interesting to the CPM community. Outline of Talk Lydia NLP pipeline Spatial and temporal analysis Blogs vs. news Current research Future visions System Architecture Spidering – text is retrieved from a given site on a daily basis using semi-custom spidering agents. Normalization – clean text is extracted with semi-custom parsers and formatted for our pipeline Text Markup – annotates parts of the source text for storage and analysis. Back Office Operations – we aggregate