Data Mining and Machine Learning with EM - 北京大学网络与 数据挖掘和机器学习的EM北京大学网络与.pptVIP

An EM Example E-step M-step Apache Mahout Industrial Strength Machine Learning May 2008 Current Situation Large volumes of data are now available Platforms now exist to run computations over large datasets (Hadoop, HBase) Sophisticated analytics are needed to turn data into information people can use Active research community and proprietary implementations of “machine learning” algorithms The world needs scalable implementations of ML under open license - ASF History of Mahout Summer 2007 Developers needed scalable ML Mailing list formed Community formed Apache contributors Academia industry Lots of initial interest Project formed under Apache Lucene January 25, 2008 Current Code Base Matrix Vector library Memory resident sparse dense implementations Clustering Canopy K-Means Mean Shift Collaborative Filtering Taste Utilities Distance Measures Parameters Under Development Na?ve Bayes Perceptron PLSI/EM Genetic Programming Dirichlet Process Clustering Clustering Examples Hama (Incubator) for very large arrays Appendix Sean Owen, Robin Anil, Ted Dunning and Ellen Friedman,Mahout in action,Manning Publications; Pap/Psc edition (October 14, 2011) From Mahout Hands on, by Ted Dunning and Robin Anil, OSCON 2011, Portland Step 1 – Convert dataset into a Hadoop Sequence File /resources/testcollections/reuters21578/reuters21578.tar.gz Download (8.2 MB) and extract the SGML files. $ mkdir -p mahout-work/reuters-sgm $ cd mahout-work/reuters-sgm tar xzf ../reuters21578.tar.gz cd .. cd .. Extract content from SGML to text file $ bin/mahout org.apache.lucene.benchmark.utils.ExtractReuters mahout-work/reuters-sgm mahout-work/reuters-out Step 1 – Convert dataset into a Hadoop Sequence File Use seqdirectory tool to convert text file into a Hadoop Sequence File $ bin/mahout seqdirectory \ -i mahout-work/reuters-out \ -o mahout-work/reuters-out-seqdir \ -c UTF-8 -chunk 5 Hadoop Sequence File Sequence of Records, where each record is a Key,