Semi-supervisedlearningandself-training分析.pptVIP

Semi-supervised learning and self-training LING 572 Fei Xia 02/14/06 Outline Overview of Semi-supervised learning (SSL) Self-training (a.k.a. Bootstrapping) Additional Reference Xiaojin Zhu (2006): Semi-supervised learning literature survey. Olivier Chapelle et al. (2005): Semi-supervised Learning. The MIT Press. Overview of SSL What is SSL? Labeled data: Ex: POS tagging: tagged sentences Creating labeled data is difficult, expensive, and/or time-consuming. Unlabeled data: Ex: POS tagging: untagged sentences. Obtaining unlabeled data is easier. Goal: use both labeled and unlabeled data to improve the performance Learning Supervised (labeled data only) Semi-supervised (both labeled and unlabeled data) Unsupervised (unlabeled data only) Problems: Classification Regression Clustering … ? Focus on semi-supervised classification problem A brief history of SSL The idea of self-training appeared in the 1960s. SSL took off in the 1970s. The interest for SSL increased in the 1990s, mostly due to applications in NLP. Does SSL work? Yes, under certain conditions. Problem itself: the knowledge on p(x) carry information that is useful in the inference of p(y | x). Algorithm: the modeling assumption fits well with the problem structure. SSL will be most useful when there are far more unlabeled data than labeled data. SSL could degrade the performance when mistakes reinforce themselves. Illustration(Zhu, 2006) Illustration (cont) Assumptions Smoothness (continuity) assumption: if two points x1 and x2 in a high-density region are close, then so should be the corresponding outputs y1 and y2. Cluster assumption: If points are in the same cluster, they are likely to be of the same class. Low density separation: the decision boundary should lie in a low density region. …. SSL algorithms Self-training Co-training Generative models: Ex: EM with generative mixture models Low Density Separations: Ex: Transductive SVM Graph-based models Which SSL method