A Systematic Cross-Comparison of Sequence Classifiers.pdf

A Systematic Cross-Comparison of Sequence Classifiers Binyamin Rozenfeld, Ronen Feldman, Moshe Fresko Bar-Ilan University, Computer Science Department, Israel grurgrur@, feldman@cs.biu.ac.il, freskom1@cs.biu.ac.il, Abstract In the CoNLL 2003 NER shared task, more than two thirds of the submitted systems used a feature-rich representation of the task. Most of them used the maximum entropy principle to combine the features together. Others used large margin linear classifiers, such as SVM and RRM. In this paper, we compare several common classifiers under exactly the same conditions, demonstrating that the ranking of systems in the shared task is due to feature se- lection and other causes and not due to inher- ent qualities of the algorithms, which should be ranked otherwise. We demonstrate that whole-sequence models generally outperform local models, and that large margin classifiers generally outperform maximum entropy-based classifiers. 1 Introduction Recently, feature-rich classifiers became state-of-the-art in sequence labeling tasks, such as NP chunking, PoS tagging, and Named Entity Recognition. Such classifiers are able to use any property of tokens and their contexts, if the property can be represented in the form of real- valued (usually binary) feature functions. Since almost all local properties can be represented in such a way, this ability is very powerful. Maximum-entropy-based models are currently the most prevalent type of feature-rich classifiers in se- quence labeling tasks. Such models define a probability distribution over the set of labelings of a sentence, given the sentence. In this, such classifiers differ from the generative probabilistic classifiers, such as HMM-based Nymble (Bikel et al., 1999) and SCFG-based TEG (Rosenfeld et al., 2004), which model the joint probabil- ity of sequences and their labelings, and which can use only a very limited range of context features. An alternative feat