Rapid Development of Hindi Named Entity Recognition Using.pdfVIP

Rapid Development of Hindi Named Entity Recognition using Conditional Random Fields and Feature Induction (Short Paper) Wei Li University of Massachusetts Amherst and Andrew McCallum University of Massachusetts Amherst This paper describes our application of Conditional Random Fields (CRFs) with feature induction to a Hindi named entity recognition task. With only five days development time and little knowl- edge of this language, we automatically discover relevant features by providing a large array of lexical tests and using feature induction to automatically construct the features that most increase conditional likelihood. In an effort to reduce overfitting, we use a combination of a Gaussian prior and early-stopping based on the results of 10-fold cross validation. Categories and Subject Descriptors: I.2.7 [Artificial Intelligence]: Natural Language Processing - Text analysis; G.3 [Mathematics of Computing]: Probability and Statistics - Probablistic algorithms General Terms: algorithms; experimentation; languages Additional Key Words and Phrases: extraction; Conditional Random Fields; feature induction 1. INTRODUCTION Conditional Random Fields (CRFs) [Lafferty et al. 2001] are undirected graphical models, a special case of which correspond to conditionally-trained probabilistic finite state automata. Being conditionally trained, these CRFs can easily incorpo- rate a large number of arbitrary, non-independent features while still having efficient procedures for non-greedy finite-state inference and training. CRFs have shown suc- cess in various sequence modeling tasks including noun phrase segmentation [Sha and Pereira 2003] and table extraction [Pinto et al. 2003]. Following [McCallum 2003], we augment CRFs with feature induction to construct only those feature conjunctions that significantly increase the training label conditional likelihood. In this