07DIctionary Based Annotation英文版本.pdf

Dictionary Based Annotation at scale with Spark, SolrTextTagger and OpenNLP Sujit Pal, Elsevier Labs Introduction •  About Me –  Work at Elsevier Labs. –  Interested in Search, NLP and Distributed Processing. –  URL: –  Email: sujit.pal@ –  Twitter: @palsujit •  About Elsevier –  World’s largest publisher of STM Books and Journals. –  Uses Data to inform and enable consumers of STM info. –  And like everybody else, we are hiring! Agenda •  Overview and Background •  Features and API •  Scaling out •  QA Overview/Background Problem Definition •  What is the problem? –  Annotate millions of documents from different corpora. •  14M docs from Science Direct alone. •  More from other corpora, dependency parsing, etc. –  Critical step for Machine Reading and Knowledge Graph applications. •  Why is this such a big deal? –  Takes advantage of existing linked data. –  No model training for multiple complex STM domains. –  However, simple until done at scale. Annotation Pipeline Dictionary Based NE Annotator (SoDA) •  Part of Document Annotation Pipeline. •  Annotates text with Named Entities from external Dictionaries. •  Built with Open Source Components –  Apache Solr – Highly reliable, scalable and fault-tolerant search index. –  SolrTextTagger – Solr component for text tagging, uses Lucene FST technology. –  Apache OpenNLP – Machine Learning based toolkit for processing Natural Language Text. –  Apache Spark – Lightning fast, large scale data processing. •  Uses ideas from other Open Source libraries –  FuzzyWuzzy – Fuzzy String Matching like a boss. •  Contributed back to Open Source –  /els