Developingamultilingualtextanalysisengine-doesusing.pptVIP

Developing a multilingual text analysis engine - does using Unicode solve all the issues? Dr. Brian O’Donovan, IBM Ireland, Sept. 2002 Agenda What is IBM LanguageWare? Where/How is it Used? Our Experiences of Conversion to Unicode Benefits Accruing Challenges to be Overcome Word Identification problems Future work What is IBM LanguageWare? A suite of tools to assist in linguistic analysis Initially developed by IBM USA, but now developed by a Globally distributed team Used in internal and external products Available to OEMs as a toolkit Where are We? Developers North Carolina, USA Dublin, Ireland Helsinki, Finland Taipei, Taiwan Yamato, Japan Seoul, Korea Major Customers Various Locations, USA B?blingen, Germany 28 or 30+ Languages Supported Afrikaans Arabic Catalan Simplified Chinese Traditional Chinese Czech Danish Dutch x2 English x4 regions x2 domains Finnish French x2 German x3 Greek Hebrew Hungarian Icelandic Italian Japanese Korean Norwegian x2 Polish Portuguese x2 Russian Spanish Swedish Tamil Thai Turkish Analysis Hierarchy Where/How is it Used? Word Processors spell aid, grammar checker Search Engines Data Mining Extracting Lemmatized form Query Expansion e.g. Synonyms Assisting in Taxonomy Generation Translation Memory Identification of linguistic units NLP tools use as first stage of analysis Machine Translation limited Speech recognition not currently Conversion to Unicode Previous version used different code pages for each language In fact a collection of different engines New version has a single engine for all languages using Unicode not all languages have been converted yet Provided many benefits Raised some challenges Benefits of Unicode No more conversion tables Standardized on utf-16 Big-Endian Dictionaries Converted to platform byte order on load Possible to deal properly with multilingual text Able to utilize ICU utilities Can edit/view all test cases on one machine Issues with Unicode Lots of work Converting dictio