Towards Improved Distantly Supervised Multilingual Named-Entity Recognition for Tweets (2022)

Abstract: Recent low-resource named-entity recognition (NER) work has shown impressive gains by leveraging a single multilingual model trained using distantly supervised data derived from cross-lingual knowledge bases. In this work, we investigate such approaches by leveraging Wikidata to build large-scale NER datasets of Tweets and propose two orthogonal improvements for low-resource NER in the Twitter social media domain: (1) leveraging domain-specific pre-training on Tweets; and (2) building a model for each language family rather than an all-in-one single multilingual model. For (1), we show that mBERT with Tweet pre-training outperforms the state-of-the-art multilingual transformer-based language model, LaBSE, by a relative increase of 34.6% in F1 when evaluated on Twitter data in a language-agnostic multilingual setting. For (2), we show that learning NER models for language families outperforms a single multilingual model by relative increases of 14.1%, 15.8% and 45.3% in F1 when utilizing mBERT, mBERT with Tweet pre-training and LaBSE, respectively. We conduct analyses and present examples for these observed improvements.

## introduction named-entity recognition (ner) is the process of detecting named mentions in text, and it is an essential subtask in several nlp applications such as information extraction @xcite , summarization @xcite and question answering @xcite . while resource-rich languages have received enormous focus over the last two decades, ner for low-resource languages is still under-explored due to the lack of resources -native speakers might not be even accessible -and the cost of labeling data needed to train supervised models for different languages. as a result, there has been emerging interest in multilingual ner, especially to process low-resource languages, in unsupervised and minimally supervised fashions. one aspect of multilingual ner is the need to build models that can generalize well across the underlying languages. however, when operating on social media text, multilingual ner becomes even harder @xcite @xcite because of linguistic diversity, short context and orthographic variation. recent research has shown success by leveraging a single multilingual model based on distantly supervised datasets derived from cross-lingual knowledge bases @xcite . we follow the work on building distantly supervised ner datasets by leveraging wikidata @xcite for tweets, where we do not assume access to long contexts nor manually labeled named entities in context. we then propose modeling techniques towards improved multilingual ner models for tweets, where we investigate how much pre-training language models on domain-specific data (tweets) and training ner models on the basis of language families improve ner performance. our contribution is threefold. we show that mbert with tweet pre-training outperforms labse @xcite , a state-ofthe-art multilingual language model, when evaluated in a language-agnostic multilingual setting on twitter data. in addition, we show that learning ner models for language families outperforms a single all-in-one multilingual model. our interpretation is that languages that belong to one family possess common linguistic features useful to learn an ner model. in contrast, joint learning of too many languages, most of which are unrelated, hinders the ability of the model to well fit any of the underlying languages. finally, we conduct analyses and present examples in german and arabic for the observed improvements. ## distantly supervised multilingual ner in order to answer our research questions, we construct distantly supervised monolingual and multilingual ner datasets of tweets (section 2.1) and train ner models of different characteristics (section 2.2). ## related work leveraging cross-lingual knowledge bases for the construction of multilingual ner datasets and gazetteers has proved successful. two largescale efforts are wikiann @xcite , wikipedia-based cross-lingual ner and entitylinking datasets in 282 languages, and polyglot-ner @xcite , ner datasets in 40 languages derived from wikipedia and freebase @xcite . on another hand, there have been a few efforts to construct distantly supervised ner datasets of tweets such as the work by @xcite and @xcite , which presented datasets of only 7,257 and 2,400 tweets, respectively. we follow similar approaches by leveraging wikidata @xcite to construct large-scale monolingual and multilingual ner datasets of tweets. fine-tuning transformer-based language models for ner has shown success. several works have utilized mbert @xcite to construct generic and domain-specific multilingual ner models @xcite @xcite . another example is labse @xcite . while mostly utilized for sentence-level nlp tasks such as hate-speech identification @xcite and claim matching @xcite , labse has also proven efficient for ner @xcite . in this work, we fine-tune both mbert and labse for ner in the twitter domain, where we learn and compare monolingual and multilingual models of different characteristics. @xcite shows that pre-training transformers towards a specific task or domains can provide significant benefits. @xcite show that pre-training mbert for the translation-pair prediction (tpp) objective improves ner. pre-training mbert on tweets has also been successful for a number of individual languages, such as english @xcite and @xcite . in this work, we pre-train mbert on tweets in 65 languages. several recent works utilize language classification towards improved multilingual models. the clustering can be based on either 1) language embeddings @xcite @xcite ; 2) language family with/without the use of hand-crafted rules such as geographical proximity @xcite ; and 3) token overlap @xcite . we perform family-based clustering for ner, similar to the first approach proposed by @xcite in the task of machine translation. however, we do not assume access to rich embeddings or linguistic knowledge for the language(s) of interest. ## conclusion and future work we proposed improvements to distantly supervised multilingual ner for tweets, where we leveraged wikidata to build large-scale monolingual and multilingual ner datasets of tweets. we showed that pre-training mbert on tweets outperforms labse by a relative f1 increase of 34.6% when evaluated on twitter data in a language-agnostic multilingual setting. we also showed that learning ner models for language families outperforms a single all-in-one multilingual model by relative f1 increases of at least 14.1%. in the future, we plan to produce larger tweet pre-trained language models, study more language families and leverage the work for multilingual entity linking for tweets in low-resource languages.