A Data Bootstrapping Recipe for Low-Resource Multilingual Relation Classification (2021)

Abstract: Relation classification (sometimes called ‘extraction’) requires trustworthy datasets for fine-tuning large language models, as well as for evaluation. Data collection is challenging for Indian languages, because they are syntactically and morphologically diverse, as well as different from resource-rich languages like English. Despite recent interest in deep generative models for Indian languages, relation classification is still not well-served by public data sets. In response, we present IndoRE, a dataset with 39K entity- and relation-tagged gold sentences in three Indian languages, plus English. We start with a multilingual BERT (mBERT) based system that captures entity span positions and type information and provides competitive monolingual relation classification. Using this system, we explore and compare transfer mechanisms between languages. In particular, we study the accuracy-efficiency tradeoff between expensive gold instances vs. translated and aligned ‘silver’ instances.

## introduction relation classification (sometimes called relation 'extraction @xcite is the task of identifying a se mantic relation (from a catalog of canonical re lations) that holds between two nominal entities in text. related to semantic role labeling, it is a critical capability for question answering and knowledge graph (kg) augmentation. the key challenge in re is the diversity of textual expres sions of canonical relations. wikidata, among the largest public kgs, has a hundred million entities with aliases but only thousands of well instantiated canonical relations, with relatively sparse textual descriptions of relations, particu larly in lowresource languages. lowresource languages, including many in dian languages, neither have welltuned standard nlp pipelines, nor unsupervised corpora as large as resourcerich languages. not only are they of ten morphologically distant from resourcerich lan guage families, but even their syntax for relation expression differs. e.g., while subjectpredicate object (infix) is more common in english, subject objectpredicate (postfix) is more common in indic languages: "ravi went to delhi" (en) vs. रिव िद-ल्ली गया था/"ravi delhi wentdid" (hi). word order is more relaxed: िदल्ली गया था रिव/"delhi wentdid ravi" is acceptable; only the entity types help ori ent the predicate. verb particles are more common than in english: "talk to him" (en) becomes उससे बात करो/ "himto talk do" (hi) or তার সােথ কথা বলু ন/"him with speech talk" (bn). transliteration and code switching are rampant: "i called him yes terday" (en) translates to " मैं ने कल उसे कॉल िकया था"/"i yesterday him call didpast", where call is transliterated. against this backdrop of diverse textual expres sion of relational facts, collecting highquality la beled data for indian language re is a major chal lenge, particularly given datahungry stateofthe art neural re systems. responding to these pres sures, we contribute a multilingual re testbed for lowresource indian languages. specifically, we build and contribute indore, a diverse and rich set of entity and relationannotated sentences in three indian languages -bengali (bn), hindi (hi) and telugu (te). in addition, we provide labeled en glish (en) re instances. our motive is to study and catalog a diversity of protocols for transfer ring re capability across languages, chiefly in the resourcerich to resourcepoor direction, start ing from the "each language for itself" (elfi) operating point. owing to distributional fidelity, we generally expect that, with 'sufficient' gold training data, elfi will show strong performance; the catch being the expense of collecting gold in stances in resourcepoor languages. to investigate the extent to which various modes of transfer can compensate for gold instances, we start by designing a competitive base re system built on mbert. 2 by sharing a multilingual word piece vocabulary and training on a vast multilin gual corpus, mbert already lends some support to go beyond elfi. beyond this baseline, we ex plore ways to train models for re in a target lan guage by 'borrowing' training instances from a given source language, translating and aligning, and preparing 'silver' instances to train an re sys tem for the target language. such model transfer @xcite ) may be attempted in a zeroshot mode, where no goldlabeled in stances are available in the target language, or in a fewshot mode where a few such instances are pre pared at some cost. we also explore an instance transfer setting, in which we train the re module using only english instances, and translate target language test instances to english as well. relation (r): spouse sentence (s): िवराट कोहली और अनु ष्का शमार् ने 2017 में इटली में शादी कर ली थी english: virat kohli and anushka sharma got married in italy in 2017. entity pair: िवराट कोहली , अनु ष्का शमार् lexical distance: 1 relation (r): award_received sentence (s): बृ हत-िंपगला गु जराती भाषा के िवख्यात सािहत्यकार रामनारायण पाठक द्वारा र चत एक छं द शा है जसके लये उन्हें सन् 1956 में गु जराती भाषा के लए मरणो-परां त सािहत्य अकादमी पु रस्कार से सम्मािनत िकया गया . english: brihatpingala is a verses scripture composed by noted gujarati litterateur ramnarayan pathak for which he was posthumously awarded the sahitya akademi award for gujarati language in 1956. entity pair: बृ हत-िंपगल , सािहत्य अकादमी पु रस्कार lexical distance: 24 the presence of multiple languages, with di verse levels of similarity between them, brings a new dimension to our study. we find that 0shot model transfer works better between indian lan guages than from english to an indian language, but is still behind elfi by 11.27% (bn), 11.06% (hi) and 13.43% (te) macro f1. additional fine tuning with 10shot target gold instances reduces the gap to 3.76% (bn), 2.8% (hi) and 4.23% (te). effectiveness of model transfer also shows inter esting and intuitive patterns and variations across language pairs in our transfer framework, tran srel. we observe that translating test instances to english helps a model trained on english instances perform well; however, the gain diminishes as we add a few training samples translated to english from the target language. ## indore data collection process we focus on three indian languages: hindi (hi), bengali (bn) and telugu (te). our intention was to pick two languages (hi and bn) that are more similar to each other than to the third language, te. however, each of these languages has certain dis tinctive features not found in the others. as an ex ample of a resourcerich language, we include en glish (en). in this section, we first discuss the limitations of existing multilingual re data sets. then we describe the data collection process, starting with relation selection, and followed by evidence sen tence collection, data processing and automatic an notation. then we discuss annotation evaluation and cleanup using human annotators. ## base re module we build a base re module which will be used for re in indian languages with various cross lingual transfer schemes. as depicted in figure 2 , our ar chitecture is built on multilingual bert (mbert), a masked language model. 8 for effective relation extraction in a multilin gual setting, at the input layer, we provide entity tagged inputs; the summarization layer collects im portant information about the sentence and entities; finally, the relation extraction layer predicts the relation using entity type information along with other contextual information. we try different vari ations of the architecture and describe the best pre forming variation here. we will describe some im portant details of preparing the input, the summa rization, and the output re layers. ## interlingual transfer mechanisms by using mbert, the base re system already exploits generalized language agnostic informa tion @xcite . in addition, our contex tual summaries and type encodings also help map entities and relations in different languages to a common space. over and above these, we explore transfer through machine translation. elfi (each language for itself): the base re system is trained using gold instances from the tar get language, and tested on other instances in the same language. at a high labeling cost, this gives an idea of the maximum achievable re quality. ## interlingual transfer performance for each language, we split the gold data for each relation into training (80% ) and testing (20%) folds. we keep the test fold fixed. we ensure zero overlap in entity pairs for every relation r between the folds, and report on 3way cross validation. we compare the re and me task variants and collect macro f1 scores (shown here) and 0/1 accuracy (given in appendix d). they show the same trend across methods. ## performance of ix here we compare the performance of instance transfer ix with that of the model transfer lmx. we experiment using the zeroshot and 10shot set tings with re and me. in ## augmented fewrel comparison here we investigate the effectiveness of extending an existing re dataset for indian languages. exist ing supervised dataset fewrel @xcite has 27 common relations with indore. we translate the en evidence sentences released by them to bn, hi and te using google trans late api, and map the entities in the translated sen tences. in ## transfer error analysis we investigate different type of errors occurring in lmx and ix setting. as we can see in the last example the major reason behind the occasional poor prediction of ix is the incomplete or wrong translation with erroneous en tity mapping from the target sentence. ## related work on re systems named entity recognition and relation extraction have been of continued interest in recent years. in this section we discuss relevant work that pro pose different architectures for increasingly accu rate named entity and relation prediction. zheng et al. (2017) jointly extract entities and relations based on a novel tagging scheme. chen et al. (2020) propose joint named entity recognition and relation extraction training in the clinical analytic space. in a similar direction, wang and lu (2020) propose tablesequence encoders for joint training. bekoulis et al. (2018) use adversarial training for joint entity and relation extraction. recently, a bertbased model (eberts and ulges, 2019) used spanbased encoding for joint extraction. all of these approaches show promising improvements in re from joint ner+re training in comparison to standalone re for monolingual data. fu et al. (2019) build a dependency graph and use a gcn to infer necessary information. the bilstmcrf based entity recognition model (nguyen and ver spoor, 2019) with a deep biaffine attention layer to model secondorder interactions among latent fea tures for relation classification shows great perfor mance for english relation extraction. however, these models incorporate specific building blocks (e.g., the dependency graph) which are not easy to procure with high accuracy in case of lowresource languages.