Massively Multilingual Language Models for Cross Lingual Fact Extraction from Low Resource I ndian Languages (2022)

Abstract: Massive knowledge graphs like Wikidata attempt to capture world knowledge about multiple entities. Recent approaches concentrate on automatically enriching these KGs from text. However a lot of information present in the form of natural text in low resource languages is often missed out. Cross Lingual Information Extraction aims at extracting factual information in the form of English triples from low resource Indian Language text. Despite its massive potential, progress made on this task is lagging when compared to Monolingual Information Extraction. In this paper, we propose the task of Cross Lingual Fact Extraction(CLFE) from text and devise an end-to-end generative approach for the same which achieves an overall F1 score of 77.46

## introduction knowledge graphs are large structured sources of information about the world. recently, a lot of attention is being put in finding ways to automatically build or enrich extensive knowledge graphs (kgs) @xcite , @xcite . wikidata @xcite is one of the largest publicly available knowledge graphs which has over 99 million entities. knowledge graphs such as wikidata have been extensively used for multiple applications like text generation @xcite , question answering @xcite , @xcite etc. a knowledge graph is composed of multiple facts linked together. a fact is often represented as a triplet which consists of two entities and a semantic relation between them. this information can be encoded as a @xmath0, r, @xmath1, r is the relation and t represents the tail entity. fact extraction refers to the task of extracting structured factual information from natural language text @xcite . previously * equal contribution there has been extensive work regarding the task of monolingual fact extraction, especially in english @xcite , however not much attention has been given to the task of cross-lingual fact extraction. in this paper we propose an important task of multi-lingual and cross-lingual fact to text extraction(clfe) for 7 low resource(lr) indian languages and english. the task aims at directly extracting english triples from 8 different languages. we also propose strong baselines and approaches for this task which produce results comparable to existing mono-lingual state-of-the-art fact extraction pipelines and significantly better than other previous cross lingual attempts at fact extraction @xcite . our work enables utilisation of factual knowledge present in indic texts in order to increase the coverage of existing knowledge graphs. this would further help in multiple downstream tasks like fact verification, text generation etc. to the best of our knowledge, this is the first attempt at multilingual and cross-lingual fact extraction from lr indian languages. figure 1 shows multiple examples of the input and output for clfe task. overall, we make the following contributions. (1) propose the problem of cross-lingual and multilingual fact extraction for lr indian languages. (2) an end-to-end generative approach for extracting subject centric factual information from lr indian language text, which shows significant improvements over classification based pipelines. (3) we train multiple multi-lingual clfe models which lead to an overall f1 score of 77.46 . ## related work extracting structured information from free form text is a problem well worked upon. t-rex @xcite ing pipelines to perform fact linking between db-pedia @xcite abstracts and wikidata @xcite triples. re-fcog @xcite works in a cross lingual space to link the facts and outperforms the existing pipeline based approaches like @xcite . but these approaches are limited in their application since they perform fact linking and need a fact set as input. openie @xcite tackles this issue by leveraging linguistic structure for open domain information extraction. while the predecessor open domain ie systems like ollie @xcite use large set of patterns with broad coverage to extract facts, openie uses a small set of patterns which works well on canonically structured sentences. however, these open domain information extractors produce facts that have long and overspecific relations which can not be used to construct kgs. @xcite ); @xcite approach the information extraction problem by jointly extracting entities and their relation from input text using neural models without referring to any repository of facts. although these works produce systems which can extract open information from text in the webnlg dataset @xcite , they are monolingual and are limited to knowledge extraction in a single language. various existing well performing relation extraction models like @xcite , @xcite rely partially on exact match of entities in the source text, which makes it harder to adapt them for the clfe task. cross lingual fact extraction i.e extracting facts from source text of different languages didn't receive as much attention as monolingual fact extraction did. although @xcite worked on this task, with just a single language, the highest reported f1 is 33.67. moreover, fact extraction from low resource languages like indic languages hasn't been attempted. in this work, we attempt to reduce these gaps in information extraction by proposing systems for cross lingual subject centric fact extraction in low resource indic languages. ## dataset for the task of clfe we leverage the xalign dataset @xcite . the dataset contains 0.45m pairs across 8 languages, of which 5402 pairs have been manually annotated. the manually annotated part of the dataset was used as the golden test set. the sentences in xalign come from the wikipedia articles, about entities belonging to the human class, written in indian languages. the extensively cross lingual and multi lingual nature of the xalign dataset is ideal for the proposed task. though originally designed for the task of cross lingual data to text generation, the xalign dataset can be leveraged for cfle as well. however the dataset poses certain challenges. if we were to consider each relation as a class (for classification based approaches), the dataset is highly imbalanced. out of approximately 367 unique relations(classes), the most frequent class alone has a frequency of 27 % and top 20 classes contribute to 90% of the data. the data contains an average of 2.02 facts aligned per sentence. along with this, another challenging aspect of the dataset is that it is partially aligned. while the sentences in the test set have complete coverage in the aligned facts, the entire information present in the sentences from the train set is not covered by the aligned facts. this attribute of the dataset, text in any language head entity inputs pre-process inputs mt5 autoregressive generator model relation 1 -entity 1, relation 2 -entity 2,... relation 3 -entity 3 outputs figure 3: end to end architecture for clfe can potentially penalise the model even for the generation of correct facts during the training time. thus impacting recall scores during the test time. more details in appendix section a.2 ## methodology we propose two approaches for the clfe task. the first approach is a classification based approach which extracts tails first and then predicts the relation. second approach is a generative one that does both of these task in one shot. ## results and discussion it can be observed that the open ended approach performs the best in terms of f1 score while also providing complete flexibility regarding the possible entities and relations. another observation is that the strategy where we train separate bilingual models, works better than the combined model for just two languages, english and bengali. this is explained by the fact that these are the two most frequent languages for our dataset, which together constitute 54.44 % of our training data. thus, multilingual training proves to be useful over all, because of the shared learning across indian languages. we also observe that script unification (transliterating input scripts to devanagari), specifically benefits all the dravidian languages (te, ta, kn) of our dataset. it should be noted that the actual performance of the model might be better than what the numbers show. the reason for this is that currently we adhere to strict evaluation schemes where a word match between the predicted and the actual tail is necessary in order to determine the prediction as correct. however, this misses out on cases where the predicted and the ground truth tails are com-pletely synonymous. an example of this is the case where the model predicts the occupation as 'writer', whereas the gt label has it as 'author'. ## conclusion and future work in this work, we introduce the task of multilingual and cross-lingual fact extraction over english and seven other lr indic languages. we conclude that though script-unification helps certain languages, a single multilingual end-to-end generative pipeline performs better with overall f1 score of 77.46. this work paves the path for upcoming research in methods of extracting knowledge from lr indic language text. in future, we plan to explore approaches that make specific effort to tackle the partially aligned nature of the dataset in order to achieve further improvements.