m OKB 6: A Multilingual Open Knowledge Base Completion Benchmark (2023)

Abstract: Automated completion of open knowledge bases (Open KBs), which are constructed from triples of the form (subject phrase, relation phrase, object phrase), obtained via open information extraction (Open IE) system, are useful for discovering novel facts that may not be directly present in the text. However, research in Open KB completion (Open KBC) has so far been limited to resource-rich languages like English. Using the latest advances in multilingual Open IE, we construct the first multilingual Open KBC dataset, called mOKB6, containing facts from Wikipedia in six languages (including English). Improvingthe previous Open KB construction pipeline by doing multilingual coreference resolution andkeeping only entity-linked triples, we create a dense Open KB. We experiment with several models for the task and observe a consistent benefit of combining languages with the help of shared embedding space as well as translations of facts. We also observe that current multilingual models struggle to remember facts seen in languages of different scripts.

## introduction open information extraction (open ie) systems @xcite such as reverb @xcite and openie6 @xcite can extract triples, or facts, of the form (subject phrase, relation phrase, object phrase), which can be @xcite , from text (e.g., wikipedia articles) without using any pre-defined ontology. open knowledge base (open kb) is constructed using these open ie triples where the subject phrases and object phrases are nodes and relation phrases are labels on edges connecting the nodes in the graph. ## related work multilingual open kbc datasets are absent in literature to the best of our knowledge, although multiple english open kbc datasets are available. olpbench @xcite , derived from opiec @xcite , is a large-scale open kbc dataset that contains 30m triples and is constructed from english wikipedia using minie system @xcite . the evaluation data contains 10k triples randomly sampled from 1.25m linked triples. reverb45k @xcite and reverb20k @xcite are smaller open kbc datasets constructed from clueweb09 corpus 2 using reverb open ie system @xcite . both the datasets keep only those tuples in which both the subject phrase and object phrase link to a finite set of freebase entities. multilingual open ie (mopenie) systems like gen2oie @xcite and multi 2 oie @xcite enable extracting facts from multiple languages. we use the gen2oie model for constructing mokb6 dataset as it is trained with language-specific facts transferred from english, while multi 2 oie relies on zero-shot transfer for languages other than english. ## dataset curation we aim to construct a dense multilingual open kb that maximizes the information about a given realworld entity, which may be represented as multiple nodes across languages. therefore, we consider those wikipedia articles foot_0 that are available in six languages: english, hindi, telugu, spanish, portuguese, and chinese foot_1 . this will also help the model learn from facts in high resource language like english and answer queries in low resource language like telugu. we work with 300 titles randomly sampled from the ones common among all six languages (found using mediawiki-langlinks (mediawiki, 2021)). thus, we extract facts from 6×300 wikipedia articles. we discuss the three stages of our pipeline below. stage 1 we first process each wikipedia article through a coreference resolution system. although language-specific end-to-end neural coref models have been developed @xcite , multilingual models that work on all our languages of interest are absent in the literature. therefore, we retrain wl-coref @xcite with xlm-r @xcite on the english training data (available in ontonotes @xcite ) that can work zero-shot for other languages. coref models detect and cluster mentions, but do not identify a canonical cluster name, which is needed for standardizing all the mentions in the cluster. to find cluster names, entity linking systems such as mgenre @xcite or wikipedia hyperlinks can be used. however, we found that they result in low recall, particularly for low resource languages. thus, we employ a heuristic to find the cluster name and replace each of the coreferent mentions with it. the score for each mention is represented by a tuple, computed as: score(mention phrase) = (#proper nouns, #nouns, #numerals, # adjectives, #pronouns, #verbs). the tuple is ordered according to the importance of each field (pos tags) for the cluster name, which is determined empirically. two tuples are compared index-wise with higher priority given to lower indices to determine the best scoring mention that is chosen as the canonical name ( stage 2 we use gen2oie to extract open ie triples from the coreference resolved sentences. mentions scores cluster name barack obama (2,0,0,0,0,0) obama (1,0,0,0,0,0) barack obama he (0,0,0,0,1,0) our construction pipeline improves over opiec in three ways: (1) we use a multilingual open ie system, instead of an english-specific open ie system like in opiec, enabling us to curate open kbs in many languages, (2) we add a multilingual coreference resolution system in our pipeline, and (3) the english test triples are manually verified. further, we manually evaluate and review the noise at each step of data curation in section 4. ## noise evaluation curating an open kb involves various stages and each stage induces its noise in the construction pipeline @xcite . we manually evaluate the noise induced at each stage of our pipeline (figure 1 ) and discuss the same in this section. we ask native speakers of four (out of six) languages -english, hindi, telugu, and chinese to assess the output quality, or precision, of each stage as discussed below. in the first stage, we assess the performance of the coreference resolution system over wikipedia articles. we find a high precision of 95.5% in coref's mention clustering and 89.82% accuracy in finding canonical cluster name (using the heuristic illustrated in for evaluating the open ie system, gen2oie, in the second stage, we mark an extraction of a sentence as correct if it has syntactically correct arguments and it is coherent with the sentence. we get an average precision of 63.4% on 80 extractions (20 in each language). we evaluate the triples, or open kb facts, at the last stage after passing through various noiseremoving filters. note that these triples also form the train set (and dev set) in mokb6 dataset. we mark triples as correct when they contain realworld entities, and also, factual information about them. if the triple is very generic or contextual (see appendix e), it is marked as incorrect. we find the train (and dev) set quality to be 69.3%, averaged over 80 triples in four languages. ## conclusion and future work we create and release the mokb6 dataset, the first multilingual open knowledge base completion dataset with 42k facts in six languages: english, hindi, telugu, spanish, portuguese, and chinese. steps to improve the quality of the extracted facts. we also report the first baselines on the task using the existing state of the art kge models trained with facts from different languages using various augmentation strategies.