MOSAIC o: a Multilingual Open-text Semantically Annotated Interlinked Corpus (2024)

Abstract: Several Natural Language Understanding (NLU) tasks focus on linking text to explicit knowledge, including Word Sense Disambiguation, Semantic Role Labeling, Semantic Parsing, and Relation Extraction. In addition to the importance of connecting raw text with explicit knowledge bases, the integration of such carefully curated knowledge into deep learning models has been shown to be beneficial across a diverse range of applications, including Language Modeling and Machine Translation. Nevertheless, the scarcity of semantically-annotated corpora across various tasks and languages limits the potential advantages significantly. To address this issue, we put forward MOSAICo, the first endeavor aimed at equipping the research community with the key ingredients to model explicit semantic knowledge at a large scale, providing hundreds of millions of silver yet high-quality annotations for four NLU tasks across five languages. We describe the creation process of MOSAICo, demonstrate its quality and variety, and analyze the interplay between different types of semantic information. MOSAICo, available at https://github.com/SapienzaNLP/mosaico, aims to drop the requirement of closed, licensed datasets and represents a step towards a level playing field across languages and tasks in NLU.

## introduction there are two predominant schools of thought on how to integrate semantics into natural language processing (nlp) systems. most researchers lean towards the implicit integration of semantics through self-supervised language modeling @xcite @xcite , a framework that proved to be successful in numerous language understanding benchmarks @xcite . however, this approach -while conceptually straightforward and effective by scaling up the number of parameters -has raised concerns regarding its cost-effectiveness and environmental impact @xcite . furthermore, the black-box nature of this approach has also prompted calls for a critical inquiry into whether the impressive results on language benchmarks genuinely reflect a system's comprehension of language @xcite @xcite . conversely, other researchers have advocated for the integration of discrete symbols into ai systems, which has recently been resurfacing under the umbrella term of "neuro-symbolic ai" (d'avila @xcite . in nlp, discrete symbols can be used to represent word meanings, and graphs of symbols can be interpreted as sentence-level semantics. the integration of such explicit semantics into nlp systems may have the potential to address the aforementioned concerns @xcite . indeed, explicit semantics has been shown to be effective in reducing the number of parameters of neural networks and in improving the interpretability of their outputs. however, to the best of our knowledge, we still lack a critical enabler for training such approaches, namely, a vast, high-quality dataset annotated with symbolic knowledge. to address this issue, we introduce mosaico, a novel resource designed to foster explorations and modeling of explicit semantics on an extensive scale and across multiple languages. with its hundreds of millions of silver annotations on wikipedia sentences across 5 languages for 4 important nlu tasks -namely, word sense disambiguation, semantic role labeling, semantic parsing, and relation extraction -mosaico is a large, diverse, and re my cat ate the mouse that i bought at the apple store when i traveled to seattle last summer wsd cat#1 eat#1 mouse#4 buy#1 shop#1 travel#1 last#1 summer#1 srl agent eat • bite patient theme agent buy location time time agent travel destination time manufacturer location sp eat-01 cat mouse i buy-01 store travel-01 company city date-entity name name summer year "apple" "seattle" last :op1 :op1 :a r g 0 :poss :a r g 1 :arg1-of :arg0 :location :mod :name :t im e :arg0 :arg4 :name :t im e :se aso n :mod :mod robust dataset that can bootstrap research on the integration of explicit semantics into nlp systems to train and evalute models. in addition, mosaico is also a tool for investigating the interaction between different types of symbols, structures, and semantics, as shown in figure 1 . in this paper, we describe the construction of mosaico, which can save critical time and compute for future research, and show its effectiveness for training state-of-the-art systems on open data as opposed to the closed and copyrighted datasets that are traditionally used in the literature, which will aid in lowering the barrier for the development of next-generation systems. we also provide insights into the interaction between different nlu tasks, as diverse sources of semantic knowledge allow us to look at new opportunities enabled by the availability of a large, cross-lingual, semantically-tagged corpus. we hope mosaico will be a tool that facilitates deeper investigations of the intersection between natural language and explicit semantics, while also fostering analysis of the connection between different nlu tasks across languages. ## background and motivation this section briefly introduces the four semantic tasks we considered in mosaico, and further explains how our resource can help the research community. due to space constraints and the extensive literature available for each task, we focus on the works that are within the scope of this paper. word sense disambiguation (wsd) aims at associating a word in context with the most suitable meaning from a predefined sense inventory @xcite . wsd is a challenging task for machines and humans, especially when using fine-grained sense inventories like wordnet @xcite . nevertheless, grounding words to discrete senses can bring several benefits, including better language modeling @xcite , machine translation @xcite , and lexical bias benchmarking in applications @xcite . however, traditional datasets for wordnet-based wsd are limited in size, outdated, and mostly english-specific. even though sense inventories are available for multilingual wsd, such as babelnet @xcite , creating highquality training datasets for non-english languages is difficult @xcite . mosaico aims to overcome two limitations in today's wsd: i) sense coverage, i.e., current english datasets provide limited coverage of the long tail of the sense distribution, and ii) multilingual coverage, i.e., the lack of highquality annotations for non-english languages. semantic role labeling (srl) is informally described as the task of answering "who did what to whom, where, when, and how?" @xcite . given a sentence, srl usually requires: i) predicate identification, i.e., finding all the words that denote an action; ii) predicate sense disambiguation, i.e., selecting the most appropriate sense for each identified predicate; iii) argument identification, i.e., detecting the text spans that represent the participants in the action; and, iv) argument classification, i.e., assigning a semantic role to each predicate-argument pair. the predicate senses and semantic roles used in srl are defined according to an inventory of predicate-argument structures, e.g., propbank @xcite , framenet @xcite , or verbatlas @xcite , among others. mosaico provides annotations using propbank -due to its popularity -and verbatlas -due to its manual linkage to wordnet and babelnet and its cross-frame relations -and addresses three shortcomings of srl: i) open data availability, i.e., popular datasets are not open @xcite @xcite , ii) multilingual coverage, i.e., limited availability of large multilingual datasets, and iii) cross-inventory annotations, i.e., data annotated with more than one type of predicate-argument structure inventory. semantic parsing (sp) is the task of encoding the meaning of a sentence in a machineinterpretable structure @xcite . since the focus of mosaico is on semantics, we direct our attention to formalisms that encode the abstract meaning of a sentence, such as amr @xcite and bmr @xcite . thanks to recent advances @xcite , modern sp systems can now be used with high accuracy. however, sp still suffers from issues similar to those in wsd and srl, perhaps amplified by the extent of the manual effort required to annotate text with complex graphs. more specifically, the role of mo-saico is to provide the first large-scale dataset of annotations for amr in five languages with the aim of mitigating two problems in sp: i) shortage of open datasets in english but also in other languages, and ii) coverage, i.e., the lack of semantic parses for a broad range of domains. relation extraction (re) is the task of extracting semantic relationships between entities from unstructured text. since mosaico is based on wikipedia, we focus on relation types from wikidata. while previous work has mostly focused on named entities @xcite , equally important relations are also present between concepts. mosaico allows us to explore interactions between entities identified through wikipedia hyperlinks, and, unlike previous efforts, concepts identified through wsd, so as to disambiguate the subject and object of a semantic relation from wikidata. this enables the creation of a vast network of concepts and entities interconnected through semantic relations tied to their multilingual textual context. to leverage them, we will explore how many of these relations are already present in wikidata and whether mo-saico can provide new and reliable relation annotations. therefore, mosaico provides for re: i) wide-coverage extraction of relational triplets foot_1 connected to their lexical-semantic context, which was unfeasible without the interconnection with wsd at hand, ii) a novel way of augmenting wikidata, and iii) multilingual coverage, i.e., addressing the limited availability of high-quality, high-recall, multilingual datasets for re. ## mosaico our main objective with mosaico is to create a large collection of silver annotations on multilingual data. however, we believe that the value of mosaico does not only lie in the creation of annotations whose quality is comparable to goldstandard datasets. indeed, while several groups may have the resources and expertise to reproduce similar efforts, we introduce mosaico to also address the following issues: we now describe the creation process of mo-saico, our novel resource of silver-quality annotations for modeling semantics at a large scale. we start with an outline of the data collection process (section 3.1) and then detail how we produce the annotations (section 3.2). ## experiments and results to evaluate the quality of mosaico, we train a set of state-of-the-art models on our silver datasets for wsd, srl, re, and sp; then, we compare their results with those obtained by the same systems trained on gold datasets. ## insights and opportunities here, we highlight the opportunities mosaico brings to interconnect the different types of knowledge captured by each semantic task. mosaico is a test for multilingual wsd. indeed, one feature often overlooked in wikipedia is that, in each article, the first mention of a term has been manually hyperlinked to the corresponding wikipedia article if it exists. therefore, we can use the 1:1 mapping @xcite between wikipedia and babelnet to convert wikipedia interlinks to babelnet synsets and create wiki-wsd, a large novel benchmark for multilingual wsd. for reference, we employ wiki-wsd to evaluate escher gold , as shown in ## conclusion and future work in this paper, we presented mosaico, the first effort in the field of nlu aimed at equipping researchers with the key ingredients for modeling symbolic semantic knowledge at a large scale. mo-saico provides hundreds of millions of annotations in four nlu tasks across five languages, filling a crucial gap in the availability of semanticallytagged corpora. after describing the creation process of mosaico, we analyzed the interplay between different sources of semantic information and provided insights into how they can be combined to bring different types of explicit knowledge together. finally, we demonstrated the quality of the annotations in mosaico, which can improve the performance of state-of-the-art systems across standard benchmarks, showing its value for the research community. we release mosaico to the community, allowing future work to drop the requirement of licensed datasets and making research in this area accessible to a wider community.