Mitigating Data Scarcity in Semantic Parsing across Languages with the Multilingual Semantic Layer and its Dataset (2024)

Abstract: Data scarcity is a prevalent challenge in the era of Large Language Models (LLMs). The insatiable hunger of LLMs for large corpora becomes even more pronounced when dealing with non-English and low-resource languages. The issue is particularly exacerbated in Semantic Parsing (SP), i.e. the task of converting text into a formal representation. The complexity of semantic formalisms makes training human annotators and subsequent data annotation unfeasible on a large scale, especially across languages. To mitigate this, we first introduce the Multilingual Semantic Layer (MSL), a conceptual evolution of previous formalisms, which decouples from disambiguation and external inventories and simplifies the task. MSL provides the necessary tools to encode the meaning across languages, paving the way for developing a high-quality semantic parsing dataset across different languages in a semi-automatic strategy. Subsequently, we manually refine a portion of this dataset and fine-tune GPT-3.5 to propagate these refinements across the dataset. Then, we manually annotate 1,100 sentences in eleven languages, including low-resource ones. Finally, we assess our dataset’s quality, showcasing the performance gap reduction across languages in Semantic Parsing.

## introduction one of the long-term goals of ai is to enable machines to comprehend human text in any language. at the core of natural language understanding (nlu) lies the task of semantic parsing (sp), aiming to convert text into machine-interpretable representations. although large language models (llms) have advanced significantly in understanding human text, semantic representations remain crucial for various applications. these range from chatbots and virtual assistants like amazon alexa, apple siri, and google assistant, as underscored in @xcite , to more specific tasks, e.g., sql queries @xcite or machineinterpretable commands @xcite . achieving proficiency in sp involves enabling machines to understand semantic relations between concepts in any language. over the years, numerous sp formalisms grounded in various linguistic theories have emerged, such as the discourse representation theory @xcite , prague dependency treebank @xcite , universal conceptual cognitive annotation @xcite or abstract meaning representation @xcite , among others. graph-based representations -e.g., amr -have attracted most of the attention since they can act as an interface that is comprehensible for humans and interpretable for machines (see figure 1 ). the objective of graphbased representations is to encode the meaning of language into a directed acyclic graph, where nodes represent concepts and edges represent semantic relations between concepts. however, even if these formalisms are able to encode the meaning in english effectively, they still struggle to scale to other languages @xcite . over the years, multiple attempts have been proposed to encode other languages: from adaptations based on language-specific repositoriese.g., spanish amr @xcite with ancoranet @xcite or the chinese amr @xcite with chinese propbank @xcite -to interlingua formalisms, such as the uniform meaning representation (gysel et al., 2021, umr) or the babelnet meaning representation @xcite . however, the main challenge persists: the scarcity of annotated data keeps these formalisms in the realm of abstraction, since annotated data is crucial for training parsers. this scarcity is primarily attributable to the high costs and complexity of manual annotation. annotators must thoroughly understand each specific language's rules, the formalism and the meaning repository, making the process highly labour-intensive. moreover, even though some projects have generated annotated data for these formalisms -like amr 3.0 @xcite , bmr 1.0 @xcite , or umr @xcite ) -these datasets are english-oriented and hindered by paywalls or restrictive licenses. furthermore, even though there have been efforts to annotate non-english languages -like umr 1.0 @xcite , turkish @xcite , persian @xcite , portuguese @xcite , or vietnamese (linh and nguyen, 2019) -they have not produced more than 200 examples per language, which is inadequate for effective parser training. this situation underscores a critical bottleneck in employing these formalisms for wider linguistic applications. this paper introduces a practical solution for tackling data scarcity, which can scale across different languages. firstly, we propose the multilingual semantic layer (msl), an evolution of previous formalisms, which acts as the necessary tool to enable data generation across languages. secondly, by leveraging this novel representation, we develop a methodology for semi-automatically generating a vast amount of high-quality annotations across languages by making use of llms (under an academic budget). lastly, we have annotated a gold standard dataset of 1,100 graphs across eleven languages, including low-resource ones, allowing us to assess the quality of our dataset. ## related work even if multiple formalisms have been proposed over the years, just a few of these provide annotated data. discourse representation theory @xcite provides a framework for representing the meaning of entire texts at the discourse level, which can be directly converted into logical forms. the parallel meaning bank @xcite is the most recent dataset under the drt formalism, which incorporates word senses from wordnet @xcite , and semantic roles from verbnet @xcite . however, the multilingual annotations use the english graph as the interlingua by automatically projecting the english annotations to other languages. moreover, the number of annotations is insufficient. prague dependency treebank (hajič, 1998, pdt) is a multi-layer formalism that defines a syntactic analysis for the sentence. the tectogrammatical layer @xcite covers semantic distinctions, such as the predicateargument structures, word senses or co-references. the corpora are in czech and english and rely on language-specific repositories. away from word order and from the actual words. ## our contributions in this section, we overcome the data scarcity in sp by providing the novel multilingual semantic layer (msl) -with the necessary tools to encode the semantic relations between concepts across languages (section 3.1) -and a multilingual dataset with millions of high-quality annotations, including manual annotations, that enables sp in different languages (section 3.2). ## potential and future of msl in the era of llms, we aimed to revamp the dream of structuring natural text across multiple languages through the use of msl. this work opens up new possibilities for applying graph-based and big-data analyses to multilingual content. we envision the potential of msl as follows: ## conclusion this paper addresses the challenge of data scarcity in multilingual sp. given the complexity and cost of training annotators for large-scale tasks, we introduce i) the msl layer and ii) provide a large, high-quality corpus across languages. the layer decouples sp from word sense disambiguation and entity linking, focusing on the extraction of sentence-level semantic relations. then, we automatically develop a preliminary silver dataset across languages, refining it through manual corrections and propagating these annotations using fine-tuned gpt llms. we also manually annotated 1,100 sentences in 11 languages, including low-resource ones, to demonstrate the flexibility of our layer and dataset over previous ones. our code and dataset are available at https://github. com/sapienzanlp/msl .