Label Bias in Symbolic Representation of Meaning (2025)

Abstract: This paper contributes to the trend of building semantic representations and exploring the relations between a language and the world it represents. We analyse alternative approaches to semantic representation, focusing on methodology of determining meaning categories, their arrangement and granularity, and annotation consistency and reliability. Using the task of semantic classification of circumstantial meanings within the Prague Dependency Treebank framework, we present our principles for analyzing meaning categories. Compared with the discussed projects, the unique aspect of our approach is its focus on how a language, in its structure, reflects reality. We employ a two-level classification: a higher, coarse-grained set of general semantic concepts (defined by questions: where, how, why, etc.) and a fine-grained set of circumstantial meanings based on data-driven analysis, reflecting meanings fixed in the language. We highlight that the inherent vagueness of linguistic meaning is crucial for capturing the limitless variety of the world but it can lead to label biases in datasets. Therefore, besides semantically clear categories, we also use fuzzy meaning categories.

## motivation natural language is a very powerful way of describing the world. communication using natural language is remarkably efficient because it allows the use of a finite grammar and lexicon to describe a potentially infinite set of situations, knowledge, emotions (i.e. content, as we will simplistically refer to the communicated reality in this paper). the means of language have many meanings. the meanings expressed may be relatively vague in relation to the content being described. the properties of natural language, such as ambiguity or vagueness, therefore pose challenging problems for symbolic representations of meaning. the research question we tackle in this contribution can be illustrated by the examples (1)-( 7 ). how can we describe the meanings of the highlighted expressions in examples (1)-( 7 )? one may simply state that, in all examples, some circumstance of john's working is expressed and to use one very coarse-grained category "circumstance" for all expressions (cf. a single label adverbial in the universal conceptual cognitive annotation project @xcite ). however, it is clear that the circumstance in (7) is semantically considerably distinct from the circumstances expressed in (1)-(6). it seems that a finer distinction into spatial and, let's say, "broad manner-related" circumstances would be more appropriate. but it is also evident that the circumstances in (1)-(6) differ. some more significantly, some less so. are to work with a chisel and with wood the same semantic category? should a semantic classification distinguish between with a smile and without a smile? the question posed in this paper is: what granularity should semantic classification have, and, more importantly, what should determine this granularity? this also raises a question for linguistic annotation: on how fine-grained categories can human annotators agree? ## introduction meaning representation has long been an important task in computational linguistics, yet it remains challenging for both machines and human annotators. new or extended symbolic representations of meaning are continuously being proposed (e.g., uniform meaning representation (umr; @xcite , abstract meaning representation (amr; @xcite , universal conceptual cognitive annotation (ucca; @xcite , deep universal dependencies @xcite , parallel meaning bank @xcite ). meaning representation (semantic role labelling, word sense disambiguation) is typically modelled by means of a dictionary or pre-defined set of meaning categories, and a meaning is then captured through the best-fitting label from this set. most of these approaches have a primary focus on verbs with varying degree of elaborate classification of the verb participant semantic roles (e.g., verbnet @xcite , framenet @xcite , propbank @xcite , pdt-vallex @xcite , synsemclass @xcite ), and there are also broader databases for word senses in general, such as wordnet @xcite , ontonotes @xcite . relatively few frameworks have focused on comprehensive accounts of non-participant (adjuncts, adverbials, circumstants) roles, though they are very frequent and contribute crucial semantics to sentences. in this respect, we have to mention the xposition project (or snacs -semantic network of adposition and case supersenses; @xcite , which focuses on the semantics of prepositions and it is relatively close to our project. in this project, 52 so-called supersenses are distinguished and organized into a multi-level hierarchy. at the highest level, circumstances, participants, and configurations (noun attributes) are differentiated. the set of labels is partially up to three levels deep, but in terms of expressed meaning, it is relatively coarse-grained. this contribution aims to critically consider the trend of building semantic representations, highlighting its challenges, and limitations in addressing the following issues in the task of semantic classification of circumstances (outlined in sect. 1): our semantic classification is developed within the prague dependency treebank (pdt) framework @xcite . the description of circumstantial meanings is based on a large volume of real examples that pdt corpora provide and the proposal is subsequently used to enrich the semantic annotation in these corpora (for the upcoming release in 2026). we support our approach with a pilot annotation and evaluate the results. the paper is organized as follows: in sect. 3, based on the analysis of recent projects dealing with semantic annotation, we discuss key points of meaning representation: description models (sect. 3.1), granularity of semantic roles (sect. 3.2), and consistency and reliability of annotation (sect. 3.3). in sect. 4, we describe our project on the semantic classification of circumstants within the pdt framework, applying these key points. the annotation experiment is presented in sect. 5. our position and findings are summarized in sect. 6. supportive material is provided in appendix a. ## meaning representation key points in the semantic representation projects, labels are determined more or less intuitively (often without any apparent underlying theory), which results in varying granularity both within a single classification and across different semantic representation systems. different degrees of granularity and (dis)arrangement of categories, as well as their (un)clear definition, influence the reliability and consistency of annotated data. we are aware of the complexity (and unresolvability) of these issues, but we believe that it is important to raise and explore them, seeking guidance toward their solution. ## prague dependency treebank we develop our semantic classification of circumstants within the prague dependency treebank (pdt) project. the pdt framework is unique in its attempt to systematically include and link different layers of language including a semantic representation at deep syntactic annotation layer called tectogrammatical. regarding the current trend in the development of semantic representations in the field of computational linguistics, it should be highlighted that in the latest version pdt-c 2.0 @xcite , there is a large amount of genre-diversified data (more than 3 million tokens) manually annotated with an interlinked semantic, syntactic, and morphological annotation. the annotation scenario of pdt is based on the original, well-developed theory of language description, so-called functional generative description (fgd; @xcite and was reflected in several detailed annotation manuals available from the project web site. foot_3 ## label bias experiment the position described in sect. 4 is supported here by a brief annotation experiment. foot_4 ## conclusion this paper puts under scrutiny the annotation of circumstantial meanings in the prague dependency treebank, addressing challenges in meaning representation. our approach centres attention on the intricate relation between language and the world it describes, emphasizing the need for a classification system that accommodates both clear-cut and vague meanings. our two-level classification balances broad semantic concepts with fine-grained distinctions, reflecting linguistic meaning. we introduce fuzzy meaning labels for cases where rigid classification fails. an annotation experiment confirms this perspective, showing varying levels of annotator agreement, from unanimous to none. by incorporating fuzzy labels and multiple annotations, we enhance the precision and explanatory power of semantic descriptions. ongoing development within the prague dependency treebank will further refine and extend this framework. description of language is far from complete.