Non-directive corpus annotation to reveal individual perspectives with underspecified guidelines: the case of mental workload (2025)

Abstract: This paper investigates personal perceptions of mental workload through an innovative, non-directive corpus annotation method, allowing individuals of diverse profiles to define their own dimensions of annotation based on their personal perception. It contrasts with traditional approaches guided by explicit objectives and strict guidelines. Mental workload, a multifaceted concept in psychology, is characterized through various academic definitions and models. Our research, aligned with the principles of the perspectivist approach, aims to examine the degree to which individuals share a common understanding of this concept when reading the same texts. It seeks to compare the corpus produced by this non-directive annotation method. The participants, mainly employees of a large French enterprise and some academic experts on mental workload, were given the freedom to propose labels and annotate a set of texts. The experimental protocol revealed notable similarities in labels, segments, and overall annotation behavior, despite the absence of predefined guidelines. These findings suggest that individuals, given the freedom, tend to develop overlapping representations of mental workload. Furthermore, they demonstrate how non-directive annotation can uncover shared and diverse perceptions of complex concepts like mental workload, contributing to a richer understanding of how such perceptions are constructed across different individuals.

## introduction defining the scheme and guidebook is a crucial step in any text annotation process. while uniform guidelines facilitate achieving consensus and homogeneity, they can also mask variability in annotators' perspectives. however, when a target phenomenon refers to complex concepts, it becomes particularly important to consider and analyze the multiplicity of viewpoints. therefore, adaptations to this methodology are necessary to avoid framing annotators and to account for the multiplicity of perspectives. perspectivism @xcite offers a reflection on this question and discusses the consequences on annotated data used for machine learning in nlp. our research aims to assist in designing a language processing tool that can detect elements related to mental workload in employee messages within companies. mental workload is a critical phenomenon, as it represents a major concern that affects many aspects of workplace wellness. in the workplace, mental workload directly influences employee wellbeing, productivity, and overall job satisfaction. as a result of various media efforts to popularize the concept, mental workload has become a common notion frequently referenced in the workplace. our goal is to develop a more grounded mental workload model that can be compared to current academic ones. subsequently, this model will be used in the analysis of text messages. as demonstrated by @xcite , individual perception plays a crucial role in mental workload. this concept relates to how individuals perceive tasks and their environment. therefore, exploring methods to account for the diversity in the perception of this phenomenon during annotation is particularly important. by incorporating diverse points of view, we can ensure that no important aspects are overlooked in developing a more grounded model of mental workload. our aim is to explore the possibility of using a personal and individual text annotation method at an early stage. to investigate this, we asked people working in different positions within the same company, as well as academic experts on mental workload, to freely express their points of view on a common multifaceted object by annotating a shared set of texts. therefore, we propose a kind of hyperperspectivist and non-directive approach, which is achieved without providing a specific definition of the target phenomenon or annotation guidelines. as a result, we constituted a collection of annotated data from participants with no prior annotation experience. this study investigates the hypothesis that there is a common representation, even partial, of mental workload existing among individuals. it can be observed and made explicit through text annotation. additionally, we hypothesize that significant variations also exist and, if formalized, they could contribute to a richer and more comprehensive view of the concept. based on this hypothesis, the study aims to address the following research questions: ## analysis and results to address the first research question concerning the similarity of individual representations of mental workload, we conducted a comparative analysis of the following components: labels, segments, relations between labels, segments, and user categories. ## conclusion and future work this study explored the subjective perceptions of mental workload through an innovative annotation method involving participants of diverse professional profiles, with different expertise related to the mwl concept. regarding the formulated research questions, we can state that firstly, text annotation can serve various purposes, notably the analysis of different points of view on mental workload. next, the analysis of the results showed that, even without clear instructions, people share a common representation of mental workload. finally, despite this convergence in the representation, we observed differences in user behavior based on their professional roles. by allowing annotators to define their own labels and freely annotate text segments, we captured a variety of perspectives on mwl. we observed similarities in the annotated segments, labels, and groups, indicating a common representation of mwl, as we anticipated. additionally, differences emerged, highlighting the influence of individual professional profiles on the perception of this topic. the comparison between experts' and non-experts' approaches allowed us to see differences in the process of identifying mwl elements in the text. therefore, we gained a better understanding of the non-expert analysis and potentially considered new candidate facets in the mwl concept by leveraging employees' knowledge of the workplace context. this holistic approach promotes richer and more representative annotation and can thereby improve models for analyzing and interpreting textual data. the collected data can be viewed as explicit, structured, exemplified, and tested individual models of the mwl. the same annotation protocol could be applied to topics beyond mental workload, enabling a more inclusive approach. since we recorded all the interviews and have a transcript of the participants' comments on their own actions (such as definition, reformulating labels, choice of segments, etc.) we possess an even richer dataset than what we have presented here, which requires further analysis and effort. as part of a larger project, we are now considering the development of an automatic annotation process for mental workload in the messages. however, we are currently at the stage of defining the annotation scheme. this innovative approach opens the way to a better understanding of the linguistic and cultural nuances that influence the assessment of mental workload, while emphasizing the importance of integrating different perspectives to enrich textual data analysis models.