From Discrete to Continuous Classes: A Situational Analysis of Multilingual Web Registers with LLM Annotations (2024)

Abstract: In corpus linguistics, registers–language varieties suited to different contexts–have traditionally been defined by their situations of use, yet recent studies reveal significant situational variation within registers. Previous quantitative studies, however, have been limited to English, leaving this variation in other languages largely unexplored. To address this gap, we apply a quantitative situational analysis to a large multilingual web register corpus, using large language models (LLMs) to annotate texts in English, Finnish, French, Swedish, and Turkish for 23 situational parameters. Using clustering techniques, we identify six situational text types, such as “Advice”, “Opinion” and “Marketing”, each characterized by distinct situational features. We explore the relationship between these text types and traditional register categories, finding partial alignment, though no register maps perfectly onto a single cluster. These results support the quantitative approach to situational analysis and are consistent with earlier findings for English. Cross-linguistic comparisons show that language accounts for only a small part of situational variation within registers, suggesting registers are situationally similar across languages. This study demonstrates the utility of LLMs in multilingual register analysis and deepens our understanding of situational variation within registers.

## introduction language varies with context as people adapt their linguistic choices to different situations. register variation refers to the distinct forms of language functionally related to specific situations and communicative purposes @xcite @xcite . in the text-linguistic approach to register analysis, the frequent use of linguistic features is assumed to be directly functional for the requirements of the situation @xcite . as a result, text-linguistic register analyses typically start with situational descriptions of registers (e.g. @xcite . nevertheless, register studies have traditionally focused on the linguistic features characterizing different registers, and much less attention has been given to analyzing the communicative situations in which texts are produced @xcite . furthermore, existing situational analyses often describe entire registers using the same categorical characteristics-such as medium, setting, communicative purpose, interactivity, and topic @xcite -which are then used to define register categories. these resulting classes are typically assumed to be situationally discrete. some recent studies, however, have provided strong evidence for register-internal situational variation (e.g. @xcite @xcite , casting doubts on the possibility of defining registers by any essential situational attributes. this has led to a reconceptualization of registers as continuous rather than discrete categories-categories that can be recognized but not strictly defined by linguistic features or situational context @xcite . however, previous research on this variation has been limited to english texts, and its extent in other languages is largely unknown. in this study, we address this gap by adopting the continuous approach to situational analysis introduced by @xcite and applying it to a large multilingual register-annotated corpus. in this framework, texts are coded for 23 parameters that capture situational variables such as purpose, background assumptions, and source of information, using an ordinal scale from 1 to 6. these annotations allow texts to be viewed within a continuous situational space and grouped into new situational text types based on their proximity within this space. the coding scheme was designed to capture the full range of situational factors identified in previous studies, including @xcite , @xcite , and @xcite . to annotate a large multilingual corpus for its situational characteristics, we apply a new approach: instead of manually annotating the texts, we use multilingual large language models (llms) for the task. specifically, we utilize gpt-4o-mini (openai, 2024) and llama 3.1 8b (ai@meta, 2024) to annotate 8,406 texts from the registerlabeled multilingual core corpus @xcite in english, finnish, french, swedish, and turkish. we evaluate the llm-generated annotations against each other and against a humanannotated sample corpus, demonstrating that the llms achieve good accuracy. by integrating these situational annotations with the texts' existing register and language labels, we conduct multilingual analyses on the relationships between situational context and register, as well as cross-linguistic comparisons of situational variation between registers. our analyses show that while registers are partially distinguishable by their situational characteristics, considerable register-internal variation exists across all included languages. these findings align with those reported by @xcite for english. we identify six situational text types-"advice", "information", "marketing", "personal", "opinion", and "speech"-each characterized by specific contextual features. these clusters partially align with established register categories but more often reveal situational overlap between registers. moreover, our cross-linguistic comparisons show that language accounts for only a small portion of the total variance in each register, suggesting that the situational characteristics of registers are generally similar across languages. the code and data used in this study are available at https://github.com/turkunlp/ situational-analysis-llm . we start by describing the corpus and the llmbased annotation process, including an evaluation of the llm annotations against a human-labeled subcorpus. we then explore the situational variation within web registers and identify situational text types that emerge from the data. next, we examine how these text types align with traditional register categories. finally, we analyze crosslinguistic situational variation within registers. ## the register-annotated core data we utilize data from the multilingual core corpus @xcite , a large manually register-annotated collection of unrestricted web for the english core, data was collected through google searches targeting highly frequent english 3-grams @xcite , and annotations were performed via amazon mechanical turk, where each document was labeled by four coders, with a label assigned if at least two coders agreed. the finnish corpus was sourced from a random sample of the finnish internet parsebank @xcite . the remaining subcorpora were derived from common crawl data, following the methodology described in @xcite , including steps such as sampling from various time periods, removing boilerplate content, and deduplication. all register annotations were made by trained experts, using a hierarchical taxonomy with 9 main categories and 16 subcategories. in this study, we focus on the five largest language datasets in multilingual core: english, finnish, french, swedish, and turkish. we include 13 registers, listed in ## situational annotation using llms large language models (llms) @xcite have emerged as powerful tools for textual analysis and annotation, with some studies suggesting that their accuracy can even surpass that of human annotators (e.g. @xcite @xcite . in this study, we experiment with two recent models-gpt-4o-mini and llama 3.1 8b-for the situational coding task. we access gpt-4o-mini via the openai api and deploy llama 3.1 8b on the mahti supercomputer (csc -it center for science ltd), using it with pytorch through the huggingface transformers library. for both models, we set the temperature to 0.01 for consistent responses. we use the two llms to code the 8,406 documents for 23 situational parameters, as listed in figure 1 . for each text, we provide the first 5,000 characters as input to the models, along with the system prompt provided in appendix a, which instructs the models to rate each parameter from 1 (completely disagree) to 6 (completely agree) based strictly on the given text. we also ask the models to briefly explain each scoring decision, which, in preliminary tests, significantly improved both models' performance. both models generated the data in the requested output format without any issues. we compare the inter-annotator agreement (iaa) of the llm-generated annotations with each other and with a human-annotated sample consisting of 150 documents across all five languages, annotated by multiple human coders. the human annotators, all experts in the core label scheme, were given the parameters and documents without any additional guidance on how to annotate them and without being shown the texts' register labels. the results of these iaa evaluations are presented in the agreement between the two llms is strong across both the full dataset (8,406 documents) and the human-annotated subset (150 documents), with cohen's kappa values of 0.72-0.73 and pearson correlations of 0.75-0.76. in comparison, the agree1 2 3 4 5 6 score a spoken transcript lyrical or artistic pre-planned and edited interactive is an expert focuses on themselves assumes technical background knowledge assumes cultural or social knowledge assumes personal knowledge about themselves to narrate past events to explain information to describe a person, place, thing or idea to persuade the reader to entertain the reader to sell a product or service to give advice or recommendations to provide "how-to" instructions to express opinions common knowledge direct quotes factual or scientific evidence opinion personal experience the text is: the author or speaker: the purpose of the text is: the basis of information is: ment between the llms and human annotators is moderate (kappa 0.43-0.50, pearson's r 0.48-0.56), but these scores are similar to those reported by @xcite for human-made annotations of the same 23 parameters. gpt-4o-mini proves to be slightly more reliable than llama 3.1 8b when compared to human annotations. figure 1 shows the means and standard deviations of the parameter scores from the two llms and human annotators. the means of most parameters are relatively close, and all parameters show similar dispersion. this suggests that the moderate-to-strong iaa scores are not simply due, for instance, to the llms uniformly selecting the same scores across the dataset. overall, our results demonstrate that llms are well-suited for this annotation task. in the following sections, we use the dataset of 8,406 texts annotated for the 23 situational parameters, register, and language, to conduct a series of analyses. we start by evaluating how well registers are defined by their situational characteristics. then, we identify distinct situational text types directly from the situational data and compare these data-driven categories to traditional register classifications to assess their alignment. finally, we examine register-internal variation across languages. ## conclusion this study explored the situational variation of web registers across multiple languages by utilizing llm-generated situational annotations alongside manual register labels. analyzing 8,406 texts in english, finnish, french, swedish, and turkish, we identified six situational text types-such as "advice" and "opinion"-that cut across the traditional register categories in the dataset. our findings indicate that while some registers correspond to specific situational clusters, there is significant variation within registers, supporting the view that registers are better described as situationally continuous rather than discrete. cross-linguistic comparisons further suggest that situational variance within registers is more influenced by internal variation than by language differences, implying that registers are similarly varied across languages rather than distinctly different. the successful use of llms for annotation in this study demonstrates their potential in corpus-linguistic register studies.