Adapting Measures of Literality for Use with Historical Language Data (2024)

Abstract: This paper concerns the adaptation of two existing computational measures relating to the estimation of the literality of expressions to enable their use in scenarios where data is scarce, as is usually the case with historical language data. Being able to determine an expression’s literality via statistical means could support a range of linguistic annotation tasks, such as those relating to metaphor, metonymy, and idiomatic expressions, however making this judgment is especially difficult for modern annotators of historical and ancient texts. Therefore we re-implement these measures using smaller corpora and count-based vectors more suited to these amounts of training data. The adapted measures are evaluated against an existing data set of particle verbs annotated with degrees of literality. The results were inconclusive, yielding low correlations between 0.05 and 0.10 (Spearman’s ρ). Further work is needed to determine which measures and types of data correspond to which aspects of literality.

## introduction though it is usually taken as a given that an expression's 'literal' meaning is readily identifiable, the notion is more ambiguous and complex than it initially seems. as @xcite explains, there are several different senses that may apply when we describe something as 'literal'. when an expression is used in a literal sense, we might mean that it is used to mean the thing it usually means, whether relative to the broader language community or to a narrower domain-specific lect. alternatively, we might mean that the meaning of the whole expression corresponds to a systematic combination of its constituent parts, i.e. that the expression's meaning is compositional. or we might be saying something about what an expression is not doing: it's not an instance of irony, metonymy, metaphor, or any other sort of context-dependent extension of its minimal meaning. yet generally, in linguistic research, we tend to regard these as being equivalent. it makes a difference which definition you choose and how you regard literality: @xcite annotators were presented with annotation guidelines highlighting one of lakoff's definitions of literality and asked to judge the literality of a range of expressions. the result was that some expressions were judged as having significantly different properties, depending on the literality definition in use. there are a range of semantic annotation tasks which, implicitly or explicitly, require the annotator to make a judgment as to the literality of an expression in a given context: this could be the case in compositionality annotation of multiword expressions (mwes) or the annotation of idioms or metaphors, which must be distinguished from what they are not: literal usages. yet, when annotators are working with historical language data, the ambiguity of these distinct variations of literality, which could otherwise be mitigated with well-written annotation guidelines, is coupled with a lack of linguistic intuition, which is what semantic annotation tasks usually rely on. in the larger research project in which this paper is situated, it is our aim to develop statistical measures to estimate the degree of literality of particular usages of a given expression in order to support annotators by giving them tools to compensate the lack of linguistic intuitions for historical language varieties. to this end, here we: ## related work the theoretical descriptions of literality outlined above highlight multiple aspects which characterize it: the literal meaning is the one that is conventional or at least typical for that expression, or its the one that is involved when an expression is read compositionally, or its the minimal meaning, which an expression is thought to have devoid of any context, its context-free interpretation. previous approaches have tended to focus either on an expression's compositionality or its conventionality when measuring the degree to which it is literal or non-literal. much of the computational work that deals with the measurement of degrees of compositionality and literality focuses on particular classes of mwes. most commonly these are nominal compounds, as in schulte im walde et al. (2013), weeds et al. (2017), and cordeiro et al. (2019) , but others address compositionality in mwes more broadly, as in @xcite or @xcite . in general, an expression is taken to be compositional to a greater degree when the semantics of the whole are more similar to a systematic combination of the expression's parts. the composition function is usually an additive model @xcite , which assigns a weight to each vector before adding them together. this approach to compositionality requires a representation for both the expression as a whole as well as for the parts individually, so it tends to work best with a fixed inventory of expressions to be analysed, since you need to know which expressions to combine before training a semantic representation ahead of time. since our data set doesn't concern a fixed set of expressions, we plan to address the aspect of compositionality in future work. there are a range of studies which deal with the annotation and classification of figurative language which are also relevant, such as those concerning metaphor, metonymy, irony, and idioms. in all of these cases, 'literal' is defined negatively, as the class not sought, the normal and default assumption. as such, the features that would characterize literality are not modelled directly. the set of studies that come closest to providing an account of literality itself are those that concern idiomatic expressions. many of the features discussed in these studies reflect the notion that literal mentions are somehow 'typical' of the expression in question and non-literal mentions are thus 'atyp-ical'. they differ in how this typicality is modelled. @xcite use an unsupervised approach that applies the notion of lexical cohesion, operationalizing this as the mean similarity of the terms within a particular window. the candidate expression is then removed and the cohesion is calculated without it. when the level of cohesion increases upon removal of the candidate expression, it suggests that this is an atypical context for that expression, and the instance is classified as idiomatic or non-literal. @xcite describes an extension of this approach, which replaces the "normalized google distance" of the original, which has a very low degree of reproducibility, with similarities between word vectors. this approach forms the basis for the cohesion measure we employ below. @xcite also address the identification -or rather the characterization -of idioms: their study is more of a characterization because they consider idioms to exist on a spectrum together with novel metaphors, collocations, and ordinary literal language. thus it isn't a distinct class as a classification task would suggest. all of these expressions can be related to one another along two orthogonal axes of conventionality and contingency, where conventionality describes the extent to which words are used in their "usual" or "typical" sense, and contingency refers to the tendency for words to be used in a particular, fixed context. the dimension that broadly differentiates literal and non-literal usages is that of conventionality. the conventionality measure that we adapt in this work stems from this study. ## experimental results for the 155 lemmas in the annotated data set, we calculated cohesion and conventionality with each of the three implementations for each of the ≈ 41.5 annotated instances, resulting in 38,556 instances total. of these, 1357 instances were omitted, either because the lemmas do not occur in the background corpus or because there wasn't sufficient context in the test sentence. for the remaining 37,199 combinations of sentence, lemma, measure, and implementation, we averaged the three raters' judgments together in order to compare them with the given value. for the sake of comparison, we also evaluate each of these measures in a setting as close as pos-sible to the original papers (*_orig). for cohesion, this means using pre-trained skip-gram word2vec vectors for the type vectors, and for conventionality, we use a pre-trained german bert model to encode each sentence, from which we retrieve a contextualized representation for each target token instance. ## discussion and conclusion comparing the values of the two measures in their various implementations to the human judgments in the literal/non-literal data set does not appear to reveal any reliable patterns. though in some cases the correlations are technically significant, the actual level of correlation is too low for either measure to be trusted in any particular case. the results suggest that these measures, implemented as described above, don't correspond, in general, to the notion of literality that the annotators had in mind. as a result, it is also impossible to say whether the adaptations of the measure for use with smaller corpora, such as for historical language varieties, were appropriate or whether they had any effect on the usefulness of the measures implemented. an analysis of the correlations of the various measures with the judges' ratings on a per-lemma basis was likewise inconclusive. as is evident in figure 1 , the correlations for particular lemmas can vary quite widely between strongly negative and positive correlations. we haven't been able to find a clear reason for this; there are no apparent tendencies towards higher correlations when a lemma has a greater proportion of non-literal instances, for instance. in future work, we plan to conduct more extensive annotation efforts specifically targeting literality in order to collect more fine-grained data to use in future experiments. ideally, such a data set will cover not just particle verbs, but all open-class lexemes, and we plan to formulate detailed guidelines that will improve reproducibility and reusability of the data set.