The Corpora They Are a-Changing: a Case Study in I talian Newspapers (2021)

Abstract: The use of automatic methods for the study of lexical semantic change (LSC) has led to the creation of evaluation benchmarks. Benchmark datasets, however, are intimately tied to the corpus used for their creation questioning their reliability as well as the robustness of automatic methods. This contribution investigates these aspects showing the impact of unforeseen social and cultural dimensions. We also identify a set of additional issues (OCR quality, named entities) that impact the performance of the automatic methods, especially when used to discover LSC.

## introduction natural languages are de facto living entities always subject to change and evolution. the diachronic dimension of natural language has played a pivotal role in the history of linguistics. understanding and explaining why a community of speakers "speak" as they do is of primary importance to access one's cultural heritage and perspectives on the world. in recent years, the natural language processing (nlp) community has developed an interest in historical linguistics, and in particular in the study of lexical semantics change (lsc). previous work has investigated lsc using different approaches, including statistical tests over time period @xcite , supervised methods @xcite , count-based distributional approaches @xcite , sense-based methods @xcite @xcite , and neural language models @xcite @xcite @xcite @xcite . this has been possible thanks to two factors: increased availability of machine-readable texts covering different periods and increased processing capabilities. the use of computational models for studying lsc is not free from problems, however, as highlighted by @xcite . almost every computational model for lsc is grounded on the distributional hypothesis of meaning according to which "the meaning of a word is its use" @xcite and the "difference in meaning correlates with difference in distribution" @xcite . distributional models are powerful, yet they suffer from some limitations, namely: (i) they require large amount of text; (ii) they are sensitive to the type of texts and the distribution (i.e., frequency) of the lexical items; and (iii) they tend to conflate different types of information and variables such as semantics, social and topical information. this contribution investigates two strictly connected aspects: the reliability of lsc benchmark data and the sensitivity of a state-of-the-art approach for lsc, grounded on the distributional hypothesis, when changing the source corpus. the results of our work will help to shed light on systems' robustness and stability by verifying whether methods tuned on one corpus can be directly applied to another. ## methodology to test benchmark independence and models' robustness for lsc, we design a set of experiments using two source corpora, a common benchmark, and a common architecture for lsc detection. the first corpus is the "l'unità" corpus @xcite . it covers a time span between 1945-2014 and it has been collected, pre-processed, and released for the diacr-ita (diachronic lexical semantics in italian) task @xcite , a lsc change shared task for italian. texts were extracted from pdf files by using the apache tika library foot_0 and pre-processed with spacy foot_1 for tokenization, pos-tagging, lemmatization, named entity recognition and dependency parsing. the second corpus was obtained by crawling a publicly available digital archive of the italian newspaper "la stampa". the corpus covers a shorter time period and it was pre-processed using the same tools and pipeline of "l'unità". each corpus is split into two sub-corpora, c 1 and c 2 , covering different time periods. ## testing for robustness and independence testing for robustness and consistency for lsc is not trivial since it requires to distinguish between two strictly connected dimensions: (i) reliability of the benchmark (dataset dimension), and (ii) variations in data distributions (corpora dimension). the first dimension (dataset) is analysed by comparing on the diacr-ita benchmark the performances of the same model trained on the two corpora. the corpora dimension is investigated by manually inspecting the disagreements between the model predictions. all 18 target words in the benchmark satisfy a minimal frequency threshold of 10 both in c 1 and c 2 in "la stampa", allowing us to reliably compare the results. to study the reliability of the diacr-ita benchmark with respect to the underlying corpus, for each target word in the benchmark, we computed the cosine similarity of its embedding representation from each sub-corpus (c 1 and c 2 ). to account for the random initialisation of the op-sgns parameters, we ran 10 experiments with different initialisations and averaged the results. the system accuracy is computed as the fraction of correctly predicted words over the total number of words in the benchmark. a target word is deemed as an instance of lsc when its cosine similarity across the two time periods is below a given threshold λ * . since the focus is on the reliability of the benchmark across corpora, and not the system performances, the threshold λ * for each corpus is set up to the value that maximises the system performance on the corpus. using the optimal threshold, our implementation of op-sgns obtained an accuracy of 0.96 ± .02 when trained on "l'unità" and 0.83 ± .00 when trained on "la stampa", a difference spawned by the incorrect classification of the words ape (lsc), rampante (lsc), and brama (stable). to understand the role of the two corpora, we compare the target word similarities between c 1 and c 2 on the two corpora. figure 1a and figure 1b illustrate the similarities of the stable and lsc target words, respectively. overall, the identification of lsc target words seems consistent among the two corpora, and lets us assume that the benchmark is reliable and the algorithm is robust. we further analyse the system's disagreements by manually exploring their occurrences in each corpus for every time period. 3 for the target brama ('yearning'), "la stampa" indicates a potential lsc. the manual inspection, however, has confirmed the annotation in the benchmark (i.e., a stable meaning) showing that the change is triggered by the presence of this word in band names in the c 2 portion of the corpus. ape ('bee') is listed in the benchmark as an lcs, since in c 2 it refers not only to the insect, but also to a three-wheeled vehicle. despite this new sense is present in the c 2 sub-corpus of "la stampa", the difference in similarity is above the threshold. interestingly, in this corpus we observe the three-wheeled vehicle sense also in c 1 , especially as part of paid advertisements. this points to a bias in the corpus (i.e, "l'unità") used to create the benchmark, namely the lack of (or extremely limited) presence of advertisements, which has obfuscated the occurrence of the three-wheeled vehicle sense and suggested ape as a good candidate for an lsc. ape is interesting also for another reason: the discrepancy between when it was first on the market (1948) and its first attestation in the sabatini coletti dictionary @xcite . further related to the more commercial nature of the "la stampa"newspaper is the higher difference in similarity with respect to the "l'unità" for the word rampante ('rampant'/'high-flying'). in "la stampa", the word occurs also in c 1 as part of the book title "il barone rampante"; this has mitigated the variation in context of usage with the occurrences of rampante in c 2 . ## models into the wild we further extended the analysis to the whole common vocabulary of the two corpora to test the ro-bustness of the computational model. in particular, we consider the vocabulary intersection v of the two sub-corpora, that consists of 48,681 lemmas. then, we compute the two sets x and y of cosine similarities for all the words in v . a first analysis was conducted to understand to which extent the rank order of the two sets x and y are correlated. the spearman correlation between the two sets is 0.67 (p-@xmath0.01), which indicates a positive correlation between the two rank orders, suggesting that the output of op-sgns is similar across the two corpora. the plots of the correlations are reported in figure 2 in appendix b. in this analysis, the optimal thresholds cannot be computed due to the lack of a gold-standard for the whole vocabulary intersection v . potential lsc instances are identified by using as threshold the difference between the average of the cosine similarities (µ) and the standard deviation (σ) over the set v : where t i is the term associated with the i th similarity x @xmath1. similarly, we compute the set lcs(y ). the intersection of the two lcs sets consists of 2,283 lemmas. a quick inspection of the proposed lscs immediately triggers observations concerning two aspects: (i) the well formedness of a lemma; and (ii) the presence of named entities (nes). by well formedness, we refer to the lemma being an actual word attested in a reference dictionary of italian. indeed, some of the lemmas with the lowest similarity scores, e.g., gaucha, bwa, bill, -anche, do not appear to be well formed italian words. reasons for this are to be found in the quality of the digitized versions of the documents of the two corpora, the presence of foreign words (e.g., frere, french for 'brother'), as well as tokenization errors of the pre-processing tool. we use the list of lemmas in the sabatini coletti dictionary to filter out all of these entries. nes appear to be an additional source of noise. lemmas like albertarelli, beraudo, napoleoni, armellini, are all instances of nes referring to people's surnames. we automatically filter nes in two steps: (i) for each word in a sequence tagged as ne by spacy, we retrieve and store separately the corresponding lemma; (ii) every candidate lsc lemma is matched against the list generated in (i), greedily filtering all lemmas found to be part of a ne. the manual exploration of the contexts of occurrence in both corpora of the 50 lemmas showed that only four of them (8% of the total sample) can be considered correct examples of an lsc. two of them, palmare ('obvious'/'palmar'/'hand-held computer') and patteggiare ('negotiate'/'plea'), are also attested in the sabatini coletti. the remaining two, handicappare ('to handicap') and orgasmo ('orgasm'), indicate a change of use rather than an actual change of meaning. in particular, handicappare, and namely its participial form, was used dur-ing the 80s/90s to refer to people with disabilities, extending the initial meaning in c 1 of "to assign an handicap to a team". the use of the word with this meaning is now derogatory and it is not attested in the dictionary. on the other hand, orgasmo was used in c 1 in its figurative meaning of great or extreme anxiety, e.g. "nell'orgasmo del momento" ('in the excitement of the moment'). on the other hand, in c 2 is used with reference to sex and sexuality. three additional lemmas are signalled as lexical changes: pula, doc, and tac. however, they are officially attested as different lemmas in the sabatini coletti, thus implying homonymy. all remaining entries are false positives being either nes or ocr errors. for the nes, these are cases where the ne also corresponds to a lemma in the reference dictionary. a good example of this is borsellino. in c 1 , both corpora present context of use with the dictionary meaning of "a small purse". however, in c 2 , the contexts of use refer to the judge paolo borsellino foot_3 , killed in a terrorist attack by the mafia. nes introduce additional challenges while constructing a benchmark for lsc, especially when they are homonyms with common nouns. a viable solution to this problem would be to detect and disregard from the corpus those entities that are homonyms of common nouns. this also calls for the development of more robust systems for ne detection: besides our efforts at filtering nes, lots of them have remained as potential targets of lsc. ## conclusion and future work this contribution has tested the reliability of the diacr-ita benchmark for lsc when the underlying corpus used to train and detect lscs varies. furthermore, it has scrutinised the robustness of the lscs, detected by a common algorithm, across different corpora. although preliminary, our results indicate that: (i) social and cultural dimensions must be carefully considered when creating lsc benchmark since potential positive examples may be biased; (ii) current approaches to unsupervised lsc are sensitive to the used corpora; (iii) quality of the data (i.e., ocr rendering) and the presence of nes, especially homonyms with common nouns, are major sources of errors when such automatic methods are applied to actively discover cases of lsc. strictly connected to this latter aspect is the hiatus between the results of the algorithm against the benchmark and its use "in the wild". this calls for the development of different and more realistic evaluation protocols for unsupervised lsc and research programmes to address the availability of high quality, distributable diachronic corpora. besides these limitations, the use of lsc methods on sources with clear differences along social, political, and cultural dimensions could promote a cross-fertilisation of disciplines. as future work, we plan to extend our analysis to both other corpora and languages, as well to other lexical change detection algorithms, in order to confirm the validity of our findings. the initial learning rate is set to 0.025, with a negative sampling of 5 and a context window size fixed to 5. (a) l'unità -la stampa (b) la stampa -l'unità figure 2: correlation plots.