A Multilingual Benchmark to Capture Olfactory Situations over Time (2022)

Abstract: We present a benchmark in six European languages containing manually annotated information about olfactory situations and events following a FrameNet-like approach. The documents selection covers ten domains of interest to cultural historians in the olfactory domain and includes texts published between 1620 to 1920, allowing a diachronic analysis of smell descriptions. With this work, we aim to foster the development of olfactory information extraction approaches as well as the analysis of changes in smell descriptions over time.

## introduction human experience is mediated through the senses, which we use to interact with the world. since the perceptual world is so important to us, all languages have resources to describe the sensory perception. nevertheless, previous research showed that, at least in western european languages, the visual dimension is prevalent in language, with a richer terminology used to describe it, while the olfactory dimension is less represented @xcite . for example, in english, there are less unique words for the smell domain than for the other senses. they are also used less frequently and olfactory descriptions are often a target of cross-modal expressions. sensory terminology has been researched previously, with the goal to build resources and to analyse how the different senses are described in language @xcite . some research is specifically devoted to smell @xcite , but they all focus on contemporary language. one notable exception is the collection of essays published in @xcite , where olfaction in different languages is analysed in a diachronic perspective. for example, strik lievers (2021) describes how the olfactory lexicon has changed from latin to italian. in this work, we contribute to the diachronic analysis of olfactory language by annotating a multilingual benchmark with smell situations spanning three centuries. compared to existing studies, our focus is not on the occurrences of single terms, but we rather capture smell events in texts, i.e. more complex structures involving different participants. the benchmark currently covers six languages (dutch, english, french, german, italian and slovene). annotation of latin data is ongoing, but we do not include here the results for this language because they are still preliminary. we describe the annotation guidelines and the document selection process. our benchmark includes texts issued between 1620 and 1920 covering ten domains of olfactory interest to cultural history. we release the benchmark at https://github.com/odeuropa/ benchmarks_and_corpora and we present a first analysis of its content. ## related work studies on olfactory language in cognitive science primarily focus on the verbal expressions of the odour perceived @xcite , while in historical studies, instead, they mainly deal with the textual accounts of experienced smells, as in @xcite . within the nlp community, little attention has been devoted to the automatic analysis of smell references in texts. most works have focused on the creation of lexical databases, for example @xcite worked on the creation of sensicon, representing the first systematic attempt to build a lex-1 icon automatically associated to the five senses. other studies have focused on synaesthetic aspects of language, dealing with the multisensoriality of sensory words. for instance, lievers and huang (2016) create a controlled lexicon of perception, while @xcite propose to use word embeddings for the extraction of sensory descriptors and their interconnections in texts. as regards smell-specific works, @xcite propose both a simple annotation scheme to capture odour-related experiences and two semi-supervised approaches to automatically replicate this annotation. @xcite present an automated analysis of wine reviews, where olfaction plays a fundamental role, while @xcite introduce an approach to automatically identify smell-related sentences in a corpus of historical medical records using distributional semantic modelling. more recently, @xcite present framenet-inspired guidelines to annotate smell events in texts. we consider this work the starting point upon which we build our annotation task. in particular, we aim at assessing the underlying assumptions of such guidelines: whether frames can be applied diachronically and across languages using the same annotation scheme. ## annotation guidelines annotation of olfactory events and situations in texts is a new task that was recently introduced in @xcite . we adopt the same framework in this work, whose guidelines are summarised below. olfactory annotation is inspired by the framenet project @xcite 1 which, focusing on the semantic dimension of situations and participants, should be easily applicable to multiple languages and constructions. in framenet, events and situations are so-called frames and are used as synonyms for schemata, semantic memory or scenarios. they represent the components of the internal model of the world that language users have created by interpreting their environment @xcite . according to frame semantics, a frame includes two main components: lexical units (lus) and frame elements (fes). the former are words, multiwords or idiomatic expressions that evoke a specific frame, while the latter are frame-specific se-1 https://framenet.icsi.berkeley.edu mantic roles that, in case of verbal lus, are usually realized by the syntactic dependents of the verb. for example, the commerce pay frame includes as lexical units 'pay', 'payment', 'disburse', 'disbursement', 'shell out', and has the following frame elements: buyer, goods, money, rate, seller. while framenet aims to be a general-purpose resource, the guidelines we follow only concern olfactory situations. therefore, the scope of our annotation considers only smell-related lexical units and a single frame of interest, the olfactory event. the same structure as the original framenet is adopted based on lexical units and related frame elements. when necessary, domainspecific semantic roles are introduced upon discussions with experts in olfactory heritage and history. for example, the roles smell source, evoked odorant and odour carrier were not originally in framenet, while some generic roles such as perceiver, time, location and circumstances are borrowed from the original resource. an overview of the frame elements included in our annotation is shown in the list of lexical units (lus) was defined with the help of domain experts, choosing smell-related lexical units that evoke olfactory situations and events. the lu lists were created in six languages, namely english, dutch, italian, french, german and slovenian. they include basic smell-related terms, which are generally comparable across languages (for instance the translation of words such as 'to smell', 'odour' 'odorous', 'smelly', 'perfume'). the lists were extended with languageand culture-specific terms, such as german compound nouns created with the roots '-gestank' and '-geruch', e.g. regengeruch ('rain smell') or viehgestank ('cattle stink'). the initial version of the list is reported in we consider these guidelines appropriate for our task because they have been designed following a multilingual perspective, with no languagespecific adaptations. furthermore, as we annotate documents from different time periods, lu lists are not fixed, giving the possibility to add new items as the outcome of the annotation process. ## quality control we implement two quality control measures: 1) a web-based consistency checker, and 2) double annotation of a set of documents for each language to compute inter-annotator agreement and discuss difficult cases. ## benchmark statistics in this section, we detail the content of our benchmark in each language. the average number of frame elements (fes) associated with each smell event is between 2.1 and 2.7 for all languages, showing an interesting common feature. furthermore, the most frequent fe is the smell source, followed by the quality for all languages. this shows a pattern in the way smell situations and events are typically described, where the source and the quality are clearly core elements that are necessary to characterise the scene. the fe element with the least annotations is instead 'creator'. this is due to the fact that this role was added at a later stage in the annotation process, mainly to cover documents related to perfumery. it is therefore present only in the benchmarks that contain this kind of documents. for further discussion see section 8. in figure 2 , we report the number of documents per domain in each language-specific benchmark (see list of domains in section 4). overall, we observe a prevalence of literary texts (lit), probably because this is the most represented domain in large repositories such as wikisource and project gutenberg. travel literature and medical texts are also well-represented in all languages. despite the effort to have a balanced benchmark covering the same domains in all languages, however, results are mixed. for some languages, well-represented in large digital repositories, this balance was possible to some extent, with english being the only one covering all domains. for other languages, the benchmarks are affected by the limited variety of resources available in digital format, see for example slovenian. availability is a major obstacle when trying to create historical corpora that cover different domains. in figure 3 , we report the temporal distribution of the documents present in the benchmark for each language. all languages overlap in the time period of interest, with the dutch benchmark including some earlier texts but no data after 1880, and the italian dataset going beyond 1930. similar to the above remarks, also in this case we observe that, due to different data availability, not all time periods are covered equally. 7 towards smell related information extraction one of the goals of this benchmark is to enable temporal-aware information extraction tasks related to the olfactory domain. as a first step in this direction, we explore sentence classification using the english benchmark. since our corpus consists of historical documents, we evaluate performance of a transformer model that is pre-trained using historical corpora, in light of lai et al. (2021)'s proposal. we focus on the task of classifying sentences as smell-related or not. since the corpus is annotated at token level, we first label the sentences that contain any smell event annotation as smell-related, which are 897 out of the total 3,141 sentences. we randomly choose 650 (190 smell-related, 460 not smell-related) sentences as a held-out to measure the performance of fine-tuning on the remaining 2,491 sentences. ## discussion during the creation of the benchmark, we have encountered two major issues related to working with historical data. the first, already mentioned in section 6, is the limited availability of documents for some languages, domains and time spans. this has affected the possibility to create balanced benchmarks for all six languages, although a remarkable effort was put in manually looking for digital collections and selecting relevant documents. another major issue was the need to clean or correct some of the texts before the annotation, mostly due to the limits of ocr applied to old documents. problematic transcriptions can be connected in part to stains or other imperfections in the paper, and in part to the evolution of language, with older documents presenting letters that have fallen into disuse in contemporary language. for instance, in french, italian and english we found lost characters (e.g. long s " ", often confused with "f" as in "perfumes", misspelled as "persumes" in english), characters used differently (v instead of u, like in "vne" for french, or "vlcers" for english), changes in word spelling ("pourquoy" instead of "pourquoi" in french), and abandoned words. another interesting element is that annotation guidelines were adapted several times during the benchmark creation process, because it was not possible to foresee all potential issues we encountered during annotation. indeed, domain specificity of some texts and the different use of language in historical documents made it difficult to straightforwardly follow annotation instructions. for example, frame element definitions have been adjusted and the 'creator' element was added. furthermore, the initial list of lexical units ( ## conclusion and future work in this paper, we presented a multilingual benchmark annotated with smell-related information and covering six languages, which we make available to the research community. we have described the document selection rationale, the annotation process and the main challenges related to the creation of a multilingual benchmark containing historical documents. annotation of latin is in progress, and it will be added to the benchmark as soon as it is complete. the benchmark is only a first step towards the analysis and extraction of olfactory information from historical documents. the work introduced in section 7 will be extended to all six languages, using historical berts when available. furthermore, we will go beyond simple sentence classification, training multilingual classifiers to iden-tify lexical units and frame elements. since the size of the benchmark is rather limited, we will try to expand it in the future but also explore semisupervised, few-shot and cross-lingual approaches to olfactory information extraction.