SENSE - LM : A Synergy between a Language Model and Sensorimotor Representations for Auditory and Olfactory Information Extraction (2024)

Abstract: The five human senses – vision, taste, smell, hearing, and touch – are key concepts that shape human perception of the world. The extraction of sensory references (i.e., expressions that evoke the presence of a sensory experience) in textual corpus is a challenge of high interest, with many applications in various areas. In this paper, we propose SENSE-LM, an information extraction system tailored for the discovery of sensory references in large collections of textual documents. Based on the novel idea of combining the strength of large language models and linguistic resources such as sensorimotor norms, it addresses the task of sensory information extraction at a coarse-grained (sentence binary classification) and fine-grained (sensory term extraction) level.Our evaluation of SENSE-LM for two sensory functions, Olfaction and Audition, and comparison with state-of-the-art methods emphasize a significant leap forward in automating these complex tasks.

## introduction sensoriality, as a psycho-physiological concept @xcite , models the human perception of the world through the five aristotelian sensory functions @xcite : visual (vis), gustatory (gus), olfactory (olf), auditory (aud) and haptic (hap). a sixth sense, interoception (int), was more recently introduced by @xcite , referring to the emotional and physical sensations inherent to the inside of the human body. sensory linguistics refers to the studying of the relationship between human language and sensory experiences @xcite . this research domain has many real-life applications, such as cognitive sciences, cultural history, or even urban planning. for instance, @xcite evidenced a strong relationship between the way olfactory experiences are expressed in the language of inpatients, and the chances of suffering from alzheimer's disease. @xcite focuses on the discovery of auditory indices in large document corpora to design a realistic reconstruction of the sound atmosphere of the city of paris during the 19th century. @xcite focuses on the sensory heritage of smells between the 17th and 20th century, with the goal of providing strong assets for museums to provide olfactory experiments for visitors. such ambitious challenges may jointly solicit complementary spheres of competences, such as art and cultural history, cognitive sciences, and more recently, computational domains such as semantic web @xcite and natural language processing @xcite , with the interest of enhancing sensory information mining processes, notably with language models such as bert @xcite . a set of lexical field generation approaches @xcite @xcite additionally provide interesting vocabulary resources referring to specific sensory domain, but employing them without integrating the text context may limit their scope to a very explicit level of sensory information. in parallel, a strong advance in the modeling of associations between concepts and sensory experiences has been opened by the appearance of the lancaster sensorimotor norms @xcite . this resource asso-ciates 40 000 english lemmas to the way they may evoke each sense, from a human judgement perspective. for instance, such a model represents the fact that, in essence, a concrete concept such as "cat" may evoke well-identified sounds and textures, and to a lesser extent odors, but probably no taste. such resources provide strong assets on the sensory definition of concepts, but still lack of context-awareness, as they focus on isolated terms. in this paper, we propose sense-lm, a novel system that combines the strengths of contextaware models such as language models (lm), linguistic resources, namely sensorimotor representations and lexical generation techniques, to provide a robust approach for detecting sensory-related information in large text corpora, at the sentence and term level. we make the following contributions: the remainder of this paper is organized as follows. section 2 provides an overview of the related work. section 3 elaborates on the objectives and design principles of our contributions. we provide experimental evaluations and analysis in section 4. we summarize our findings and draw our conclusions in section 5, and discuss the current limitations of our solution in section 6. an appendix provides further analyses of our experiments. ## related work one of the main challenges of textual sensory information research, that we address in this paper, is about finding terms or expressions related to a sensory experience in a corpus of textual documents. in this section, we describe the existing approaches for addressing this task. ## methodology in this section, we present our system sense-lm, designed for detecting text information describing sensory experiences in documents. sense-lm allows extracting sensory references in large corpora, at a sentence and at a token level. figure 1 depicts its global workflow. step 1 performs a coarsegrained classification task, aiming at identifying, within a set of documents d, the sentences that evoke the presence (or not) of references related to one of the five sensory functions. we may sum up this binary classification task by the following question: "does a sentence s expresses an idea evoking a given sense m among the five senses ?" then, step 2 applies a fine-grained classification process, for extracting word utterances that reflect the presence of the target sensory function in the sentence context. we sum up this classification task with the following question : "which words, in this sentence s, evoke the presence of the given sensory function m ?". it is worth noting that such a method adresses the task of researching multi-sensory information, i.e. finding, in a same document, information that refer to several sensory functions. in that case, it is enough to apply a one-vs-rest strategy, which consists to split the multi-class classification problem into several binary classification problems, one per class, and to learn a model on each. thus, for instance, if a sentence contained several sensory information, it will classified positively by several instances of the model, whereas if it does not contain any, it will be classified negatively by all the models. ## experiments and analyses this section presents an experimental evaluation of the effectiveness of sense-lm. the performances are measured for each step and compared with those provided by baselines that address the same task. an ablative study is also carried out to evaluate the interest of each of the components implemented in step 2. the software and hardware environments of these experiments are described in appendix b, and an analysis of the computational costs of sense-lm is provided in appendix d. ## conclusion and future works in this paper, we presented sense-lm, a novel framework for coarse-grained, at the sentence level, and fine-grained, at the word level, sensory references detection. as far as we know, sense-lm is the first approach proposing a combination of sensorimotor representations with the text features of language models such as bert for sensory information extraction in text documents. in addition, unlike other systems which are dedicated to a particular sensoriality, it offers the advantage of being generic and applicable to any sense. its evaluation on two datasets for two different sensory functions, olfaction and audition, provides enhanced and encouraging results compared to state-of-the-art solutions. moreover, an ablative study confirms the contribution of each component of the system, highlighting that using sentence context-aware approaches and human-judgement based approaches together brings a new step forward in the task of identifying sensory references in text, as these two approaches are complementary. this work opens interesting directions for future works. our approach, evaluated on a sensory information research task, could be transferred to similar tasks involving human judgement, such as sentiment analysis or political polarity analysis, by replacing the sensorimotor representation function by an equivalent function built on human-judgement based resources tailored for other domain-specific tasks. thus, our work on sensoriality shows a new way to enhance a human judgement oriented task with the help of multimodality, and opens a set of interesting research directions for other application domains. from a language-models study perspective, we may inspire from existing works that enrich language models with extra modalities such as images alongside sensorimotor representations @xcite . the principle of combining the three aforementioned modalities (text, image and sensorimotor), has been applied to purely text-oriented tasks, but has not been applied yet to the research of sensory indices in text corpora. conversely, the synergy of text and sensorimotor modalities, that we valued in this paper, could be employed to enrich computer vision and multi-modal architectures for extracting visual sensory information from images.