ymoslem/acl-abstracts huggingface dataset:
Temporal Word Embeddings in the Study of Metaphor Change over Time and across Genres: A Proof-of-concept Study on English (2024)

Temporal word embeddings have been successfully employed in semantic change research to identify and trace shifts in the meaning of words. In a previous work, we developed an approach to study the diachrony of complex expressions, namely literary metaphors extracted from Italian literary texts. Capitalizing on the evidence that measures of cosine similarity between the two terms of a metaphor approximate human judgments on the difficulty of the expression, we used time-locked measures of similarity to reconstruct the evolution of processing costs of literary metaphors over the past two centuries. In this work, we present a proof-of-concept study testing the crosslinguistic applicability of this approach on a set of 19th-century English literary metaphors. Our results show that metaphors changed as a function of textual genre but not of epoch: cosine similarity between the two terms of literary metaphors is higher in literary compared to nonliterary texts, and this difference is stable across epochs. We show that the difference between genres is affected by the frequency of the metaphor`s vehicle and the stability of the meaning of both topic and vehicle. Overall, the processing costs of English literary metaphors do not differin different time points, but are influenced by the textual genres of language. In a broader perspective, general considerations can be drawn about the history of literary and nonliterary English language and the semantic change of words
Fine-grained Sexism Detection in Italian Newspapers (2024)

In recent years, tasks revolving around hate speech detection have experienced a growing interest in the field of Natural Language Processing. Two main trends stand out in the context of sexism recognition: the focus on overt forms of sexism such as misogyny on social media and tackling the problem as a text classification task. The main objective of this work is to introduce a new approach to tackle sexism recognition as a sequence labelling task, operating on the token level rather than the document one. To achieve this goal, we introduce (i) the FGSDI (Fine-Grained Sexism Detection in Italian) corpus, containing Italian newspaper articles annotated with fine-grained linguistic markers of sexism, and (ii) a two-step pipeline that sequentially performs sexism detection on the sentence level and sexism classification on the token one. Our primary findings include that (i) tackling the task of sexism recognition as a sequence labelling task is possible, however, a large amount of labelled data is needed; (ii) leveraging few-shot learning for sexism detection proves to be an effective solution in scenarios where only a limited amount of data are available; (iii) the proposed pipeline approach allows for better results compared to the baseline by doubling the overall precision and achieving a better F1-score.
Towards a More Comprehensive Evaluation for Italian LLMs (2024)

Recent Large Language Models (LLMs) have shown impressive performance in addressing complex aspects of human language. These models have also demonstrated significant capabilities in processing and generating Italian text, achieving state-of-the-art results on current benchmarks for the Italian language. However, the number of such benchmarks is still insufficient. A case in point is the {\textquotedblleft}Open Ita LLM Leaderboard{\textquotedblright} which only supports three benchmarks, despite being one of the most popular evaluation suite for the evaluation of Italian-speaking LLMs. In this paper, we analyze the current pitfalls of existing evaluation suites and propose two ways to this gap: i) a new suite of automatically-translated benchmarks, drawn from the most popular English benchmarks; and ii) the adaptation of existing manual dataset so that they can be used to complement the evaluation of Italian LLMs. We discuss the pros and cons of both approaches and release all our data to foster further research on the evaluation of Italian-speaking LLMs.
A Study on the Soundness of Closed-ended Evaluation of Large Language Models Adapted to the Italian Language (2024)

With the rising interest in Large Language Models, deep architectures capable of solving a wide range of Natural LanguageGeneration tasks, an increasing number of open weights architectures have been developed and released online. In contrastwith older architectures, which were aimed at solving specific linguistic assignments, Large Language Models have shownoutstanding capabilities in solving several tasks at once, raising the question of whether they can truly comprehend naturallanguage. Nevertheless, evaluating this kind of capability is far from easy. One of the proposed solutions so far is usingbenchmarks that combine various types of tasks. This approach is based on the premise that achieving good performance ineach of these individual tasks can imply having developed a model capable of understanding language. However, while thisassumption is not incorrect, it is evident that it is not sufficient, and the evaluation of Large Language Models still remains anopen challenge. In this paper, we conduct a study aimed at highlighting the potential and limitations of current datasets andhow a new evaluation setting applied to language-adapted Large Language Models may provide more insight than traditionalapproaches.
Understanding the Future Green Workforce through a Corpus of Curricula Vitae from Recent Graduates (2024)

In view of the much-heralded ecological transition, to stay competitive and participate in the collective effort to face global warming and climate change, organisations need to select employees interested in and able to develop environmentally sustainable and innovative ideas. The existing literature however does not present consistent nor concordant results on the effective interest, involvement and expertise of Generation Z members {--} namely, the newest entrants into the workforce {--} in green issues. The aim of this study is to explore the profile of the upcoming workforce expected to present itself to companies, and to support them in managing the green transition. With CVs as one of the first interfaces between candidate and company in the recruitment process, this study is based on a purpose-built corpus consisting of 8,096 Curricula Vitae from recent graduates of the University of Modena and Reggio Emilia. Data is investigated through a Corpus-Assisted Discourse Studies (CADS) framework, proposing a novel interaction between structured metadata and textual information. The original contribution of this approach lies in the extraction of information from the narrative structure of CVs which, guiding the evaluation and exploration of metadata, ensures that the knowledge value of the data can be explored in a discursive manner and not reduced to lists of competences and qualifications.
Exploring Italian Sentence Embeddings Properties through Multi-tasking (2024)

We investigate to what degree existing LLMs encode abstract linguistic information in Italian in a multi-task setting. We exploit curated synthetic data on a large scale {--} several Blackbird Language Matrices (BLMs) problems in Italian {--} and use them to study how sentence representations built using pre-trained language models encode specific syntactic and semantic information. We use a two-level architecture to model separately a compression of the sentence embeddings into a representation that contains relevant information for a task, and a BLM task. We then investigate whether we can obtain compressed sentence representations that encode syntactic and semantic information relevant to several BLM tasks. While we expected that the sentence structure {--} in terms of sequence of phrases/chunks {--} and chunk properties could be shared across tasks, performance and error analysis show that the clues for the different tasks are encoded in different manners in the sentence embeddings, suggesting that abstract linguistic notions such as constituents or thematic roles does not seem to be present in the pretrained sentence embeddings.
Exploring Syntactic Information in Sentence Embeddings through Multilingual Subject-verb Agreement (2024)

In this paper, our goal is to investigate to what degree multilingual pretrained language models capture cross-linguistically valid abstract linguistic representations. We take the approach of developing curated synthetic data on a large scale, with specific properties, and using them to study sentence representations built using pretrained language models. We use a new multiple-choice task and datasets, Blackbird Language Matrices (BLMs), to focus on a specific grammatical structural phenomenon {--} subject-verb agreement across a variety of sentence structures {--} in several languages. Finding a solution to this task requires a system detecting complex linguistic patterns and paradigms in text representations. Using a two-level architecture that solves the problem in two steps {--} detect syntactic objects and their properties in individual sentences, and find patterns across an input sequence of sentences {--} we show that despite having been trained on multilingual texts in a consistent manner, multilingual pretrained language models have language-specific differences, and syntactic structure is not shared, even across closely related languages.
Dynamic Prompting: Large Language Models for Task Oriented Dialog (2024)

Large Language Models show impressive results in many different applications, most notably in the context of question-answering and open dialog situations. However, it is still an open question how to use those models for task-oriented dialogs such as booking or customer information systems, and such. In this work, we propose Dynamic Prompting, an architecture for task-oriented dialog, integrating the benefits of Large Language Models and showcasing the approach on the MultiWOZ 2.2 dataset. Our architecture leads to a high task success rate, provides sensible and specific answers, and is resistant to hallucinations. Further, we show that Dynamic Prompting is able to answer questions that were not anticipated by the dialog systems designer and that it can correct several types of errors and other characteristics of the system.
Exploring Text-Embedding Retrieval Models for the Italian Language (2024)

Text retrieval systems have become essential in the field of natural language processing (NLP), serving as the backbone for applications such as search engines, document indexing, and information retrieval. With the rise of generative AI, particularly Retrieval-Augmented Generation (RAG) systems, the demand for robust text retrieval models has increased. However, existing large language models (LLMs) and datasets are often insufficiently optimized for Italian, limiting their performance in Italian text retrieval tasks. This paper addresses this gap by proposing both a data collection and specialized models tailored for Italian text retrieval. Through extensive experimentation, we analyze the improvements and limitations in retrieval performance, paving the way for more effective Italian NLP applications.
Introducing MultiLS-IT: A Dataset for Lexical Simplification in Italian (2024)

Lexical simplification is a fundamental task in Natural Language Processing, aiming to replace complex words with simpler synonyms while preserving the original meaning of the text. This task is crucial for improving the accessibility of texts for different user groups. In this article, we present MultiLS-IT, the first dataset specifically designed for automatic lexical simplification in Italian, as part of the larger multilingual Multi-LS dataset. We offer a detailed description of the data collection and annotation process, along with a comprehensive statistical analysis of the dataset. Our dataset provides a basis for the development and evaluation of automatic simplification models, contributing to the broader goal of making texts more accessible to all readers.
Enhancing Lexical Complexity Prediction in Italian through Automatic Morphological Segmentation (2024)

Morphological analysis is vital for various NLP tasks as it provides insights into word structures and enhances the understanding of morphological and syntactic relationships. This study focuses on surface morphological segmentation for the Italian language, addressing the lack of detailed morphological representation in existing corpora. By utilizing an automatic segmenter, we aim to extract quantitative morphological parameters to understand their impact on word complexity perception. Our correlation analysis reveals that morphological features significantly influence the perceived complexity of words.
Measuring Bias in Instruction-Following Models with ItaP-AT for the Italian Language (2024)

Instruction-Following Language Models (IFLMs) are the state-of-the-art for solving many downstream tasks. Given their widespread use, there is an urgent need to measure whether the sentences they generate contain toxic information or social biases. In this paper, we propose Prompt Association Test for the Italian language (ItaP-AT): a new resource for testing the presence of social bias in different domains in IFLMs. This work also aims to understand whether it is possible to make the responses of these models more fair by using context learning, using {\textquotedblleft}one-shot anti-stereotypical prompts{\textquotedblright}.
Minerva LLMs: The First Family of Large Language Models Trained from Scratch on Italian Data (2024)

The increasing popularity of Large Language Models (LLMs) has led to a surge in research on adapting existing models to different languages. However, the pretraining of non-English LLMs is still an underexplored area and there is no open-source endeavor that explores what is achievable with open Italian data. To address this issue, we present Minerva, the first family of LLMs trained from scratch on Italian data. The creation of Minerva is an opportunity to explore and investigate the pretraining of LLMs for the Italian language, outlining the challenges that arise when training LLMs with native Italian texts. Minerva demonstrates that an LLM for a specific language brings a number of practical benefits compared to the adaptation of an existing one, including deep control over the composition of the vocabulary and the training data. With this paper, we aim to provide a comprehensive overview of the design choices, results, and evaluation of our Minerva models, showing promising results on Italian benchmarks and downstream tasks. Most importantly, we share what we learned and the findings obtained during the development of Minerva, as we believe that our experience will be valuable for the academic and industrial communities interested in training non-English LLMs from scratch.
Benchmarking the Semantics of Taste: Towards the Automatic Extraction of Gustatory Language (2024)

In this paper, we present a benchmark containing texts manually annotated with gustatory semantic information. We employ a FrameNet-like approach previously tested to address olfactory language, which we adapt to capture gustatory events. We then propose an exploration of the data in the benchmark to show the possible insights brought by this type of approach, addressing the investigation of emotional valence in text genres. Eventually, we present a supervised system trained with the taste benchmark for the extraction of gustatory information from historical and contemporary texts.
Nominal Class Assignment in Swahili: A Computational Account (2024)

We discuss the open question of the relation between semantics and nominal class assignment in Swahili. We approach theproblem from a computational perspective, aiming first to quantify the extent of this relation, and then to explicate its nature,taking extra care to suppress morphosyntactic confounds. Our results are the first of their kind, providing a quantitativeevaluation of the semantic cohesion of each nominal class, as well as a nuanced taxonomic description of its semantic content.
Did Somebody Say \textquoteleftGest-IT'? A Pilot Exploration of Multimodal Data Management (2024)

The paper presents a pilot exploration of the construction, management and analysis of a multimodal corpus. Through athree-layer annotation that provides orthographic, prosodic, and gestural transcriptions, the gest-IT resource allows oneto investigate the variation of gesture-making patterns in conversations between sighted people and people with visualimpairment. After discussing the transcription methods and technical procedures employed in our study, we will propose aunified CoNLL-U corpus and indicate our future steps.
Confronto tra Diversi Tipi di Valutazione del Miglioramento della Chiarezza di Testi Amministrativi in Lingua Italiana (2024)

The paper presents a comparison of different types of evaluation of administrative texts in the Italian language on which a clarity improvement intervention was carried out. The clarity improvement was performed by human experts and ChatGPT. The evaluation was carried out in four different ways: by expert evaluators, used as a reference; by evaluators with good skills, subject to dedicated training; by generic evaluators recruited through a crowdsourcing platform; by ChatGPT. The results show that the closest match to the results of the evaluation by expert evaluators was reached, by a wide margin, by evaluators with good skills and dedicated training; the second best approach was reached by requesting evaluation from ChatGPT; the worst approach was reached by generic evaluators recruited through a crowdsourcing platform. Task features that may have influenced the outcome are also discussed.
Towards an Automatic Evaluation of (In)coherence in Student Essays (2024)

Coherence modeling is an important task in natural language processing (NLP) with potential impact on other NLP taskssuch as Natural Language Understanding or Automated Essay Scoring. But it can also offer interesting linguistic insightswith pedagogical implications. Early work on coherence modeling has focused on exploring definitions of the phenomenonand in recent years, neural models have entered also this field of research allowing to successfully distinguish coherent fromincoherent (synthetically created) texts or to identify the correct continuation for a given sample of texts as demonstratedfor Italian in the DisCoTex task of EVALITA 2023. In this article, we target coherence modeling for Italian language in astrongly domain-specific scenario, i.e. education. We use a corpus of student essays, collected to analyse student`s textcoherence and data augmentation techniques to experiment with the effect of various linguistically informed features ofincoherent writing on current coherence modelling strategies used in NLP. Our results show the capabilities of encodermodels to capture features of (in)coherence in a domain-specific scenario discerning natural from artificially corrupted texts.Our code is available at the following url https://gitlab.inf.unibz.it/commul/itaca/automatic{\_}eval
MONICA: Monitoring Coverage and Attitudes of Italian Measures in Response to COVID-19 (2024)

Modern social media have long been observed as a mirror for public discourse and opinions. Especially in the face of exceptional events, computational language tools are valuable for understanding public sentiment and reacting quickly. During the coronavirus pandemic, the Italian government issued a series of financial measures, each unique in target, requirements, and benefits. Despite the widespread dissemination of these measures, it is currently unclear how they were perceived and whether they ultimately achieved their goal.In this paper, we document the collection and release of MONICA, a new social media dataset for MONItoring Coverage and Attitudes to such measures. Data include approximately ten thousand posts discussing a variety of measures in ten months. We collected annotations for sentiment, emotion, irony, and topics for each post. We conducted an extensive analysis using computational models to learn these aspects from text. We release a compliant version of the dataset to foster future research on computational approaches for understanding public opinion about government measures. We will release the data at URL.
Unraveling the Enigma of SPLIT in Large-Language Models: The Unforeseen Impact of System Prompts on LLMs with Dissociative Identity Disorder (2024)

Our work delves into the unexplored territory of Large-Language Models (LLMs) and their interactions with System Prompts, unveiling the previously undiscovered implications of SPLIT (System Prompt Induced Linguistic Transmutation) in commonly used state-of-the-art LLMs. Dissociative Identity Disorder, a complex and multifaceted mental health condition, is characterized by the presence of two or more distinct identities or personas within an individual, often with varying levels of awareness and control. The advent of large-language models has raised intriguing questions about the presence of such conditions in LLMs. Our research investigates the phenomenon of SPLIT, in which the System Prompt, a seemingly innocuous input, profoundly impacts the linguistic outputs of LLMs. The findings of our study reveal a striking correlation between the System Prompt and the emergence of distinct, persona-like linguistic patterns in the LLM`s responses. These patterns are not only reminiscent of the dissociative identities present in the original data but also exhibit a level of coherence and consistency that is uncommon in typical LLM outputs. As we continue to explore the capabilities of LLMs, it is imperative that we maintain a keen awareness of the potential for SPLIT and its significant implications for the development of more human-like and empathetic AI systems.
The limits of Italian in Reasoning Tasks (2024)

Previous studies have demonstrated the effectiveness of \textit{reasoning methods} in eliciting multi-step reasoned answers from Large Language Models (LLMs) by leveraging in-context demonstrations. These methods, exemplified by Chain-of-Thought (CoT) and Program-Aided Language Models (PAL), have been shown to reason well in monolingual contexts, primarily in English. There has, however, been limited exploration of their abilities in other languages, especially in Italian.To gain a deeper understanding of the role of reasoning methods in in-context demonstrations, we propose a multidimensional analysis tailored to Italian, focusing on arithmetic and symbolic reasoning tasks. Our findings indicate that the effectiveness of reasoning methods varies significantly beyond English. Specifically, CoT, which relies on natural language demonstrations, is limited to English. Conversely, the structured nature of PAL in-context demonstrations facilitates multilingual comprehension, enabling LLMs to generate programmatic answers in Italian as well. Finally, for a more comprehensive overview, we observe that additional alignment methods do not improve downstream performances; in contrast, in some cases, they limit the abilities of the original models. This leads to significant improvements in the accuracy and quality of the generated responses.