ymoslem/acl-abstracts huggingface dataset:
Lost in Disambiguation: How Instruction-Tuned LLMs Master Lexical Ambiguity (2024)

This paper investigates how decoder-only instruction-tuned LLMs handle lexical ambiguity. Two distinct methodologies are employed: Eliciting rating scores from the model via prompting and analysing the cosine similarity between pairs of polysemous words in context. Ratings and embeddings are obtained by providing pairs of sentences from Haber and Poesio (2021) to the model. These ratings and cosine similarity scores are compared with each other and with the human similarity judgments in the dataset.Surprisingly, the model scores show only a moderate correlation with the subjects' similarity judgments and no correlation with the target word embedding similarities. A vector space anisotropy inspection has also been performed, as a potential source of the experimental results. The analysis reveals that the embedding spaces of two out of the three analyzed models exhibit poor anisotropy, while the third model shows relatively moderate anisotropy compared to previous findings for models with similar architecture (Ethayarajh 2019). These findings offer new insights into the relationship between generation quality and vector representations in decoder-only LLMs.
BaBIEs: A Benchmark for the Linguistic Evaluation of Italian Baby Language Models (2024)

The possibility of comparing the linguistic competence of Language Models (LMs) to that of children has gained growing attention lately, raising the need for effective tools for evaluating both the former and the latter. To this purpose, we developed a resource for the linguistic evaluation of BabyLMs, which are LMs trained on datasets that comparable to the linguistic stimulus received by children. This resource adapts four standardized tests for the evaluation of linguistic skills of Italian-speaking children (BVL, TROG-2, TCGB-2 and Peabody). To verify the effectiveness of our benchmark, we administered it to Minerva, a LLM pretrained from scratch on Italian. Our results indicate that Minerva struggles to master certain linguistic aspects, achieving an age-equivalent score of 4 years, and that the type of task administered affects the model`s performance.
Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers (2024)

How newspapers cover news significantly impacts how facts are understood, perceived, and processed by the public. This is especially crucial when serious crimes are reported, e.g., in the case of femicides, where the description of the perpetrator and the victim builds a strong, often polarized opinion of this severe societal issue. This paper presents FMNews, a new dataset of articles reporting femicides extracted from Italian newspapers. Our core contribution aims to promote the development of a deeper framing and awareness of the phenomenon through an original resource available and accessible to the research community, facilitating further analyses on the topic. The paper also provides a preliminary study of the resulting collection through several example use cases and scenarios.
Women`s Professions and Targeted Misogyny Online (2024)

With the increasing popularity of social media platforms, the dissemination of misogynistic content has become more prevalent and challenging to address. In this paper, we investigate the phenomenon of online misogyny on Twitter through the lens of hurtfulness, qualifying its different manifestation considering the profession of the targets of misogynistic attacks.By leveraging manual annotation and a BERTweet model trained for fine-grained misogyny identification, we find that specific types of misogynistic speech are more intensely directed towards particular professions: derailing discourse predominantly targets authors and cultural figures, while dominance-oriented speech and sexual harassment are mainly directed at politicians and athletes. Additionally, we use the HurtLex lexicon and ItEM to assign hurtfulness scores to tweets based on different hate speech categories. Our analysis reveals that these scores align with the profession-based distribution of misogynistic speech, highlighting the targeted nature of such attacks.
DWUGs-IT: Extending and Standardizing Lexical Semantic Change Detection for Italian (2024)

Lexical Semantic Change Detection (LSCD) is the task of determining whether a word has undergone a change in meaning over time. There has been a marked increase in interest in this task, accompanied by a corresponding growth in the scientific community involved in developing computational approaches to semantic change. In recent years, a number of resources have been made available for the evaluation of LSC models in a number of languages, including English, Swedish, German, Latin, Russian and Chinese. DIACR-ITA is the only existing resource for LSCD in Italian. However, DIACR-ITA has a different format from that used for other languages. In this paper we present DWUGs-IT, which extends the DIACR-ITA dataset with additional target words and usage-sense pair annotations and adapts it to the DURel format, including the first implementation of a LSCD graded task for Italian.
History Repeats: Historical Phase Recognition from Short Texts (2024)

This paper introduces a new multi-class classification task: the prediction of the Structural-Demographic phase of historical cycles - such as growth, impoverishment and crisis - from text describing historical events. To achieve this, we leveraged data from the Seshat project, annotated it following specific guidelines and then evaluated the consistency between three annotators. The classification experiments, with transformers and Large Language Models, show that 2 of 5 phases can be detected with good accuracy. We believe that this task could have a great impact on comparative history and can be helped by event extraction in NLP.
Emojilingo: Harnessing AI to Translate Words into Emojis (2024)

This paper presents an AI experiment of translation in emoji conducted on a glossary from Dante Alighieri`s Comedy. The experiment is part of a project aiming to build up an automated emojibased pivot language providing an interlingua as a tool for linguistic simplification, accessibility, and international communication: Emojilingo. The present test involves human (Emojitaliano) and machine (Chat-GPT) translations in a comparative analysis to devise an automated integrated model highlighting emojis' expressive ability in transferring senses, clarifying semantic obscurities and ambiguities, and simplifying language. A first preliminary evaluation highlights Chat-GPT`s ability to deal with a classic archaic literary vocabulary, also raising issues on managing criteria for better grasping the meanings and forms and about the multicultural extent of content transfer.
Towards an ASR System for Documenting Endangered Languages: A Preliminary Study on Sardinian (2024)

Speech recognition systems are still highly dependent on textual orthographic resources, posing a challenge for low-resourcelanguages. Recent research leverages self-supervised learning of unlabeled data or employs multilingual models pre-trainedon high resource languages for fine-tuning on the target low-resource language. These are effective approacheswhen the target language has a shared writing tradition, but when we are confronted with mainly spoken languages, beingthem endangered minority languages, dialects, or regional varieties, other than labeled data, we lack a shared metric toassess speech recognition performance. We first provide a research background on ASR for low-resource languages anddescribe the specific linguistic situation of Campidanese Sardinian, we then evaluate five multilingual ASR models usingtraditional evaluation metrics and an exploratory linguistic analysis. The paper addresses key challenges in developing a toolfor researchers to document and analyze the phonetics and phonology of spoken (endangered) languages.
Controllable Text Generation to Evaluate Linguistic Abilities of Italian LLMs (2024)

State-of-the-art Large Language Models (LLMs) demonstrate exceptional proficiency across diverse tasks, yet systematic evaluations of their linguistic abilities remain limited. This paper addresses this gap by proposing a new evaluation framework leveraging the potentialities of Controllable Text Generation. Our approach evaluates the models' capacity to generate sentences that adhere to specific linguistic constraints and their ability to recognize the linguistic properties of their own generated sentences, also in terms of consistency with the specified constraints. We tested our approach on six Italian LLMs using various linguistic constraints.
A Modal Sense Classifier for the French Modal Verb Pouvoir (2024)

In this paper we address the problem of modal sense classification for the French modal verb pouvoir in a transcribed spoken corpus. To the best of our knowledge, no studies have focused on this task in French. We fine-tuned various BERT-based models for French in order to determine which one performed best. It was found that the Flaubert-base-cased model was the most effective (F1-score of 0.94) and that the most frequent categories in our corpus were material possibility and ability, which are both part of the more global alethic category.
Topic Similarity of Heterogeneous Legal Sources Supporting the Legislative Process (2024)

The legislative process starts with a deep analysis of the existing regulations at European and national levels to avoid conflicts and fostering the into force norms. Also the Constitutional Court decisions play a fundamental role in this analysis for checking the compliance with the constitutional framework and for including the inputs coming from this relevant court in the law-making process. Finally, it is also significant to compare the forthcoming proposal with the already presented bills regarding the same topic. This comparison is crucial to avoid overlapping and to coordinate the democratic dialogue with the different parties. In this light, this paper presents an unsupervised approach for calculating similarity between heterogeneous documents annotated in Akoma Ntoso XML, with the aim to support the information retrieval of similar documents using thematic taxonomy used in legal domain. The prototype has been developed for answering to a call for manifestation of interests launched by the Chamber of Deputy of Italy in order to adopt hybrid AI in the legislation process. It uses a completely unsupervised approach based on Sentence Transformers, meaning that neither annotated data or any fine-tuning process is required.
Join Together? Combining Data to Parse Italian Texts (2024)

In this paper, we create and evaluate non-combined and combined models using Old and Contemporary Italian data to determine whether increasing the size of the training data with a combined model could improve parsing accuracy to facilitate manual annotation. We find that, despite the increased size of the training data, in-domain parsing performs better. Additionally, we discover that models trained on Old Italian data perform better on Contemporary Italian data than the reverse. We attempt to explain this result in terms of syntactic complexity, finding that Old Italian text exhibits higher sentence length and non-projectivity rate.
Using Large Speech Models for Feature Extraction in Cross-Lingual Speech Emotion Recognition (2024)

Large Speech Models (LSMs), pre-trained on extensive unlabeled data using Self-Supervised Learning (SSL) or WeaklySupervised Learning (WSL), are increasingly employed for tasks like Speech Emotion Recognition (SER). Their capability to extract general-purpose features makes them a strong alternative to low-level descriptors. Most studies focus on English, with limited research on other languages. We evaluate English-Only and Multilingual LSMs from the Wav2Vec 2.0 and Whisper families as feature extractors for SER in eight languages. We have stacked three alternative downstream classifiers of increasing complexity, named Linear, Non-Linear, and Multi-Layer, on top of the LSMs. Results indicate that Whisper models perform best with a simple linear classifier using features from the last transformer layer, while Wav2Vec 2.0 models benefit from features from the middle and early transformer layers. When comparing English-Only and Multilingual LSMs, we find that Whisper models benefit from multilingual pre-training, excelling in Italian, Canadian French, French, Spanish, German and competitively on Greek, Egyptian Arabic, Persian. In contrast, English-Only Wav2Vec 2.0 models outperform their multilingual counterpart, XLS-R, in most languages, achieving the highest performance in Greek, Egyptian Arabic.
Building a Pragmatically Annotated Diachronic Corpus: The DIADIta Project (2024)

We present here the initial stages of the construction of the DIADIta corpus, a diachronic corpus of Italian annotated for interactional pragmatic phenomena. First, we describe the annotation scheme, which is structured into four levels: speech acts (e.g., apology; threat), forms (e.g., discourse marker; expressive), pragmatic functions (which are speaker-oriented, e.g., mitigation; turn-taking), and pragmatic aims (which are interlocutor-oriented, e.g., attention-getting; request for agreement). Next, we discuss how the results of a first annotation exercise provide indications for refining the annotation procedure.
Building CorefLat. a Linguistic Resource for Coreference and Anaphora Resolution in Latin (2024)

This paper presents the initial stages of a project focused on coreference and anaphora resolution in Latin texts. By building a corpus enhanced with coreference/anaphora annotation, the project wants to explore empirically a layer of metalinguistic analysis that has not been yet extensively investigated in linguistic resources and natural language processing for Latin. After reviewing the related work, the paper discusses annotation criteria and data analysis, providing examples about a few issues that emerged during the annotation process.
Is Explanation All You Need? An Expert Survey on LLM-generated Explanations for Abusive Language Detection (2024)

Explainable abusive language detection has proven to help both users and content moderators, and recent research has focused on prompting LLMs to generate explanations for why a specific text is hateful. Yet, understanding the alignment of these generated explanations with human expectations and judgements is far from being solved. In this paper, we design a before-and-after study recruiting AI experts to evaluate the usefulness and trustworthiness of LLM-generated explanations for abusive language detection tasks, investigating multiple LLMs and learning strategies. Our experiments show that expectations in terms of usefulness and trustworthiness of LLM-generated explanations are not met, as their ratings decrease by 47.78{\%} and 64.32{\%}, respectively, after treatment. Further, our results suggest caution in using LLMs for explanation generation of abusive language detection due to (i) their cultural bias, and (ii) difficulty in reliably evaluating them with empirical metrics. In light of our results, we provide three recommendations to use LLMs responsibly for explainable abusive language detection.
Scalable Query Understanding for E-commerce: An Ensemble Architecture with Graph-based Optimization (2024)

Query understanding is a critical component of e-commerce platforms, enabling accurate interpretation of users' intents and efficient retrieval of relevant products. This paper presents a study on scalable query understanding techniques applied to a real use case in the e-commerce grocery domain. We propose a novel architecture that combines deep learning models with traditional ML models to capture query nuances and provide robust performance. Our model ensemble approach aims to capture the nuances of user queries and provide robust performance across various query types and categories. We conduct experiments on real-life datasets and demonstrate the effectiveness of our proposed solution in terms of accuracy and scalability. An optimized graph-based architecture using Ray enables efficient processing of high-volume traffic. The experimental results highlight the benefits of combining diverse models.
ELIta: A New Italian Language Resource for Emotion Analysis (2024)

Emotions and language are strongly associated. In recent years, many resources have been created to investigate this association and automatically detect emotions from texts.Presenting ELIta (Emotion Lexicon for Italian), this study provides a new language resource for the analysis and detection of emotions in Italian texts. It describes the process of lexicon creation, including lexicon selection and annotation methodologies, and compares the collected data with existing resources. By offering a non-aggregated lexicon, ELIta fills a crucial gap and is applicable to various research and practical applications. Furthermore, the work utilises the lexicon by analysing the relationships between emotions and gender.
Comparing Large Language Models Verbal Creativity to Human Verbal Creativity (2024)

This study investigates verbal creativity differences and similarities between Large Language Models and humans, based ontheir answers given to the integrated verbal creativity test in [1 ]. Since this article reported a very small difference of scoresin favour of the machines, the aim of the present work is to thoroughly analyse the data through four methods: scoring theuniqueness of the answers of one human or one machine compared to all the others, semantic similarity clustering, binaryclassification and manual inspection of the data. The results showed that humans and machines are on a par in terms ofuniqueness scores, that humans and machines group in two well defined clusters based on semantics similarities, and that theanswers are not so easy to automatically classify in human answers and LLM answers.
ItGraSyll: A Computational Analysis of Graphical Syllabification and Stress Assignment in Italian (2024)

In this paper we build a dataset of Italian syllables. We perform quantitative and qualitative analyses on the syllabification and stress assignment in Italian. We propose a machine learning model, based on deep-learning techniques, for automatically inferring syllabification and stress assignment. For stress prediction we report 94.45{\%} word-level accuracy, and for syllabification we report 98.41{\%} word-level accuracy and 99.82{\%} hyphen-level accuracy.
Generation and Evaluation of English Grammar Multiple-Choice Cloze Exercises (2024)

English grammar Multiple-Choice Cloze (MCC) exercises are crucial for improving learners' grammatical proficiency andcomprehension skills. However, creating these exercises is labour-intensive and requires expert knowledge. Effective MCCexercises must be contextually relevant and engaging, incorporating distractors{---}plausible but incorrect alternatives{---}tobalance difficulty and maintain learner motivation. Despite the increasing interest in utilizing large language models (LLMs)in education, their application in generating English grammar MCC exercises is still limited. Previous methods typicallyimpose constraints on LLMs, producing grammatically correct yet uncreative results. This paper explores the potentialof LLMs to independently generate diverse and contextually relevant MCC exercises without predefined limitations. Wehypothesize that LLMs can craft self-contained sentences that foster learner`s communicative competence. Our analysisof existing MCC exercise datasets revealed issues of diversity, completeness, and correctness. Furthermore, we addressthe lack of a standardized automatic metric for evaluating the quality of generated exercises. Our contributions includedeveloping an LLM-based solution for generating MCC exercises, curating a comprehensive dataset spanning 19 grammartopics, and proposing an automatic metric validated against human expert evaluations. This work aims to advance theautomatic generation of English grammar MCC exercises, enhancing both their quality and creativity.
ReCLAIM Project: Exploring Italian Slurs Reappropriation with Large Language Models (2024)

Recently, social networks have become the primary means of communication for many people, leading computational linguistics researchers to focus on the language used on these platforms. As online interactions grow, recognizing and preventing offensive messages targeting various groups has become urgent. However, finding a balance between detecting hate speech and preserving free expression while promoting inclusive language is challenging. Previous studies have highlighted the risks of automated analysis misinterpreting context, which can lead to the censorship of marginalized groups. Our study is the first to explore the reappropriative use of slurs in Italian by leveraging Large Language Models (LLMs) witha zero-shot approach. We revised annotations of an existing Italian homotransphobic dataset, developed new guidelines, and designed various prompts to address the LLMs task. Our findings illustrate the difficulty of this challenge and provide preliminary results on using LLMs for such a language specific task.
You Write like a GPT (2024)

We investigate how Raymond Queneau`s Exercises in Style are evaluated by automatic methods for detection of artificially-generated text. We work with the Queneau`s original French version, the Italian translation by Umberto Eco andthe English translation by Barbara Wright.We start by comparing how various methods for the detection of automatically generated text, also using different large language models and evaluate the different styles in the opera. We then link this automatic evaluation to distinct characteristic related to content and structure of the various styles.This work is an initial attempt at exploring how methods for detection artificially-generated text can find application as tools to evaluate the qualities and characteristics of human writing, to support better writing in terms of originality, informativeness, clarity.
Constructing a Multimodal, Multilingual Translation and Interpreting Corpus: A Modular Pipeline and an Evaluation of ASR for Verbatim Transcription (2024)

This paper presents a novel pipeline for constructing multimodal and multilingual parallel corpora, with a focus on evaluating state-of-the-art ASR tools for verbatim transcription. Our findings indicate that current technologies can streamline corpus construction, with fine-tuning showing promising results in terms of transcription quality compared to out-of-the-box Whisper models. The lowest overall WER achieved for English was 0.180, using a fine-tuned Whisper-small model. As for Italian, the fine-tuned Whisper-small model obtained a lower WER of 0.201 compared to the baseline Whisper-small`s WER of 0.219. While limitations remain, the updated pipeline is expected to drastically reduce the human efforts involved.
Exploring YouTube Comments Reacting to Femicide News in Italian (2024)

In recent years, the Gender Based Violence (GBV) has become an important issue in modern society and a central topic in different research areas due to its alarming spread. Several Natural Language Processing (NLP) studies, concerning Hate Speech directed against women, have focused on slurs or incel communities. The main contribution of our work is the creation of the first dataset on social media comments to GBV, in particular to a femicide event. Our dataset, named GBV-Maltesi, contains 2,934 YouTube comments annotated following a new schema that we developed in order to study GBV and misogyny with an intersectional approach. During the experimental phase, we trained models on different corpora for binary misogyny detection and found that datasets that mostly include explicit expressions of misogyny are an easier challenge, compared to more implicit forms of misogyny contained in GVB-Maltesi.