ymoslem/acl-abstracts huggingface dataset:
Transformer-driven Multi-task Learning for Fake and Hateful Content Detection (2024)

Social media has revolutionized communica-tion these days in addition to facilitating thespread of fake and hate content. While fakecontent is the manipulation of facts by disin-formation, hate content is textual violence ordiscrimination targeting a group or an individ-ual. Fake narratives have the potential to spreadhate content making people aggressive or hurt-ing the sentiments of an individual or a group.Further, false narratives often dominate discus-sions on sensitive topics, amplifying harmfulmessages contributing to the rise of hate speech.Hence, understanding the relationship betweenhate speech driven by fake narratives is cru-cial in this digital age making it necessary todevelop automatic tools to identify fake andhate content. In this direction, Decoding FakeNarratives in Spreading Hateful Stories (Faux-Hate) - a shared task organized at the Inter-national Conference on Natural Language Pro-cessing (ICON) 2024, invites researchers totackle both fake and hate detection in socialmedia comments, with additional emphasis onidentifying the target and severity of hatefulspeech. The shared task consists of two sub-tasks - Task A (Identifying fake and hate con-tent) and Task B (Identifying the target andseverity of hateful speech). In this paper, we -team MUCS, describe the models proposed toaddress the challenges of this shared task. Wepropose two models: i) Hing{\_}MTL - a Multi-task Learning (MTL) model implemented us-ing pre-trained Hinglish Bidirectional EncoderRepresentations from Transformers (Hinglish-BERT), and ii) Ensemble{\_}MTL - a MTL modelimplemented by ensembling two pre-trainedmodels (HinglishBERT, and Multilingual Dis-tiled version of BERT (MDistilBERT)), to de-tect fake and hate content and identify the targetand severity of hateful speech. Ensemble{\_}MTLmodel outperformed Hing{\_}MTL model withmacro F1 scores of 0.7589 and 0.5746 for TaskA and Task B respectively, securing 6th placein both subtasks.
Rejected Cookies @ Decoding Faux-Hate: Predicting Fake Narratives and Hateful Content (2024)

This paper reports the results of our team for theICON 2024 shared task Decoding Fake Narra-tives in Spreading Hateful Stories (Faux-Hate).The task aims at classifying tweets in a multi-label and multi-class framework. It comprisestwo subtasks: (A) Binary Faux-Hate Detec-tion, which involves predicting whether a tweetis fake (1/0) and/or hate speech (1/0, and (B)Target and Severity Prediction, which cate-gorizes tweets based on their target (Individ-ual, Organization, Religion) and severity (Low,Medium, High). We evaluated Machine Learn-ing (ML) approaches, including Logistic Re-gression, Support Vector Machines (SVM), andRandom Forest; Deep Learning (DL) methods,such as Artificial Neural Networks (ANN) andBidirectional Encoder Representations fromTransformers (BERT); and innovative quantumhybrid models, like Hybrid Quantum NeuralNetworks (HQNN), for identifying and classi-fying tweets across these subtasks. Our exper-iments trained and compared multiple modelarchitectures to assess their comparative per-formance and detection capabilities in these di-verse modeling strategies.The best-performingmodels achieved F1 scores of 0.72, 0.76, 0.64,and 0.54 for the respective labels Hate, Fake,Target and Severity. We have open-sourced ourimplementation code for both tasks on Github1 .
A Machine Learning Framework for Detecting Hate Speech and Fake Narratives in Hindi-English Tweets (2024)

This paper presents a novel system developed for the Faux-Hate Shared Task at ICON2024, addressing the detection of hate speechand fake narratives within Hindi-English code-mixed social media data. Our approach com-bines advanced text preprocessing, TF-IDFvectorization, and Random Forest classifiersto identify harmful content, while employingSMOTE to address class imbalance. By lever-aging ensemble learning and feature engineer-ing, our system demonstrates robust perfor-mance in detecting hateful and fake content,classifying targets, and evaluating the sever-ity of hate speech. The results underscore thepotential for real-world applications, such asmoderating online platforms and identifyingharmful narratives. Furthermore, we highlightethical considerations for deploying such tools,emphasizing responsible use in sensitive do-mains, thereby advancing research in multilin-gual hate speech detection and online abusemitigation.
Faux-Hate Multitask Framework for Misinformation and Hate Speech Detection in Code-Mixed Languages (2024)

The Faux-Hate task looks at two big issues:misinformation and hate speech. It focuses onHindi-English social media posts. This papershares our methods for both parts of the task.For Task A, we built a special model based onXLM-RoBERTa. It has features that help usspot both fake news and hate speech at the sametime. For Task B, we wanted to identify whothe hate is aimed at (like individuals or groups)and how severe it is (high, medium, low). So,we added different tools to our model for thiskind of sorting. To get ready for all this, wecarefully cleaned the data, especially dealingwith mixing languages and different spellings.In Task A, our results show that our model canclearly tell the difference between fake and realstories, as well as between hate and non-hatespeech. For Task B, it does a good job withidentifying targets and severity levels, givingstrong predictions for multiple labels. Overall,these results show that cross-lingual models,combined with specific tweaks, can really helptackle complex text classification in languageswith fewer resources.
Multi-Task Learning for Faux-Hate Detection in Hindi-English Code-Mixed Text (2024)

The prevalence of harmful internet content is on the rise, especially among young people. Thismakes social media sites breeding grounds forhate speech and negativity even though theirpurpose is to create connections. The study pro-poses a multi-task learning model for the iden-tification and analysis of harmful social mediacontent. This classifies the text into fake/realand hate/non-hate categories and further identi-fies the target and severity of the harmful con-tent. The proposed model showed significantimprovements in performance with training ontransliterated data as compared to code-mixeddata. It ranked 2nd and 3rd in the ICON 2024Faux-Hate Shared Task and the performanceshave made it very effective against harmful con-tent.
LoRA adapter weight tuning with multi-task learning for Faux-Hate detection (2024)

Detecting misinformation and harmful language in bilingual texts, particularly those com-bining Hindi and English, poses considerabledifficulties. The intricacies of mixed-languagecontent and limited available resources compli-cate this task even more. The proposed workfocuses on unraveling deceptive stories thatpropagate hate. We have developed an inno-vative attention-weight-tuned LoRA Adopter-based model for such Faux-Hate content de-tection. This work is conducted as a partof the ICON 2024 shared task on DecodingFake narratives in spreading Hateful stories.The LoRA-enhanced architecture secured 13thplace among the participating teams for TaskA.
Shared Feature-Based Multitask Model for Faux-Hate Classification in Code-Mixed Text (2024)

In recent years, the rise of harmful narratives online has highlighted the need for advancedhate speech detection models. One emergingchallenge is the phenomenon of Faux Hate, anew type of hate speech that originates fromthe intersection of fake narratives and hatespeech. Faux Hate occurs when fabricatedclaims fuel the generation of hateful language,often blurring the line between misinforma-tion and malicious intent. Identifying suchspeech becomes especially difficult when thefake claim itself is not immediately apparent.This paper provides an overview of a sharedtask competition focused on detecting FauxHate, where participants were tasked with de-veloping methodologies to identify this nu-anced form of harmful speech.
Quality and Quantity of Machine Translation References for Automatic Metrics (2024)

Automatic machine translation metrics typically rely on human translations to determine the quality of system translations. Common wisdom in the field dictates that the human references should be of very high quality. However, there are no cost-benefit analyses that could be used to guide practitioners who plan to collect references for machine translation evaluation. We find that higher-quality references lead to better metric correlations with humans at the segment-level. Having up to 7 references per segment and taking their average (or maximum) helps all metrics. Interestingly, the references from vendors of different qualities can be mixed together and improve metric success. Higher quality references, however, cost more to create and we frame this as an optimization problem: given a specific budget, what references should be collected to maximize metric success. These findings can be used by evaluators of shared tasks when references need to be created under a certain budget.
Exploratory Study on the Impact of English Bias of Generative Large Language Models in Dutch and French (2024)

The most widely used LLMs like GPT4 and Llama 2 are trained on large amounts of data, mostly in English but are still able to deal with non-English languages. This English bias leads to lower performance in other languages, especially low-resource ones. This paper studies the linguistic quality of LLMs in two non-English high-resource languages: Dutch and French, with a focus on the influence of English. We first construct a comparable corpus of text generated by humans versus LLMs (GPT-4, Zephyr, and GEITje) in the news domain. We proceed to annotate linguistic issues in the LLM-generated texts, obtaining high inter-annotator agreement, and analyse these annotated issues. We find a substantial influence of English for all models under all conditions: on average, 16{\%} of all annotations of linguistic errors or peculiarities had a clear link to English. Fine-tuning a LLM to a target language (GEITje is fine-tuned on Dutch) reduces the number of linguistic issues and probably also the influence of English. We further find that using a more elaborate prompt leads to linguistically better results than a concise prompt. Finally, increasing the temperature for one of the models leads to lower linguistic quality but does not alter the influence of English.
Adding Argumentation into Human Evaluation of Long Document Abstractive Summarization: A Case Study on Legal Opinions (2024)

Human evaluation remains the gold standard for assessing abstractive summarization. However, current practices often prioritize constructing evaluation guidelines for fluency, coherence, and factual accuracy, overlooking other critical dimensions. In this paper, we investigate argument coverage in abstractive summarization by focusing on long legal opinions, where summaries must effectively encapsulate the document`s argumentative nature. We introduce a set of human-evaluation guidelines to evaluate generated summaries based on argumentative coverage. These guidelines enable us to assess three distinct summarization models, studying the influence of including argument roles in summarization. Furthermore, we utilize these evaluation scores to benchmark automatic summarization metrics against argument coverage, providing insights into the effectiveness of automated evaluation methods.
A Gold Standard with Silver Linings: Scaling Up Annotation for Distinguishing Bosnian, Croatian, Montenegrin and Serbian (2024)

Bosnian, Croatian, Montenegrin and Serbian are the official standard linguistic varieties in Bosnia and Herzegovina, Croatia, Montenegro, and Serbia, respectively. When these four countries were part of the former Yugoslavia, the varieties were considered to share a single linguistic standard. After the individual countries were established, the national standards emerged. Today, a central question about these varieties remains the following: How different are they from each other? How hard is it to distinguish them? While this has been addressed in NLP as part of the task on Distinguishing Between Similar Languages (DSL), little is known about human performance, making it difficult to contextualize system results. We tackle this question by reannotating the existing BCMS dataset for DSL with annotators from all target regions. We release a new gold standard, replacing the original single-annotator, single-label annotation by a multi-annotator, multi-label one, thus improving annotation reliability and explicitly coding the existence of ambiguous instances. We reassess a previously proposed DSL system on the new gold standard and establish the human upper bound on the task. Finally, we identify sources of annotation difficulties and provide linguistic insights into the BCMS dialect continuum, with multiple indicators highlighting an intermediate position of Bosnian and Montenegrin.
Insights of a Usability Study for KBQA Interactive Semantic Parsing: Generation Yields Benefits over Templates but External Validity Remains Challenging (2024)

We present our findings from a usability study of an interactive semantic parsing system for knowledge based question answering (KBQA). The system is designed to help users access information within a knowledge base without having to know its query language. The system translates the user`s question into the query language, retrieves an answer, then presents an English explanation of the process so that the user can make corrections if necessary. To our knowledge, our work is the most thorough usability study conducted for such a system and the only one that uses crowdworkers as participants to verify that the system is usable for average users. Our crowdworkers participate in KBQA dialogues using 4 versions of a system based on the framework by Mo et al. (2022) and answer surveys about their experiences. Some key takeaways from this work are: 1) we provide evidence for the benefits of interactivity in semantic parsing with human users and using generated questions in lieu of templated representations, 2) we identify limitations of simulations and provide contrasting evidence from actual system use, and 3) we provide an examination of crowdsourcing methodology, in particular the trade-offs of using crowdworkers vs. a specially trained group of evaluators.
Extrinsic evaluation of question generation methods with user journey logs (2024)

There is often a significant disparity between the performance of Natural Language Processing (NLP) tools as evaluated on benchmark datasets using metrics like ROUGE or BLEU, and the actual user experience encountered when employing these tools in real-world scenarios. This highlights the critical necessity for user-oriented studies aimed at evaluating user experience concerning the effectiveness of developed methodologies. A primary challenge in such {\textquotedblleft}ecological{\textquotedblright} user studies is their assessment of specific configurations of NLP tools, making replication under identical conditions impractical. Consequently, their utility is limited for the automated evaluation and comparison of different configurations of the same tool. The objective of this study is to conduct an {\textquotedblleft}ecological{\textquotedblright} evaluation of a question generation within the context of an external task involving document linking. To do this we conducted an ''\textit{ecological}'' evaluation of a document linking tool in the context of the exploration of a Social Science archives and from this evaluation, we aim to derive a form of a {\textquotedblleft}reference corpus{\textquotedblright} that can be used offline for the automated comparison of models and quantitative tool assessment. This corpus is available on the following link: https://gitlab.lis-lab.fr/archival-public/autogestion-qa-linking
Towards Holistic Human Evaluation of Automatic Text Simplification (2024)

Text simplification refers to the process of rewording within a single language, moving from a standard form into an easy-to-understand one. Easy Language and Plain Language are two examples of simplified varieties aimed at improving readability and understanding for a wide-ranging audience. Human evaluation of automatic text simplification is usually done by employing experts or crowdworkers to rate the generated texts. However, this approach does not include the target readers of simplified texts and does not reflect actual comprehensibility. In this paper, we explore different ways of measuring the quality of automatically simplified texts. We conducted a multi-faceted evaluation study involving end users, post-editors, and Easy Language experts and applied a variety of qualitative and quantitative methods. We found differences in the perception and actual comprehension of the texts by different user groups. In addition, qualitative surveys and behavioral observations proved to be essential in interpreting the results.
Decoding the Metrics Maze: Navigating the Landscape of Conversational Question Answering System Evaluation in Procedural Tasks (2024)

Conversational systems are widely used for various tasks, from answering general questions to domain-specific procedural tasks, such as cooking. While the effectiveness of metrics for evaluating general question answering (QA) tasks has been extensively studied, the evaluation of procedural QA remains a challenge as we do not know what answer types users prefer in such tasks. Existing studies on metrics evaluation often focus on general QA tasks and typically limit assessments to one answer type, such as short, SQuAD-like responses or longer passages. This research aims to achieve two objectives. Firstly, it seeks to identify the desired traits of conversational QA systems in procedural tasks, particularly in the context of cooking (RQ1). Second, it assesses how commonly used conversational QA metrics align with these traits and perform across various categories of correct and incorrect answers (RQ2). Our findings reveal that users generally favour concise conversational responses, except in time-sensitive scenarios where brief, clear answers hold more value (e.g. when heating in oil). While metrics effectively identify inaccuracies in short responses, several commonly employed metrics tend to assign higher scores to incorrect conversational answers when compared to correct ones. We provide a selection of metrics that reliably detect correct and incorrect information in short and conversational answers.
The 2024 ReproNLP Shared Task on Reproducibility of Evaluations in NLP: Overview and Results (2024)

This paper presents an overview of, and the results from, the 2024 Shared Task on Reproducibility of Evaluations in NLP (ReproNLP`24), following on from three previous shared tasks on reproducibility of evaluations in NLP, ReproNLP`23, ReproGen`22 and ReproGen`21. This shared task series forms part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP and machine learning, against a backdrop of increasing recognition of the importance of reproducibility across the two fields. We describe the ReproNLP`24 shared task, summarise results from the reproduction studies submitted, and provide additional comparative analysis of their results.
Once Upon a Replication: It is Humans' Turn to Evaluate AI`s Understanding of Children`s Stories for QA Generation (2024)

The following paper presents the outcomes of a collaborative experiment on human evaluation from the ReproNLP 2024 shared task, track B, part of the ReproHum project. For this paper, we evaluated a QAG (question-answer generation) system centered on English children`s storybooks that was presented in a previous research, by using human evaluators for the study. The system generated relevant QA (Question-Answer) pairs based on a dataset with storybooks for early education (kindergarten up to middle school) called FairytaleQA. In the framework of the ReproHum project, we first outline the previous paper and the reproduction strategy that has been decided upon. The complete setup of the first human evaluation is then described, along with the modifications required to replicate it. We also add other relevant related works on this subject. In conclusion, we juxtapose the replication outcomes with those documented in the cited publication. Additionally, we explore the general features of this endeavor as well as its shortcomings.
Exploring Reproducibility of Human-Labelled Data for Code-Mixed Sentiment Analysis (2024)

Growing awareness of a {\textquoteleft}Reproducibility Crisis' in natural language processing (NLP) has focused on human evaluations of generative systems. While labelling for supervised classification tasks makes up a large part of human input to systems, the reproduction of such efforts has thus far not been been explored. In this paper, we re-implement a human data collection study for sentiment analysis of code-mixed Malayalam movie reviews, as well as automated classification experiments. We find that missing and under-specified information makes reproduction challenging, and we observe potentially consequential differences between the original labels and those we collect. Classification results indicate that the reliability of the labels is important for stable performance.
Reproducing the Metric-Based Evaluation of a Set of Controllable Text Generation Techniques (2024)

Rerunning a metric-based evaluation should be more straightforward and results should be closer than in a human-based evaluation, especially where code and model checkpoints are made available by the original authors. As this brief report of our efforts to rerun a metric-based evaluation of a set of multi-aspect controllable text generation (CTG) techniques shows however, such reruns of evaluations do not always produce results that are the same as the original results, and can reveal errors in the orginal work.
ReproHum: \#0033-03: How Reproducible Are Fluency Ratings of Generated Text? A Reproduction of August et al. 2022 (2024)

In earlier work, August et al. (2022) evaluated three different Natural Language Generation systems on their ability to generate fluent, relevant, and factual scientific definitions. As part of the ReproHum project (Belz et al., 2023), we carried out a partial reproduction study of their human evaluation procedure, focusing on human fluency ratings. Following the standardised ReproHum procedure, our reproduction study follows the original study as closely as possible, with two raters providing 300 ratings each. In addition to this, we carried out a second study where we collected ratings from eight additional raters and analysed the variability of the ratings. We successfully reproduced the inferential statistics from the original study (i.e. the same hypotheses were supported), albeit with a lower inter-annotator agreement. The remainder of our paper shows significant variation between different raters, raising questions about what it really means to reproduce human evaluation studies.