ymoslem/acl-abstracts huggingface dataset:
Towards Robust and Multilingual Task-Oriented Dialogue Systems (2024)

In this position paper, I present my research interests regarding the field of task-oriented dialogue systems. My work focuses on two main aspects: optimizing the task completion ability of dialogue systems using reinforcement learning, and developing language resources and exploring multilinguality to support the advancement of dialogue systems across different languages. I discuss the limitations of current approaches in achieving robust task completion performance and propose a novel optimization approach called Post-Processing Networks. Furthermore, I highlight the importance of multilingual dialogue datasets and describe our work on constructing JMultiWOZ, the first large-scale Japanese task-oriented dialogue dataset.
Toward Faithful Dialogs: Evaluating and Improving the Faithfulness of Dialog Systems (2024)

My primary research interests lie in evaluating and improving the faithfulness of language model-based text generation systems. Recent advances in large language models (LLMs) such as GPT-4 and Llama have enabled the wide adoption of LLMs in various aspects of natural language processing (NLP). Despite their widespread use, LLMs still suffer from the problem of hallucination, limiting the practicality of deploying such systems in use cases where being factual and faithful is of critical importance. My research specifically aims to evaluate and improve the faithfulness, i.e. the factual alignment between the generated text and a given context, of text generation systems. By developing techniques to reliably evaluate, label, and improve generation faithfulness, we can enable wider adoption of dialog systems that need to converse with human users using accurate information.
Cognitive Model of Listener Response Generation and Its Application to Dialogue Systems (2024)

In this position paper, we introduce our efforts in modeling listener response generation and its application to dialogue systems. We propose that the cognitive process of generating listener responses involves four levels: attention level, word level, propositional information level, and activity level, with different types of responses used depending on the level. Attention level responses indicate that the listener is listening to and paying attention to the speaker`s speech. Word-level responses demonstrate the listener`s knowledge or understanding of a single representation. Propositional information level responses indicate the listener`s understanding, empathy, and emotions towards a single propositional information. Activity level responses are oriented towards activities. Additionally, we briefly report on our current initiative in generating propositional information level responses using a knowledge graph and LLMs.
Topological Deep Learning for Term Extraction (2024)

Ben is a postdoctoral researcher in the Dialog Systems and Machine Learning research group led by Milica Ga{\v{s}}i{\'c} at the Heinrich-Heine-Universit{\"a}t D{\"u}sseldorf, which he joined in 2022. In collaboration with the Topology and Geometry group in the Mathematics Department, under the supervision of Marcus Zibrowius, Ben is developing applications of Topological Data Analysis in Natural Language Processing, focusing on dialogue systems. Before transitioning to machine learning research, Ben was a pure mathematician at the Max-Planck-Institute for Mathematics in Bonn, where he specialized in knotted surfaces in 4-dimensional manifolds. He graduated from the University of Bonn in 2022.
Dialogue Management with Graph-structured Knowledge (2024)

I am a postdoctoral researcher at Otto-Friedrich University of Bamberg, and my research interests include the knowledge-grounded dialogue systems, logical rule-based reasoning for dialogue management, and human-robot interaction.
Towards a Co-creation Dialogue System (2024)

In this position paper, I present my research interests in dialogue systems, where the user and the system collaboratively work on tasks through conversation. My work involves analyzing dialogues in which two parties collaborate through conversation, focusing on tasks that yield outcomes with no single correct answer. To support this research, I have created a tagline co-writing dialogue corpus, which I have analyzed from various perspectives. Additionally, I developed a prototype for a tagline co-writing dialogue system.
Enhancing Decision-Making with AI Assistance (2024)

My research interests broadly lie in the influence of artificial intelligence (AI) agents on human decision-making. Specifically, I aim to develop applications for conversational agents in decision-making support. During my master`s program, I developed a system that uses an interview dialogue system to support user review writing. In this approach, the conversational agent gathers product information such as users' impressions and opinions during the interview, to create reviews, facilitating the review writing process. Additionally, I conducted a comprehensive evaluation from the perspectives of system users and review readers. Although experimental results have shown that the system is capable of generating helpful reviews, the quality of the reviews still depends on how effectively the agent elicits the information from users. Therefore, I believe that personalizing the agent`s interview strategy to users' preferences regarding the review writing process can further enhance both the user experience and the helpfulness of the review.
Ontology Construction for Task-oriented Dialogue (2024)

My research interests lie generally in dialogue ontology construction, that uses techniques from information extraction to extract relevant terms from task-oriented dialogue data and order them by finding hierarchical relations between them.
Generalized Visual-Language Grounding with Complex Language Context (2024)

My research focus on \textbf{Visual Dialogues} and \textbf{Generalized Visual-Language Grounding with Complex Language Context}. Specifically, my research aim to utilize Large Language Models (LLMs) to build \textit{conversational agents capable of comprehending and responding to visual cues}. Visual-Language Pre-trained (VLP) models, primarily utilizing transformer-based encoder-decoder architectures, are extensively employed across a range of visual-language tasks, such as visual question answering (VQA) and referring expression comprehension (REC). The effectiveness of these models stems from their robust visual-language integration capabilities. However, their performance is constrained in more complex applications like multimodal conversational agents, where intricate and extensive language contexts pose significant challenges. These tasks demands language-only reasoning before engaging in multimodal fusion. In response, my research investigates the application of Large Language Models (LLMs) with advance comprehension and generation capabilities to enhance performance in complex multimodal tasks, particularly multimodal dialogues. In brief, my work in visual dialogues revolves around two major research questions. i) How to redefine visually grounded conversational agent architectures to benefit from LLMs ii) How to transfer the large body of knowledge encoded in LLMs to conversational systems.
Towards a Real-Time Multimodal Emotion Estimation Model for Dialogue Systems (2024)

This position paper presents my research interest in establishing human-like chat-oriented dialogue systems. To this end, my work focuses on two main areas: the construction and utilization of multimodal datasets and real-time multimodal affective computing. I discuss the limitations of current multimodal dialogue corpora and multimodal affective computing models. As a solution, I have constructed a human-human dialogue dataset containing various synchronized multimodal information, and I have conducted preliminary analyses on it. In future work, I will further analyze the collected data and build a real-time multimodal emotion estimation model for dialogue systems.
Innovative Approaches to Enhancing Safety and Ethical AI Interactions in Digital Environments (2024)

Ensuring safe online environments is a formidable challenge, but nonetheless an important one as people are now chronically online. The increasing online presence of people paired with the prevalence of harmful content such as toxicity, hate speech, misinformation and disinformation across various social media platforms and within different video calls for stronger detection and prevention methods. My research interests primarily lie in applied natural language processing for social good. Previously, I focused on measuring partisan polarization on social media during the COVID-19 pandemic and its societal impacts. Currently, at Ubisoft La Forge, I am dedicated to enhancing player safety within in-game chat systems by developing methods to detect toxicity, evaluating the biases in these detection systems, and assessing the current ecological state of online interactions. Additionally, I am engaged in simulating social media environments using LLMs to ethically test detection methods, evaluate the effectiveness of current mitigation strategies, and potentially introduce new, successful strategies. My suggested topics for discussion: 1. Understanding and mitigating social harms through high fidelity simulated social media environments 2. Enhancing safety in online environments such as within in-game chats (text and speech) 3. Personification of LLM agents 4. Ethically simulating social media sandbox environments at scale with LLM agents 5. Re-balancing the playing field between good and bad actors: Strategies for countering societal-scale manipulation.
Leveraging Linguistic Structural Information for Improving the Model`s Semantic Understanding Ability (2024)

This position paper describes research interests of the author (semantic structure comprehension in multimodal dialogue environments), his point of view on Spoken Dialogue System research that a new wave is to be carried out for coexistence with LLMs, and discussion topic proposals. Those three topics are as follows: 1) How to keep up with, or manipulate LLM for academia research, 2) How representational languages for semantic structural information could be used in this new era, and 3) how to deal with disambiguating the user`s language during the actual dialogue scenario.
Multi-User Dialogue Systems and Controllable Language Generation (2024)

My research interests include multi-user dialogue systems with a focus on user modelling and the development of moderation strategies. Contemporary Spoken Dialogue Systems (SDSs) frequently lack the ability to deal with more than one user simultaneously. Moreover, I am interested in researching on the Controllability of Language Generation using Large Language Models (LLMs). Our hypothesis is that an integration of explicit dialogue control signals improves the Controllability and Reliability of generated sequences independently of the underlying LLM.
Enhancing Role-Playing Capabilities in Persona Dialogue Systems through Corpus Construction and Evaluation Methods (2024)

My research interest involves persona dialogue systems, which use the profile information of a character or real person, called a persona, and responds accordingly. Persona dialogue systems can improve the consistency of the system`s responses, users' trust, and user enjoyment. My current research focuses on persona dialogue systems, especially dialogue agents that role-play as fictional characters. The first task involves obtaining the dialogue and personas of novel characters and building a dialogue corpus. The second task involves evaluating whether the dialogue agent`s responses are character-like relative to the context. The goal of these studies is to allow dialogue agents to generate responses that are more character-like.
Character Expression and User Adaptation for Spoken Dialogue Systems (2024)

The author is interested in building dialogue systems with character and user adaptation. The goal is to create a dialogue system capable of establishing deeper relationships with users. To build a trustful relationship with users, it is important for the system to express its character. The author particularly aims to convey the system`s character through multimodal behavior. Users currently try to speak clearly to avoid speech recognition errors when interacting with SDSs. However, it is necessary to develop SDSs that allow users to converse naturally, as if they were speaking with a human. The author focused on user adaptation by considering user personality. In particular, the author proposes a system that adjusts its manner of speaking according to the user`s personality. Furthermore, the author is interested not only in adjusting the system`s speaking style to match the user but also in making the system`s listening style more conducive to natural conversation.
Interactive Explanations through Dialogue Systems (2024)

The growing need for transparency in AI systems has led to the increased popularity of explainable AI (XAI), with dialogue systems emerging as a promising approach to provide dynamic and interactive explanations. To overcome the limitations of non-conversational XAI methods, we proposed and implemented a generic dialogue architecture that integrates domain-specific knowledge, enhancing user comprehension and interaction. By incorporating computational argumentation and argumentative tree structures into our prototype, we found a positive impact on the dialogue`s effectiveness. In future research, we plan to improve Natural Language Understanding (NLU) to reduce error rates and better interpret user queries, and to advance Natural Language Generation (NLG) techniques for generating more fluid and contextually appropriate responses using large language models. Additionally, we will refine argument annotation to enable better selection and presentation of information, ensuring the system provides the most relevant and coherent explanations based on user needs. Over the next 5 to 10 years, we anticipate significant advancements in dialogue systems' flexibility, personalization, and cultural adaptability, driven by large language models and open domain dialogues. These developments will enhance global communication, user satisfaction, and the effectiveness of virtual assistants across various applications while addressing ethical and social implications.
Towards Emotion-aware Task-oriented Dialogue Systems in the Era of Large Language Models (2024)

My research interests lie in the area of \textbf{modelling affective behaviours of interlocutors in conversations}. In particular, I look at emotion perception, expression, and management in information-retrieval task-oriented dialogue (ToD) systems. Traditionally, ToD systems focus primarily on fulfilling the user`s goal by requesting and providing appropriate information. Yet, in real life, the user`s emotional experience also contributes to the overall satisfaction. This requires the system`s ability to recognise, manage, and express emotions. To this end, I incorporated emotion in the entire ToD system pipeline (Feng et al., 2024, to appear in SIGDIAL 2024). In addition, in the era of large language models (LLMs), emotion recognition and generation have been made easy even under a zero-shot set-up (Feng et al., 2023; Stricker and Paroubek, 2024). Therefore, I am also interested in building ToD systems with LLMs and examining various types of affect in other ToD set-ups such as depression detection in clinical consultations.
Utilizing Large Language Models for Customized Dialogue Data Augmentation and Psychological Counseling (2024)

Large language models (LLMs), such as GPT-4, have driven significant technological advances in spoken dialogue systems (SDSs). In the era of LLMs, my research focuses on: (1) employing these models for customized dialogue data augmentation to improve SDS adaptability to various speaking styles, and (2) utilizing LLMs to support counselors with psychological counseling dialogues. In the future, I aim to integrate these themes, applying user adaptability to psychological counseling dialogues to facilitate smoother conversations.
Toward More Human-like SDSs: Advancing Emotional and Social Engagement in Embodied Conversational Agents (2024)

The author`s research advances human-AI interaction across two innovative domains to enhance the depth and authenticity of communication. Through Emotional Validation, which leverages psychotherapeutic techniques, the research enriches SDSs with advanced capabilities for understanding and responding to human emotions. On the other hand, while utilizing Embodied Conversational Agents (ECAs), the author focuses on developing agents that simulate sophisticated human social behaviors, enhancing their ability to engage in context-sensitive and personalized dialogue. Together, these initiatives aim to transform SDSs and ECAs into empathetic, embodied companions, pushing the boundaries of conversational AI.
Investigating radicalisation indicators in online extremist communities (2024)

We identify and analyse three sociolinguistic indicators of radicalisation within online extremist forums: hostility, longevity and social connectivity. We develop models to predict the maximum degree of each indicator measured over an individual`s lifetime, based on a minimal number of initial interactions. Drawing on data from two diverse extremist communities, our results demonstrate that NLP methods are effective at prioritising at-risk users. This work offers practical insights for intervention strategies and policy development, and highlights an important but under-studied research direction.
Detection of Conspiracy Theories Beyond Keyword Bias in German-Language Telegram Using Large Language Models (2024)

The automated detection of conspiracy theories online typically relies on supervised learning. However, creating respective training data requires expertise, time and mental resilience, given the often harmful content. Moreover, available datasets are predominantly in English and often keyword-based, introducing a token-level bias into the models. Our work addresses the task of detecting conspiracy theories in German Telegram messages. We compare the performance of supervised fine-tuning approaches using BERT-like models with prompt-based approaches using Llama2, GPT-3.5, and GPT-4 which require little or no additional training data. We use a dataset of {\ensuremath{\sim}}4, 000 messages collected during the COVID-19 pandemic, without the use of keyword filters.Our findings demonstrate that both approaches can be leveraged effectively: For supervised fine-tuning, we report an F1 score of {\ensuremath{\sim}} 0.8 for the positive class, making our model comparable to recent models trained on keyword-focused English corpora. We demonstrate our model`s adaptability to intra-domain temporal shifts, achieving F1 scores of {\ensuremath{\sim}}0.7. Among prompting variants, the best model is GPT-4, achieving an F1 score of {\ensuremath{\sim}}0.8 for the positive class in a zero-shot setting and equipped with a custom conspiracy theory definition.
EkoHate: Abusive Language and Hate Speech Detection for Code-switched Political Discussions on Nigerian Twitter (2024)

Nigerians have a notable online presence and actively discuss political and topical matters. This was particularly evident throughout the 2023 general election, where Twitter was used for campaigning, fact-checking and verification, and even positive and negative discourse. However, little or none has been done in the detection of abusive language and hate speech in Nigeria. In this paper, we curated code-switched Twitter data directed at three musketeers of the governorship election on the most populous and economically vibrant state in Nigeria; Lagos state, with the view to detect offensive speech in political discussions. We developed EkoHate{---}an abusive language and hate speech dataset for political discussions between the three candidates and their followers using a binary (normal vs offensive) and fine-grained four-label annotation scheme. We analysed our dataset and provided an empirical evaluation of state-of-the-art methods across both supervised and cross-lingual transfer learning settings. In the supervised setting, our evaluation results in both binary and four-label annotation schemes show that we can achieve 95.1 and 70.3 F1 points respectively. Furthermore, we show that our dataset adequately transfers very well to three publicly available offensive datasets (OLID, HateUS2020, and FountaHate), generalizing to political discussions in other regions like the US.
A Study of the Class Imbalance Problem in Abusive Language Detection (2024)

Abusive language detection has drawn increasing interest in recent years. However, a less systematically explored obstacle is label imbalance, i.e., the amount of abusive data is much lower than non-abusive data, leading to performance issues. The aim of this work is to conduct a comprehensive comparative study of popular methods for addressing the class imbalance issue. We explore 10 well-known approaches on 8 datasets with distinct characteristics: binary or multi-class, moderately or largely imbalanced, focusing on various types of abuse, etc. Additionally, we pro-pose two novel methods specialized for abuse detection: AbusiveLexiconAug and ExternalDataAug, which enrich the training data using abusive lexicons and external abusive datasets, respectively. We conclude that: 1) our AbusiveLexiconAug approach, random oversampling, and focal loss are the most versatile methods on various datasets; 2) focal loss tends to yield peak model performance; 3) oversampling and focal loss provide promising results for binary datasets and small multi-class sets, while undersampling and weighted cross-entropy are more suitable for large multi-class sets; 4) most methods are sensitive to hyperparameters, yet our suggested choice of hyperparameters provides a good starting point.
HausaHate: An Expert Annotated Corpus for Hausa Hate Speech Detection (2024)

We introduce the first expert annotated corpus of Facebook comments for Hausa hate speech detection. The corpus titled HausaHate comprises 2,000 comments extracted from Western African Facebook pages and manually annotated by three Hausa native speakers, who are also NLP experts. Our corpus was annotated using two different layers. We first labeled each comment according to a binary classification: offensive versus non-offensive. Then, offensive comments were also labeled according to hate speech targets: race, gender and none. Lastly, a baseline model using fine-tuned LLM for Hausa hate speech detection is presented, highlighting the challenges of hate speech detection tasks for indigenous languages in Africa, as well as future advances.
VIDA: The Visual Incel Data Archive. A Theory-oriented Annotated Dataset To Enhance Hate Detection Through Visual Culture (2024)

Images increasingly constitute a larger portion of internet content, encoding even more complex meanings. Recent studies have highlight the pivotal role of visual communication in the spread of extremist content, particularly that associated with right-wing political ideologies. However, the capability of machine learning systems to recognize such meanings, sometimes implicit, remains limited. To enable future research in this area, we introduce and release VIDA, the Visual Incel Data Archive, a multimodal dataset comprising visual material and internet memes collected from two main Incel communities (Italian and Anglophone) known for their extremist misogynistic content. Following the analytical framework of Shifman (2014), we propose a new taxonomy for annotation across three main levels of analysis: content, form, and stance (hate). This allows for the association of images with fine-grained contextual information that help to identify the presence of offensiveness and a broader set of cultural references, enhancing the understanding of more nuanced aspects in visual communication. In this work we present a statistical analysis of the annotated dataset as well as discuss annotation examples and future line of research.