pwc-archive/papers-with-abstracts huggingface dataset:
Augmenting Topic Aware Knowledge-Grounded Conversations with Dynamic Built Knowledge Graphs
Authors: Junjie Wu, Hao Zhou
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Knowledge Graphs, Management, Response Generation

Dialog topic management and background knowledge selection are essential factors for the success of knowledge-grounded open-domain conversations. However, existing models are primarily performed with symmetric knowledge bases or stylized with pre-defined roles between conversational partners, while people usually have their own knowledge before a real chit-chat. To address this problem, we propose a dynamic knowledge graph-based topical conversation model (DKGT). Given a dialog history context, our model first builds knowledge graphs from the context as an imitation of human’s ability to form logical relationships between known and unknown topics during a conversation. This logical information will be fed into a topic predictor to promote topic management, then facilitate background knowledge selection and response generation. To the best of our knowledge, this is the first attempt to dynamically form knowledge graphs between chatting topics to assist dialog topic management during a conversation. Experimental results manifest that our model can properly schedule conversational topics and pick suitable knowledge to generate informative responses comparing to several strong baselines.
What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity
Authors: Alessio Miaschi, Dominique Brunato, Felice Dell’Orletta, Giulia Venturi
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Sentence

This paper presents an investigation aimed at studying how the linguistic structure of a sentence affects the perplexity of two of the most popular Neural Language Models (NLMs), BERT and GPT-2. We first compare the sentence-level likelihood computed with BERT and the GPT-2’s perplexity showing that the two metrics are correlated. In addition, we exploit linguistic features capturing a wide set of morpho-syntactic and syntactic phenomena showing how they contribute to predict the perplexity of the two NLMs.
How Do BERT Embeddings Organize Linguistic Knowledge?
Authors: Giovanni Puccetti, Alessio Miaschi, Felice Dell’Orletta
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Sentence

Several studies investigated the linguistic information implicitly encoded in Neural Language Models. Most of these works focused on quantifying the amount and type of information available within their internal representations and across their layers. In line with this scenario, we proposed a different study, based on Lasso regression, aimed at understanding how the information encoded by BERT sentence-level representations is arrange within its hidden units. Using a suite of several probing tasks, we showed the existence of a relationship between the implicit knowledge learned by the model and the number of individual units involved in the encodings of this competence. Moreover, we found that it is possible to identify groups of hidden units more relevant for specific linguistic properties.
ERNIE-NLI: Analyzing the Impact of Domain-Specific External Knowledge on Enhanced Representations for NLI
Authors: Lisa Bauer, Lingjia Deng, Mohit Bansal
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Language Modeling, Language Modelling, Natural Language Inference

We examine the effect of domain-specific external knowledge variations on deep large scale language model performance. Recent work in enhancing BERT with external knowledge has been very popular, resulting in models such as ERNIE (Zhang et al., 2019a). Using the ERNIE architecture, we provide a detailed analysis on the types of knowledge that result in a performance increase on the Natural Language Inference (NLI) task, specifically on the Multi-Genre Natural Language Inference Corpus (MNLI). While ERNIE uses general TransE embeddings, we instead train domain-specific knowledge embeddings and insert this knowledge via an information fusion layer in the ERNIE architecture, allowing us to directly control and analyze knowledge input. Using several different knowledge training objectives, sources of knowledge, and knowledge ablations, we find a strong correlation between knowledge and classification labels within the same polarity, illustrating that knowledge polarity is an important feature in predicting entailment. We also perform classification change analysis across different knowledge variations to illustrate the importance of selecting appropriate knowledge input regarding content and polarity, and show representative examples of these changes.
Enhancing Multiple-Choice Question Answering with Causal Knowledge
Authors: Dhairya Dalal, Mihael Arcan, Paul Buitelaar
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Multiple-choice, Question Answering

The task of causal question answering aims to reason about causes and effects over a provided real or hypothetical premise. Recent approaches have converged on using transformer-based language models to solve question answering tasks. However, pretrained language models often struggle when external knowledge is not present in the premise or when additional context is required to answer the question. To the best of our knowledge, no prior work has explored the efficacy of augmenting pretrained language models with external causal knowledge for multiple-choice causal question answering. In this paper, we present novel strategies for the representation of causal knowledge. Our empirical results demonstrate the efficacy of augmenting pretrained models with external causal knowledge. We show improved performance on the COPA (Choice of Plausible Alternatives) and WIQA (What If Reasoning Over Procedural Text) benchmark tasks. On the WIQA benchmark, our approach is competitive with the state-of-the-art and exceeds it within the evaluation subcategories of In-Paragraph and Out-of-Paragraph perturbations.
KW-ATTN: Knowledge Infused Attention for Accurate and Interpretable Text Classification
Authors: Hyeju Jang, Seojin Bang, Wen Xiao, Giuseppe Carenini, Raymond Ng, Young ji Lee
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Classification, text-classification, Text Classification

Text classification has wide-ranging applications in various domains. While neural network approaches have drastically advanced performance in text classification, they tend to be powered by a large amount of training data, and interpretability is often an issue. As a step towards better accuracy and interpretability especially on small data, in this paper we present a new knowledge-infused attention mechanism, called KW-ATTN (KnoWledge-infused ATTentioN) to incorporate high-level concepts from external knowledge bases into Neural Network models. We show that KW-ATTN outperforms baseline models using only words as well as other approaches using concepts by classification accuracy, which indicates that high-level concepts help model prediction. Furthermore, crowdsourced human evaluation suggests that additional concept information helps interpretability of the model.
Multi-input Recurrent Independent Mechanisms for leveraging knowledge sources: Case studies on sentiment analysis and health text mining
Authors: Parsa Bagherzadeh, Sabine Bergler
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Sentiment Analysis

This paper presents a way to inject and leverage existing knowledge from external sources in a Deep Learning environment, extending the recently proposed Recurrent Independent Mechnisms (RIMs) architecture, which comprises a set of interacting yet independent modules. We show that this extension of the RIMs architecture is an effective framework with lower parameter implications compared to purely fine-tuned systems.
What BERTs and GPTs know about your brand? Probing contextual language models for affect associations
Authors: Vivek Srivastava, Stephen Pilli, Savita Bhat, Niranjan Pedanekar, Shirish Karande
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Attribute, Marketing, Question Answering, Recommendation Systems, Sentiment Analysis

Investigating brand perception is fundamental to marketing strategies. In this regard, brand image, defined by a set of attributes (Aaker, 1997), is recognized as a key element in indicating how a brand is perceived by various stakeholders such as consumers and competitors. Traditional approaches (e.g., surveys) to monitor brand perceptions are time-consuming and inefficient. In the era of digital marketing, both brand managers and consumers engage with a vast amount of digital marketing content. The exponential growth of digital content has propelled the emergence of pre-trained language models such as BERT and GPT as essential tools in solving myriads of challenges with textual data. This paper seeks to investigate the extent of brand perceptions (i.e., brand and image attribute associations) these language models encode. We believe that any kind of bias for a brand and attribute pair may influence customer-centric downstream tasks such as recommender systems, sentiment analysis, and question-answering, e.g., suggesting a specific brand consistently when queried for innovative products. We use synthetic data and real-life data and report comparison results for five contextual LMs, viz. BERT, RoBERTa, DistilBERT, ALBERT and BART.
Predicting Numerals in Natural Language Text Using a Language Model Considering the Quantitative Aspects of Numerals
Authors: Taku Sakamoto, Akiko Aizawa
Date: 2021-06-01 00:00:00
Proceeding: NAACL (DeeLIO) 2021 6
Tasks: Common Sense Reasoning, Language Modeling, Language Modelling

Numerical common sense (NCS) is necessary to fully understand natural language text that includes numerals. NCS is knowledge about the numerical features of objects in text, such as size, weight, or color. Existing neural language models treat numerals in a text as string tokens in the same way as other words. Therefore, they cannot reflect the quantitative aspects of numerals in the training process, making it difficult to learn NCS. In this paper, we measure the NCS acquired by existing neural language models using a masked numeral prediction task as an evaluation task. In this task, we use two evaluation metrics to evaluate the language models in terms of the symbolic and quantitative aspects of the numerals, respectively. We also propose methods to reflect not only the symbolic aspect but also the quantitative aspect of numerals in the training of language models, using a loss function that depends on the magnitudes of the numerals and a regression model for the masked numeral prediction task. Finally, we quantitatively evaluate our proposed approaches on four datasets with different properties using the two metrics. Compared with methods that use existing language models, the proposed methods reduce numerical absolute errors, although exact match accuracy was reduced. This result confirms that the proposed methods, which use the magnitudes of the numerals for model training, are an effective way for models to capture NCS.
Towards Hybrid Human-Machine Workflow for Natural Language Generation
Authors: Neslihan Iskender, Tim Polzehl, Sebastian Möller
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4
Tasks: Text Generation

In recent years, crowdsourcing has gained much attention from researchers to generate data for the Natural Language Generation (NLG) tools or to evaluate them. However, the quality of crowdsourced data has been questioned repeatedly because of the complexity of NLG tasks and crowd workers’ unknown skills. Moreover, crowdsourcing can also be costly and often not feasible for large-scale data generation or evaluation. To overcome these challenges and leverage the complementary strengths of humans and machine tools, we propose a hybrid human-machine workflow designed explicitly for NLG tasks with real-time quality control mechanisms under budget constraints. This hybrid methodology is a powerful tool for achieving high-quality data while preserving efficiency. By combining human and machine intelligence, the proposed workflow decides dynamically on the next step based on the data from previous steps and given constraints. Our goal is to provide not only the theoretical foundations of the hybrid workflow but also to provide its implementation as open-source in future work.
Spellchecking for Children in Web Search: a Natural Language Interface Case-study
Authors: Casey Kennington, Jerry Alan Fails, Katherine Landau Wright, Maria Soledad Pera
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4
Tasks: Information Retrieval, Retrieval

Given the more widespread nature of natural language interfaces, it is increasingly important to understand who are accessing those interfaces, and how those interfaces are being used. In this paper, we explore spellchecking in the context of web search with children as the target audience. In particular, via a literature review we show that, while widely used, popular search tools are ill-designed for children. We then use spellcheckers as a case study to highlight the need for an interdisciplinary approach that brings together natural language processing, education, human-computer interaction to address a known information retrieval problem: query misspelling. We conclude that it is imperative that those for whom the interfaces are designed have a voice in the design process.
Capturing Covertly Toxic Speech via Crowdsourcing
Authors: Alyssa Lees, Daniel Borkan, Ian Kivlichan, Jorge Nario, Tesh Goyal
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4

We study the task of labeling covert or veiled toxicity in online conversations. Prior research has highlighted the difficulty in creating language models that recognize nuanced toxicity such as microaggressions. Our investigations further underscore the difficulty in parsing such labels reliably from raters via crowdsourcing. We introduce an initial dataset, COVERTTOXICITY, which aims to identify and categorize such comments from a refined rater template. Finally, we fine-tune a comment-domain BERT model to classify covertly offensive comments and compare against existing baselines.
Towards Human-Centered Summarization: A Case Study on Financial News
Authors: Tatiana Passali, Alexios Gidiotis, Efstathios Chatzikyriakidis, Grigorios Tsoumakas
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4
Tasks: Text Summarization

Recent Deep Learning (DL) summarization models greatly outperform traditional summarization methodologies, generating high-quality summaries. Despite their success, there are still important open issues, such as the limited engagement and trust of users in the whole process. In order to overcome these issues, we reconsider the task of summarization from a human-centered perspective. We propose to integrate a user interface with an underlying DL model, instead of tackling summarization as an isolated task from the end user. We present a novel system, where the user can actively participate in the whole summarization process. We also enable the user to gather insights into the causative factors that drive the model’s behavior, exploiting the self-attention mechanism. We focus on the financial domain, in order to demonstrate the efficiency of generic DL models for domain-specific applications. Our work takes a first step towards a model-interface co-design approach, where DL models evolve along user needs, paving the way towards human-computer text summarization interfaces.
Context-sensitive evaluation of automatic speech recognition: considering user experience & language variation
Authors: Nina Markl, Catherine Lai
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

Commercial Automatic Speech Recognition (ASR) systems tend to show systemic predictive bias for marginalised speaker/user groups. We highlight the need for an interdisciplinary and context-sensitive approach to documenting this bias incorporating perspectives and methods from sociolinguistics, speech & language technology and human-computer interaction in the context of a case study. We argue evaluation of ASR systems should be disaggregated by speaker group, include qualitative error analysis, and consider user experience in a broader sociolinguistic and social context.
Embodied Multimodal Agents to Bridge the Understanding Gap
Authors: Nikhil Krishnaswamy, Nada Alalyani
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4

In this paper we argue that embodied multimodal agents, i.e., avatars, can play an important role in moving natural language processing toward “deep understanding.” Fully-featured interactive agents, model encounters between two “people,” but a language-only agent has little environmental and situational awareness. Multimodal agents bring new opportunities for interpreting visuals, locational information, gestures, etc., which are more axes along which to communicate. We propose that multimodal agents, by facilitating an embodied form of human-computer interaction, provide additional structure that can be used to train models that move NLP systems closer to genuine “understanding” of grounded language, and we discuss ongoing studies using existing systems.
Can You Distinguish Truthful from Fake Reviews? User Analysis and Assistance Tool for Fake Review Detection
Authors: Jeonghwan Kim, Junmo Kang, Suwon Shin, Sung-Hyon Myaeng
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4

Customer reviews are useful in providing an indirect, secondhand experience of a product. People often use reviews written by other customers as a guideline prior to purchasing a product. Such behavior signifies the authenticity of reviews in e-commerce platforms. However, fake reviews are increasingly becoming a hassle for both consumers and product owners. To address this issue, we propose You Only Need Gold (YONG), an essential information mining tool for detecting fake reviews and augmenting user discretion. Our experimental results show the poor human performance on fake review detection, substantially improved user capability given our tool, and the ultimate need for user reliance on the tool.
Challenges in Designing Games with a Purpose for Abusive Language Annotation
Authors: Federico Bonetti, Sara Tonelli
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4
Tasks: Abusive Language

In this paper we discuss several challenges related to the development of a 3D game, whose goal is to raise awareness on cyberbullying while collecting linguistic annotation on offensive language. The game is meant to be used by teenagers, thus raising a number of issues that need to be tackled during development. For example, the game aesthetics should be appealing for players belonging to this age group, but at the same time all possible solutions should be implemented to meet privacy requirements. Also, the task of linguistic annotation should be possibly hidden, adopting so-called orthogonal game mechanics, without affecting the quality of collected data. While some of these challenges are being tackled in the game development, some others are discussed in this paper but still lack an ultimate solution.
Challenges in Designing Natural Language Interfaces for Complex Visual Models
Authors: Henrik Voigt, Monique Meuschke, Kai Lawonn, Sina Zarrieß
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4

Intuitive interaction with visual models becomes an increasingly important task in the field of Visualization (VIS) and verbal interaction represents a significant aspect of it. Vice versa, modeling verbal interaction in visual environments is a major trend in ongoing research in NLP. To date, research on Language & Vision, however, mostly happens at the intersection of NLP and Computer Vision (CV), and much less at the intersection of NLP and Visualization, which is an important area in Human-Computer Interaction (HCI). This paper presents a brief survey of recent work on interactive tasks and set-ups in NLP and Visualization. We discuss the respective methods, show interesting gaps, and conclude by suggesting neural, visually grounded dialogue modeling as a promising potential for NLIs for visual models.
Situation-Specific Multimodal Feature Adaptation
Authors: Özge Alacam
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4
Tasks: Diversity

In the next decade, we will see a considerable need for NLP models for situated settings where diversity of situations and also different modalities including eye-movements should be taken into account in order to grasp the intention of the user. However, language comprehension in situated settings can not be handled in isolation, where different multimodal cues are inherently present and essential parts of the situations. In this research proposal, we aim to quantify the influence of each modality in interaction with various referential complexities. We propose to encode the referential complexity of the situated settings in the embeddings during pre-training to implicitly guide the model to the most plausible situation-specific deviations. We summarize the challenges of intention extraction and propose a methodological approach to investigate a situation-specific feature adaptation to improve crossmodal mapping and meaning recovery from noisy communication settings.
Machine Translation Believability
Authors: Marianna Martindale, Kevin Duh, Marine Carpuat
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4
Tasks: Machine Translation, Translation

Successful Machine Translation (MT) deployment requires understanding not only the intrinsic qualities of MT output, such as fluency and adequacy, but also user perceptions. Users who do not understand the source language respond to MT output based on their perception of the likelihood that the meaning of the MT output matches the meaning of the source text. We refer to this as believability. Output that is not believable may be off-putting to users, but believable MT output with incorrect meaning may mislead them. In this work, we study the relationship of believability to fluency and adequacy by applying traditional MT direct assessment protocols to annotate all three features on the output of neural MT systems. Quantitative analysis of these annotations shows that believability is closely related to but distinct from fluency, and initial qualitative analysis suggests that semantic features may account for the difference.
An IDR Framework of Opportunities and Barriers between HCI and NLP
Authors: Nanna Inie, Leon Derczynski
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4

This paper presents a framework of opportunities and barriers/risks between the two research fields Natural Language Processing (NLP) and Human-Computer Interaction (HCI). The framework is constructed by following an interdisciplinary research-model (IDR), combining field-specific knowledge with existing work in the two fields. The resulting framework is intended as a departure point for discussion and inspiration for research collaborations.
RE-AIMing Predictive Text
Authors: Matthew Higgs, Claire McCallum, Selina Sutton, Mark Warner
Date: 2021-04-01 00:00:00
Proceeding: EACL (HCINLP) 2021 4

Our increasing reliance on mobile applications means much of our communication is mediated with the support of predictive text systems. How do these systems impact interpersonal communication and broader society? In what ways are predictive text systems harmful, to whom, and why? In this paper, we focus on predictive text systems on mobile devices and attempt to answer these questions. We introduce the concept of a ‘text entry intervention’ as a way to evaluate predictive text systems through an interventional lens, and consider the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) of predictive text systems. We finish with a discussion of opportunities for NLP.