pwc-archive/papers-with-abstracts huggingface dataset:
Long-Tail Classification for Distinctive Image Captioning: A Simple yet Effective Remedy for Side Effects of Reinforcement Learning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Image Captioning, Reinforcement Learning (RL)

Distinctiveness is a desirable feature of image captions. Captions should cover the characteristic details of input images. However, recent high-performing captioning models that are trained with reinforcement learning (RL) tend to generate overly generic captions despite their high performance in various other criteria. Interestingly, it has also been reported that their outputs are composed of a limited number of common words and rarely contain tail-class words, i.e., low-frequency words in the training corpus. Vocabulary size is closely related to distinctiveness as it is difficult for a model to describe details beyond its vocabulary. Based on this insight, we hypothesize that the limited vocabulary of RL models is the major factor limiting their distinctiveness. We recast distinctive image captioning as a simpler task of long-tail classification to increase the vocabulary and then propose lightweight fine-tuning methods to encourage tail-class word generation. The experimental results demonstrate that our methods significantly enhance the distinctiveness of existing RL models as well as their vocabulary size, without sacrificing quality. Our methods also outperform previous distinctiveness-aware methods with a small computational cost of minor modifications to pre-trained RL models.
Active Gradual Machine Learning for Entity Resolution
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Active Learning, BIG-bench Machine Learning, Entity Resolution

Recent work has shown that the task of entity resolution (ER) can be effectively performed by gradual machine learning (GML). GML begins with some easy instances, which can be automatically labeled by the machine with high accuracy, and then gradually labels more challenging instances by iterative knowledge conveyance in a factor graph. Without involving manual labeling effort, the current GML solution for ER is unsupervised. However, its performance is limited by inaccurate and insufficient knowledge conveyance. Therefore, there is a need to investigate how to improve knowledge conveyance by manual labeling effort.In this paper, we propose an active learning (AL) approach based on GML for ER. It iteratively generates new knowledge in the form of one-sided rules by manual label verification and instills them into a factor graph for improved knowledge conveyance. We first present a technique of knowledge discovery based on genetic mutations, which can generate effective knowledge rules with very small manual verification cost. Then, we demonstrate how to leverage the generated rules for improved knowledge conveyance by measuring their influence over label status by the metric of skyline distance. We have evaluated the performance of the proposed approach by a comparative study on real benchmark data. Our extensive experiments have shown that it can significantly improve the performance of unsupervised GML with very small manual cost; furthermore, it outperforms the state-of-the-art AL solutions for deep learning by considerable margins in terms of learning efficiency.
Roof-BERT: Divide Understanding Labour and Join in Work
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Knowledge Graphs, Reading Comprehension

Recent work on enhancing BERT-based language representation models with knowledge graphs (KGs) and knowledge bases (KBs) has promising results on multiple NLP tasks. State-of-the-art approaches typically integrate the original input sentences with triples in KGs, and feed the combined representation into a BERT model. However, as the sequence length of a BERT model is limited, the framework can not contain too much knowledge besides the original input sentences and is thus forced to discard some knowledge. The problem is especially severe for those downstream tasks that input is a long paragraph or even a document, such as QA or reading comprehension tasks. To address the problem, we propose Roof-BERT, a model with two underlying BERTs and a fusion layer on them. One of the underlying BERTs encodes the knowledge resources and the other one encodes the original input sentences, and the fusion layer like a roof integrates both BERTs’ encodings. Experiment results on QA task and GLUE benchmark reveal the effectiveness of the proposed model.
Pinyin-BART: An End-to-End Chinese Input Method
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Decoder

A Chinese Input Method Engine helps user convert a keystroke sequence into the desired Chinese character sequence. It is usually a cascaded process in which the original input sequence is firstly corrected to remove typos, then segmented into the pinyin token sequence, and finally converted into a Chinese character sequence. Errors are prone to accumulate and propagate in that pipeline. This paper summarizes that process as a Key-to-Character (K2C) conversion task and solve it in a unified end-to-end way. Pinyin-bart is proposed which can effectively solve the error propagation problem and improve the IME engine performance significantly in experiments. Moreover, we model the user real input behaviors and design a method to generate the training corpus with typos for the K2C task. It further improves the robustness of Pinyin-bart. Finally, we design a non-autoregressive (NAR) decoder for Pinyin-bart and obtain 9x+ acceleration with limited performance degradation, which makes the deployment possible on the commercial input software.
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Contrastive Learning, Natural Language Understanding

Masked language models (MLMs) such as BERT have revolutionized the field of Natural Language Understanding in the past few years. However, existing pre-trained MLMs often output an anisotropic distribution of token representations that occupies a narrow subset of the entire representation space. Such token representations are not ideal, especially for tasks that demand discriminative semantic meanings of distinct tokens. In this work, we propose TaCL (Token-aware Contrastive Learning), a novel continual pre-training approach that encourages BERT to learn an isotropic and discriminative distribution of token representations. TaCL is fully unsupervised and requires no additional data. We extensively test our approach on a wide range of English and Chinese benchmarks. The results show that TaCL brings consistent and notable improvements over the original BERT model. Furthermore, we conduct detailed analysis to reveal the merits and inner-workings of our approach.
GraphCache: Message Passing as Caching for Sentence-Level Relation Extraction
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Graph Neural Network, Relation, Relation Extraction, Sentence

Entity types and textual context are essential properties for sentence-level relation extraction (RE). Existing work only encodes these properties within individual instances, which limits the performance of RE given the insufficient features in a single sentence. In contrast, we model these properties from the whole dataset and use the dataset-level information to enrich the semantics of every instance. We propose the GraphCache (Graph Neural Network as Caching) module, that propagates the features across sentences to learn better representations for RE. GraphCache aggregates the features from sentences in the whole dataset to learn global representations of properties, and use them to augment the local features within individual sentences. The global property features act as dataset-level prior knowledge for RE, and a complement to the sentence-level features. Inspired by the classical caching technique in computer systems, we develop GraphCache to update the property representations in an online manner. Overall, GraphCache yields significant effectiveness gains on RE and enables efficient message passing across all sentences in the dataset.
How Conservative are Language Models? Adapting to the Introduction of Gender-Neutral Pronouns
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Gender-neutral pronouns have recently been introduced in many languages to a) include non-binary people and b) as a generic singular. Recent results from psycho-linguistics suggest that gender-neutral pronouns (in Swedish) are not associated with human processing difficulties. This, we show, is in sharp contrast with automated processing. We show that gender-neutral pronouns in Danish, English, and Swedish are associated with higher perplexity, more dispersed attention patterns, and worse downstream performance. We argue that such conservativity in language models may limit widespread adoption of gender-neutral pronouns and must therefore be resolved.
Comparative Analysis of Existing and a Novel Approach to Topic Detection on Conversational Dialogue Corpora
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Clustering

Topic detection in dialogue corpora has become a major challenge for a conversational systems, with efficient conversational topic prediction being a critical part of constructing cohesive and engaging dialogue systems (Sunet al., 2019). This paper proposed unsupervised and semi-supervised techniques for topic detection in conversational dialogue corpora and compared them with existing techniques. However, these existing topic detection techniques are widely applied to textual tweets, blogs, documents, textual data on the web. Therefore, we applied these existing techniques to dialogue corpora to detect the topics and compared them with the proposed approach because textual dialogues typically are irregular and short sentences. The paper proposes a novel approach for topic detection, which combines the clustering of known similar words, TF-IDF scores and 'bag of words' techniques (BOW) with the Parallel Latent Dirichlet Allocation (PLDA) Model to achieve topic detection. The approach also integrates the elbow method for interpretation and validation to select the optimal number of clusters. The paper comprises a comparative analysis of traditional LDA and clustering approaches across both unlabelled (unsupervised) and partially labelled (semi-supervised) switchboard corpus with a proposed novel approach. The evaluation results shows that proposed approach performs best using partially labelled topic dialogue corpora and out performed traditional and unsupervised methods.
Pathway2Text: Dataset and Method for Biomedical Pathway Description Generation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Text Generation

Biomedical pathways have been extensively used to characterize the mechanism of complex diseases. One essential step in biomedical pathway analysis is to curate the description of this pathway based on its graph structure and node features. Neural text generation could be a plausible technique to circumvent the tedious manual curation. In this paper, we propose a new dataset Pathway2Text, which contains 2094 pairs of biomedical pathway and textual descriptions. All pathway graphs are experimentally derived or manually curated. All textual descriptions are written by domain experts. We form this problem as a Graph2Text task and propose a novel graph-based text generation approach $k$NN-Graph2Text, which explicitly exploited descriptions of similar graphs to generate new descriptions. We observed substantial improvement of our method on both Graph2Text and the reverse task of Text2Graph. We further illustrated how our dataset can be used as a novel benchmark for biomedical name entity recognition. Collectively, we envision our method will become an important benchmark for evaluating Graph2Text methods and advance biomedical research for complex diseases.
DUCK: Rumour Detection on Social Media by Modelling User and Comment Propagation Networks
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Graph Attention, Misinformation, Rumour Detection

Social media rumours, a form of misinformation, can mislead the public and cause significant economic and social disruption. Motivated by the observation that the user network --- which captures $\textit{who}$ engage with a story  --- and the comment network --- which captures $\textit{how}$ they react to it --- provide complementary signals for rumour detection, in this paper, we propose DUCK  (rumour $\underline{d}$etection with $\underline{u}$ser and $\underline{c}$omment networ$\underline{k}$s) for rumour detection on social media. We study how to leverage transformers and graph attention networks to jointly model the contents and structure of social media conversations, as well as the network of users who engaged in these conversations. Over four widely used benchmark rumour datasets in English and Chinese, we show that DUCK produces superior performance for detecting rumours, creating a new state-of-the-art. Source code for DUCK is available at: ANONYMISED.
Bayesian Deep Learning for Interactive Community Question Answering
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Bayesian Optimisation, Community Question Answering, Deep Learning, Question Answering

Human-in-the-loop interactive learning has been shown to be effective for best solution selection tasks. Bayesian Optimisation (BO) reduces the amount of user interaction required but has so far relied on shallow models rather than end-to-end deep learning. This paper leverages recent advances in Bayesian deep learning (BDL) to more accurately identify the best solution from a few rounds of interaction. We apply our approach to community question answering (cQA), finding that our BDL approach significantly outperforms existing methods while remaining robust to noise in the user feedback.
DialogueEIN: Emotional Interaction Network for Emotion Recognition in Conversations
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Emotion Recognition

Emotion Recognition in Conversations (ERC) is a necessary step for developing empathetic human-computer interaction system. The existing methods on ERC primarily focus on capturing the context-level and speaker-level information from utterances. However, these methods ignore the causes of human emotion change, resulting in insufficient in capturing useful information for emotional prediction. In this work, we propose more explanatory Emotional Interaction Network (DialogueEIN) based on two main stages to capture the contexual information over intra- and inter-speaker dependencies directly from utterances, and to explore and analyze the differentiated contributions over the both kinds of information to boost better understanding of current utterance in conversation. Experimental results on two benchmark datasets demonstrate the effectiveness and superiority of our proposed model.
Disaggregating Hops: Can We Guide a Multi-Hop Reasoning Language Model to Incrementally Learn at each Hop?
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Language Modeling, Language Modelling, Multiple-choice, Multiple Choice Question Answering (MCQA), Question Answering, Reading Comprehension, Transfer Learning

Despite the success of state-of-the-art pre-trained language models (PLMs) on a series of multi-hop reasoning tasks, they still suffer from their limited abilities to transfer learning from simple to complex tasks and vice-versa. We argue that one step forward to overcome this limitation is to better understand the behavioral trend of PLMs at each hop over the inference chain. Our critical underlying idea is to mimic human-style reasoning: we envision the multi-hop reasoning process as a sequence of explicit one-hop incremental reasoning steps. Using the SHINRA and ConceptNet resources jointly, we provide automatically generated datasets built upon a set of inference heuristics on relevant phrases and distractors, allowing us to teach the models incremental reasoning skills.  We empirically show the effectiveness of the proposed models on multiple-choice question answering (MCQA) and reading comprehension (RC), with a relative improvement of $68.4\%$ and $16.0\%$ accuracy improvement w.r.t. classic PLMs, respectively.
Towards Interactive Language Modeling
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Language Acquisition, Language Modeling, Language Modelling

Interaction between caregivers and children plays a critical role in human language acquisition and development. Given this observation, it is remarkable that explicit interaction plays little to no role in artificial language modeling---which also targets the acquisition of human language, yet by artificial models. Moreover, an interactive approach to language modeling has the potential to make language models substantially more versatile and to considerably impact downstream applications. Motivated by these considerations, we pioneer the space of interactive language modeling. First we present a road map in which we detail the steps that need to be taken towards interactive language modeling. We then lead by example and take the first steps on this road map, showing the initial feasibility of our approach. As such, this work aims to be the start of a larger research agenda on interactive language modeling.
Exploring the Value of Multi-View Learning for Session-Aware Query Representation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Document Ranking, MULTI-VIEW LEARNING, Representation Learning, Retrieval

Recent years have witnessed a growing interest towards learning distributed query representations that are able to capture search intent semantics. Most existing approaches learn query embeddings using relevance supervision making them suited only to document ranking tasks. Besides, they generally consider either user’s query reformulations or system’s rankings whereas previous findings show that user’s query behavior and knowledge change depending on the system’s results,  intertwine and affect each other  during  the completion of a search task.  In this paper, we explore the value of multi-view learning for generic and unsupervised session-aware query representation learning. First, single-view query embeddings are obtained in separate  spaces from query reformulations and document ranking representations using transformers. Then, we investigate the use of linear (CCA)  and non linear (UMAP) multi-view learning methods, to align those spaces with the aim of revealing  similarity  traits  in the multi-view shared space. Experimental evaluation is carried out  in a query classification and session-based retrieval downstream tasks using respectively the KDD and TREC session datasets. The results show that multi-view learning is an effective and controllable approach for unsupervised learning of generic query representations and can reflect search behavior patterns.
Practical Dataless Text Classification Through Dense Retrieval
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Classification, Natural Language Inference, Retrieval, text-classification, Text Classification

Dataless text classification aims to classify documents using only class descriptions without any training data. Recent  research shows that pre-trained textual entailment models can achieve state-of-the-art dataless classification performance on various tasks. However, such models are not practical in that their prediction speed is slow as they need k forward passes to predict k classes and they are not built for fine-tuning to further improve the initial (often mediocre) performance.This work proposes a simple, effective, and practical dataless classification approach. We use class descriptions as queries to retrieve task-specific or external unlabeled data on which  pseudo-labels are assigned to train a classifier. Experiments on a wide range of classification tasks show that the proposed approach consistently outperforms entailment-based models in terms of classification accuracy, prediction speed, and performance gain when fine-tuned on labeled data.
EVI: Multilingual Spoken Dialogue Tasks and Dataset for Knowledge-Based Enrolment, Verification, and Identification
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Spoken Dialogue Systems

Knowledge-based authentication is crucial for task-oriented spoken dialogue systems that offer personalised and privacy-focused services. Such systems should be able to enrol (E), verify (V), and identify (I) new and recurring users based on their personal information, e.g. postcode, name, and date-of-birth. In this work, we formalise the three authentication tasks and their evaluation protocols, and we present EVI, a challenging spoken multilingual dataset with 5,506 dialogues in English, Polish, and French. Our proposed models set the first competitive benchmarks, explore the challenges of multilingual natural language processing of spoken dialogue, and set directions for future research.
Improving Fairness without Demographic by Lagged Dynamic Grouping
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Fairness

Machine learning models are prone to social biases in datasets and thus could make discriminatory decisions against demographic minority groups. Most existing fairness-promoting methods usually assume access to the annotations of the demographic information. However, such information could be inaccessible due to the high data annotation cost and privacy restrictions. Recently, distributionally robust optimization (DRO) techniques have been applied to promote fairness without demographic labels. DRO-based methods optimize the individuals/groups with the worst prediction performance, with the intuition that these groups roughly correspond to the minority groups being biased against. However, in complex real-world settings with multiple strong bias attributes, the simple grouping schemes in the existing DRO-based methods can fail to identify the ground truth minority groups. In this paper, we propose FreeDRO, a demographic-free group DRO method featuring a more principled grouping scheme, call lagged dynamic grouping. Specifically, FreeDRO dynamically splits the training data based on the ground truth labels and the prediction of the model at an earlier iteration and then optimizes worst group performance. Extensive experiments on five real-world datasets show that our method can effectively alleviate the biases and even achieve comparable results with methods with full demographic annotations. The results also verify that our grouping scheme has a good correspondence with the ground truth demographic grouping.