pwc-archive/papers-with-abstracts huggingface dataset:
OkwuGbé: End-to-End Speech Recognition for Fon and Igbo
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Machine Translation, speech-recognition, Speech Recognition

Language is inherent and compulsory for human communication. Whether expressed in a written or spoken way, it ensures understanding between people of the same and different regions. With the growing awareness and effort to include more low-resourced languages in NLP research, African languages have recently been a major subject of research in machine translation, and other text-based areas of NLP. However, there is still very little comparable research in speech recognition for African languages. Interestingly, some of the unique properties of African languages affecting NLP, like their diacritical and tonal complexities, have a major root in their speech, suggesting that careful speech interpretation could provide more intuition on how to deal with the linguistic complexities of African languages for text-based NLP. OkwuGbé is a step towards building speech recognition systems for African low-resourced languages. Using Fon and Igbo as our case study, we conduct a comprehensive linguistic analysis of each language and describe the creation of end-to-end, deep neural network-based speech recognition models for both languages. We present a state-of-the-art ASR model for Fon, as well as benchmark ASR model results for Igbo. Our linguistic analyses (for Fon and Igbo) provide valuable insights and guidance into the creation of speech recognition models for other African low-resourced languages, as well as guide future NLP research for Fon and Igbo. The Fon and Igbo models source code will be publicly available.
Few-Shot Learning with Siamese Networks and Label Tuning
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Few-Shot Learning, Few-Shot Text Classification, Natural Language Inference, text-classification, Text Classification

We study the problem of building text classifiers with little or no training data, commonly known as zero and few-shot text classification.
In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.
In this work, we show that with proper pre-training, Siamese networks that embed texts and labels are
a competitive alternative.
These models allow for a large reduction in inference cost: constant in the number of labels rather than linear.
Furthermore, we introduce label tuning, a simple and computationally efficient approach that allows to adapt the models in a few-shot setup by only changing the label embeddings.
While giving lower performance than model fine-tuning, this approach has the architectural advantage that a single encoder can be shared by many different tasks.
Should We Trust This Summary? Bayesian Abstractive Summarization to The Rescue
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Abstractive Text Summarization, Active Learning, Bayesian Inference

We explore the notion of uncertainty in the context of modern abstractive summarization models, using the tools of Bayesian Deep Learning. Our approach approximates Bayesian inference by first extending state-of-the-art summarization models with Monte Carlo dropout and then using them to perform multiple stochastic forward passes. Based on Bayesian inference we are able to effectively quantify uncertainty at prediction time. Having a reliable uncertainty measure, we can improve the experience of the end user by filtering out generated summaries of high uncertainty. Furthermore, uncertainty estimation could be used as a criterion for selecting samples for annotation, and can be paired nicely with active learning and human-in-the-loop approaches. Finally, Bayesian inference enables us to find a Bayesian summary which performs better than a deterministic one and is more robust to uncertainty. In practice, we show that our Variational Bayesian equivalents of BART and PEGASUS can outperform their deterministic counterparts on multiple benchmark datasets.
Multi-Label Text Classification by Graph Neural Network with Mixing Operations
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Graph Neural Network, Multi Label Text Classification, Multi-Label Text Classification, Representation Learning, text-classification, Text Classification

Multi-label text classification is one of the fundamental tasks in natural language processing. Recently, the graph convolution network (GCN) is leveraged to boost the performance of such a task. However, the best way for label correlation modeling and feature learning with label system awareness is still unclear. This paper proposes Mix-GCN, a graph network with two mixing operations, to improve the conventional GCN framework for multi-label text classification in the following two steps. Firstly, we model the label correlations by mixing the graph built from statistical co-occurrence information and the graph constructed from prior knowledge. Secondly, we propose a mixing operation to continuously inject GCN embedding into LSTM representation learning for better label-aware representation. Experimental results on four benchmarks demonstrate that Mix-GCN significantly outperforms the state-of-the-art models and performs better in long-tail label cases.
Attending to Visual Differences for Situated Language Generation in Changing Scenes
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Text Generation

We investigate the problem of generating utterances from pairs of images showing a before and an after state of a change in a visual scene. We present a transformer model with difference attention heads that learns to attend to visual changes in consecutive images via a difference key. We test our approach in instruction generation, change captioning, and difference spotting and compare these tasks in terms of their linguistic phenomena and reasoning abilities. Our model outperforms the state-of-the-art for instruction generation on the BLOCKS and difference spotting on the Spot-the-diff dataset and generates accurate referential and compositional spatial expressions. Finally, we identify linguistic phenomena that pose challenges for generation in changing scenes.
PPT: Pre-trained Prompt Tuning for Few-shot Learning
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Attribute, Few-Shot Learning

Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and only tunes soft prompts, provides an efficient and effective solution for adapting large-scale PLMs to downstream tasks. However, prompt tuning is yet to be fully explored. In our pilot experiments, we find that prompt tuning performs comparably with conventional full-model tuning when downstream data are sufficient, whereas it is much worse under few-shot learning settings, which may hinder the application of prompt tuning. We attribute this low performance to the manner of initializing soft prompts. Therefore, in this work, we propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization. We name this Pre-trained Prompt Tuning framework "PPT". To ensure the generalization of PPT, we formulate similar classification tasks into a unified task form and pre-train soft prompts for this unified task. Extensive experiments show that tuning pre-trained prompts for downstream tasks can reach or even outperform full-model fine-tuning under both full-data and few-shot settings. Our approach is effective and efficient for using large-scale PLMs in practice.
Qualitative and Quantitative Analysis of Diversity in Cross-document Coreference Resolution Datasets
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: coreference-resolution, Coreference Resolution, Cross Document Coreference Resolution, Diversity

Established cross-document coreference resolution (CDCR) datasets contain manually annotated event-centric mentions of events and entities that form coreference chains with identity relations. In this paper, we qualitatively and quantitatively compare the annotation schemes of ECB+, a CDCR dataset with identity coreference relations, and NewsWCL50, a CDCR dataset with identity, bridging, and near-identity coreference relations. The analysis shows that coreference chains of NewsWCL50 are more lexically diverse ECB+ but annotating of NewsWCL50 leads to the lower inter-coder reliability. We propose a phrasing diversity metric (PD) that encounters for the diversity of full phrases unlike the previously proposed metrics. We discuss the different tasks that both CDCR datasets create, i.e., lexical disambiguation and lexical diversity challenges for CDCR models, and propose a direction for further CDCR evaluation.
Generating Scientific Definitions with Controllable Complexity
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Reranking

Unfamiliar terminology and complex language can present barriers to understanding  science. Natural language processing stands to help address these issues by automatically defining unfamiliar terms. We introduce a new task and dataset for defining scientific terms and controlling the complexity of generated definitions as a way of adapting to a specific reader's background knowledge. We test four definition generation methods for this new task, finding that a sequence-to-sequence approach is most successful. We then explore the version of the task in which definitions are generated at a target complexity level.  We introduce a novel reranking approach and find in human evaluations that it offers superior fluency while also controlling complexity, compared to several controllable generation baselines.
Rare but Severe Errors Induced by Minimal Deletions in English-Chinese Neural Machine Translation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Hallucination, Machine Translation, Translation

We examine the inducement of rare but severe errors in English-Chinese and Chinese-English Transformer-based neural machine translation by minimal deletion in the source text. We also examine the effect of training data size on the number and types of pathological cases induced by these perturbations, finding significant variation. We find that one type of hallucination can be remedied through data preprocessing and that deleting words hurts more than deleting characters in a character-based model, even though deleting characters introduces nonsense words.
Collaborative Multi-Task Representation for Natural Language Understanding
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Multi-Task Learning, Natural Language Understanding

Multi-task learning has shown large benefits in natural language understanding tasks. However, the seesaw phenomenon is still prominent in existing multi-task learning solutions. In other words, it is difficult to balance the importance of different tasks to learn a unified representation of natural language. In this paper, we propose a Collaborative Multi-Task Representation (CMTR) framework to tackle this problem. Specifically, we capture instance-level task relations by a task interaction layer and calculate the final representation for each instance as a mixture of task-oriented representations. We also introduce auxiliary losses to ease optimization and improve the generalization ability of multi-task representations. Empirically, CMTR outperforms state-of-the-art multi-task learning frameworks for natural language understanding tasks. We also reveal the effectiveness of CMTR through extensive analyses.
Explainable Assessment of Healthcare Articles with QA
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Articles, Explanation Generation, Question Answering, text-classification, Text Classification

The healthcare domain suffers from the spread of poor quality articles on the Internet. While manual efforts exist, they are not sufficient to assess the amount of articles in circulation. The task can be automated as text classification, however explanations for the labels are necessary for the users. While current explainable systems tackle explanation generation as summarization, we propose a new approach based on Question-Answering that allows us to generate explanations for multiple criteria. We show that QA-based models are competitive with current state-of-the-art systems and complement summarization-based models for explainable quality assessment.
Cross-lingual Inference with A Chinese Entailment Graph
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Entity Typing, Question Answering, Relation, Relation Extraction

Predicate entailment detection is a crucial task for question-answering from text, where previous work has explored unsupervised learning of entailment graphs from typed open relation triples. In this paper, we present the first pipeline for building Chinese entailment graphs. In this pipeline, we present a novel high-recall open relation extraction (ORE) method and the first Chinese fine-grained entity typing dataset following the FIGER type ontology. Through experiments on the popular Levy-Holt dataset, translated into Chinese, we show that our Chinese entailment graph outperforms a range of strong baselines by large margins. Moreover, an ensemble of Chinese and English entailment graphs sets a new unsupervised SOTA on the original Levy-Holt dataset, surpassing previous SOTA by more than 4 AUC points.
SpelLM: Augmenting Chinese Spell Check Using Input Salience
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Sentence

The task of Chinese Spell Check (CSC) has a goal of detecting and correcting the misspelled Chinese characters in a sentence.  Due to the complex nature of Chinese characters, the CSC task is very challenging and has attracted great attention in the literature. Recent works have shown that the masked language models (e.g. BERT), if combined with the use of confusion sets or filtering mechanisms, can be used for handling the error sparsity and domain shift issues inherent in the CSC task.However, the confusion sets require human intervention and have to be regularly updated to cater for new errors. Also, the filtering methods are sensitive to the similarity measurement between characters. Moreover, the manually-determined filters rely on expert experience and are not adaptable to changed errors. To overcome the shortcomings, we develop a two-stage model, SpelLM, which can exploit BERT for the CSC task without relying on the confusion sets or filtering mechanisms. Specifically, in the first stage, we tune BERT as a binary classifier to predict whether a sentence contains spell errors. Then, we can compute the ``salience'' of each character in the input sentence, which measures how much a character contributes to the prediction. In the second stage, we tune another BERT using error pairs and incorporate at each self-attention layer the salience information of each input sentence. We train a linear layer to distribute the salience information into the query vectors, which serves as prior knowledge pertaining to spell errors for the attention computation. Finally, for each character we use the encoding output by the second BERT to predict the correction. As shown in our empirical study, our model-only method outperforms existing BERT solutions with the confusion sets and the filtering scheme by a notable margin.
D2U: Distance-to-Uniform Learning for Out-of-Scope Detection
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10

Supervised models trained for single-label classification tasks with cross-entropy loss are implicitly enforced to produce probability distributions that follow a discrete delta distribution in training. Model predictions in test time are expected to be similar to delta distributions given that the classifier determines the class of an input correctly. However, the shape of the predicted probability distribution becomes similar to the uniform distribution when the model cannot infer properly. We exploit this observation for detecting out-of-scope (OOS) utterances in conversational systems. Specifically, we propose a zero-shot post-processing step, called Distance-to-Uniform (D2U), exploiting not only the classification confidence score, but the shape of the entire output distribution. We also introduce a learning procedure that uses D2U for loss calculation in the supervised setup. We conduct experiments using six publicly available datasets. Experimental results show that the performance of out-of-scope detection is improved with our post-processing when there is no OOS training data, as well as with D2U learning procedure when OOS training data is available.
IMPLI: Investigating NLI Models' Performance on Figurative Language
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Natural Language Inference, Sentence

Natural language inference (NLI) has been widely used as a task to train and evaluate models for language understanding. However, the ability of NLI models to perform inferences that require understanding of figurative languages such as idioms and metaphors remains understudied. We introduce the IMPLI (Idiomatic and Metaphoric Paired Language Inference) dataset consisting of over 25K semi-automatically generated and 1.5K hand-written English sentence pairs based on idiomatic and metaphoric phrases. We use \dataset to evaluate NLI models based on RoBERTa fine-tuned on the MNLI dataset, and show that while they can reliably detect entailment relationship between figurative phrases with their literal definition, they perform poorly on examples where the phrases are designed to not entail the paired definition. This dataset suggests the limits of current NLI models with regard to understanding figurative language and provides a benchmark for future improvements in this direction.
GCPG: A General Framework for Controllable Paraphrase Generation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Decoder, Paraphrase Generation, Sentence

Controllable paraphrase generation (CPG) incorporates various external conditions to obtain desirable paraphrases. However, existing works only highlight a special condition under two indispensable aspects of CPG (i.e., lexically and syntactically CPG) individually, lacking a unified circumstance to explore and analyze their effectiveness. In this paper, we propose a general controllable paraphrase generation framework (GCPG), which represents both lexical and syntactical conditions as text sequences and uniformly processes them in an encoder-decoder paradigm. Under GCPG, we reconstruct commonly adopted lexical condition (i.e., Keywords) and syntactical conditions (i.e., Part-Of-Speech sequence, Constituent Tree, Masked Template and Sentential Exemplar) and study the combination of the two types. In particular, for Sentential Exemplar condition, we propose a novel exemplar construction method --- Syntax-Similarity based Exemplar (SSE). SSE retrieves a syntactically similar but lexically different sentence as the exemplar for each target sentence, avoiding exemplar-side words copying problem. Extensive experiments demonstrate that GCPG with SSE achieves state-of-the-art performance on two popular benchmarks. In addition, the combination of lexical and syntactical conditions shows the significant controllable ability of paraphrase generation, and these empirical results could provide novel insight to user-oriented paraphrasing.
Alleviating the Inequality of Attention Heads for Neural Machine Translation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Machine Translation, Translation

Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.
xGQA: Cross-Lingual Visual Question Answering
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Cross-Lingual Transfer, Language Modeling, Language Modelling, Question Answering, Visual Question Answering, Visual Question Answering (VQA), Zero-Shot Cross-Lingual Transfer

Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based models to become multilingual, and---vice versa---multilingual models to become multimodal. Our proposed methods outperform current state-of-the-art multilingual multimodal models (e.g., M3P) in zero-shot cross-lingual settings, but the accuracy remains low across the board; a performance drop of around 38 accuracy points in target languages showcases the difficulty of zero-shot cross-lingual transfer for this task. Our results suggest that simple cross-lingual transfer of multimodal models yields latent multilingual multimodal misalignment, calling for more sophisticated methods for vision and multilingual language modeling. The xGQA dataset is available online at: URL
ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Abstractive Text Summarization, Machine Translation, Translation

Residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODE). This paper explores a deeper relationship between Transformer and numerical ODE methods. We first show that a residual block of layers in Transformer can be described as a higher-order solution to ODE. Inspired by this, we design a new architecture, {\it ODE Transformer}, which is analogous to the Runge-Kutta method that is well motivated in ODE. As a natural extension to Transformer, ODE Transformer is easy to implement and efficient to use. Experimental results on the large-scale machine translation, abstractive summarization, and grammar error correction tasks demonstrate the high genericity of ODE Transformer. It can gain large improvements in model performance over strong baselines (e.g., 30.77 and 44.11 BLEU scores on the WMT'14 English-German and English-French benchmarks) at a slight cost in inference efficiency.