pwc-archive/papers-with-abstracts huggingface dataset:
DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: counterfactual, Domain Adaptation

Natural language processing (NLP) algorithms have become very successful, but they still struggle when applied to out-of-distribution examples. In this paper we propose a controllable generation approach in order to deal with this domain adaptation (DA) challenge. 
Given an input text example, our DoCoGen algorithm generates a domain-counterfactual textual example (D-CON) – that is similar to the original in all aspects, including the task label, but its domain is changed to a desired one. Importantly, DoCoGen is trained using only unlabeled examples from multiple domains – no NLP task labels or pairs of textual examples and their domain-counterfactuals are required.
We use the D-CONs generated by DoCoGen to augment a sentiment classifier in 20 DA setups, where source-domain labeled data is scarce. Our model outperforms strong baselines and improves the accuracy of a state-of-the-art unsupervised DA algorithm.
BACN: Bi-direction Attention Capsule-based Network for Multimodal Sentiment Analysis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Multimodal Sentiment Analysis, Sentiment Analysis

Capsule-based network has currently identified its effectiveness in analyzing the heterogeneity issue of multimodal sentiment analysis. 
However, existing manners could only exploit the spatial relation between representation and output layer via down-top attention, which fails to effectively explore both inter-modality and intra-modality context. In this paper, during the preprocess period, we first present the multimodal dynamic enhanced module to facilitate the intra-modality context, which significantly boost the learning efficiency in dealing with multimodal heterogeneity issue. Furthermore, the bi-direction attention capsule-based network (BACN) is proposed to capture dynamic inter-modality context via the novel bi-direction dynamic routing mechanism. Specifically, BACN firstly highlights the static and low-level inter-modality context based on top-down attention. Then, the static multimodal context is transmitted to dynamic routing procedure, naturally allowing us to investigate dynamic and high-level inter-modality context. This indeed unleash the expressive power and provides the superior capability to bridge the modality gap among all the modalities. The experiments demonstrate that BACN can achieve state-of-the-art performance.
Autoregressive Language Model for Zero-shot Constrained Keyphrase Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Keyphrase Generation, Language Modeling, Language Modelling

Recently, most of the state-of-the-art keyphrase prediction models are based on a supervised generative model.
It shows significantly better than before. Nevertheless, it still faces domain robustness and building datasets on high-resource. 
To overcome these limitations, unsupervised methods have also been critical and studied. We analyzed it also have a defect in a necessary process, which extracts candidates beforehand selecting keyphrase. As not including various forms of phrases, we note that the unsupervised method can't ensure oracle keyphrase.
In this paper, we present zero-shot constrained keyphrase generation by leveraging a large-scale language model. To generate diverse keyphrases, we explore controlling a phrase during the generation. Finally, we evaluate benchmark datasets of the scholar domain. It results in better performances than unsupervised methods on several datasets without going through the candidate extraction stage. For domain robustness, we evaluate out-of-domain DUC compare with NUS. Since our method doesn't fine-tune to a corpus of a specific domain, it's better than supervised methods based on Sequence-to-Sequence.
Aggregating Pairwise Semantic Differences for Few-Shot Claim Veracity Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Fact Checking, Language Modeling, Language Modelling, Veracity Classification

As part of an automated fact-checking pipeline, the claim veracity classification task consists in determining if a claim is supported by an associated piece of evidence. The complexity of gathering labelled claim-evidence pairs leads to a scarcity of datasets, particularly when dealing with new domains. In this paper, we introduce SEED, a novel vector-based method to few-shot claim veracity classification that aggregates pairwise semantic differences for claim-evidence pairs. We build on the hypothesis that we can find class representative vectors that capture average semantic differences for claim-evidence pairs in a class, which can then be used for classification of new instances. We compare the performance of our method with competitive baselines including fine-tuned BERT/RoBERTa models, as well as the state-of-the-art few-shot veracity classification method that leverages language model perplexity. Experiments conducted on the FEVER and SCIFACT datasets show consistent improvements over competitive baselines in few-shot settings. Our code will be made publicly available upon publication.
Self-Competitive Learning for Solving Math Word Problem
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Math

Math word problem (MWP) aims to automatically solve mathematical questions given in texts. Most previous MWP models tend to fit the sole ground-truth solution provided by the dataset, without considering the diverse but equivalent solution expressions. To mitigate this issue, we propose a self-competitive learning framework (called SCL), which attempts to get different predictions and improve the generalization ability of the model by cooperatively learning a source network and a pruned competitor network. The competitor network is created by pruning a source network, which perturbs the source network’s structure and is conducive to generate diverse solutions.  The source network and the competitor network learn collaboratively and teach each other throughout the training process. Extensive experiments on two large-scale benchmarks demonstrate that our model substantially outperforms the strong baseline methods. In particular, our method improves the best performance (accuracy) by 8.4% (78.4% $\rightarrow$ 86.8%) for Math23k and 6.2% (70.5% $\rightarrow$ 76.7%) for Ape210K.
A self-reliant finite automata for reduplication detection
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Reduplication is a common phenomenon in almost all human languages. It implies the repetition of the smallest linguistic unit partially (e.g. flip flop) or completely (e.g. bye bye). Symbolically it can be written as WiWj , where Wi and Wj are similar or almost similar and Wi is any linguistic unit or token or sequence of symbols. Identifying such reduplication in a text is an important aspect of various language processing tasks. Several researchers have tried to address the issue and solve using the heuristic approach, 1-way / 2-way finite-state transducers. This paper presented a sophisticated generic (NFA-based) automaton that is self-reliant generates dynamically and identifies all the reduplicated words from the text.
A Graph Enhanced BERT Model for Event Prediction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: model, Prediction

Predicting the subsequent event for an existing event context is an important but challenging task, as it requires understanding the underlying relationship between events. Previous methods propose to retrieve relational features from event graph to enhance the modeling of event correlation. However, the sparsity of event graph may restrict the acquisition of relevant graph information, and hence influence the model performance. To address this issue, we consider automatically building of event graph using a BERT model. To this end, we incorporate an additional structured variable into BERT to learn to predict the event connections in the training process.
Hence, in the test process, the connection relationship for unseen events can be predicted by the structured variable.
Results on two event prediction tasks: script event prediction and story ending prediction, show that our approach can outperform state-of-the-art baseline methods.
Improving Paraphrase Generation models with machine translation generated pre-training
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Paraphrase Generation, Translation

Paraphrase generation is a fundamental and longstanding problem in the Natural Language Processing field. With the huge success of pre-trained transformers, the pre-train–fine-tune approach has become a standard choice. At the same time, popular task-agnostic pre-trainings usually require terabyte datasets and hundreds of GPUs, while available pre-trained models are limited to architecture and size. We propose a simple and efficient pre-training approach specifically for paraphrase generation, which noticeably boosts model quality and doesn't require significant computing power. We also investigate how this procedure influences the scores across different architectures and show that it helps them all.
Sharper Reasons: Argument Mining Leveraged with Confluent Knowledge
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Argument Mining, Transfer Learning

Relevant to all application domains where it is important to get at the reasons underlying decisions and sentiments, argument mining seeks to obtain structured arguments from unstructured text and has been addressed recently by approaches typically involving some feature and/or neural architecture engineering.

By embracing a transfer learning viewpoint, the aim of this paper is to empirically assess the potential of transferring knowledge learned with confluent tasks to argument mining by means of a systematic study with a wide range of sources of related knowledge possibly suitable to leverage argument mining.

This permitted to gain new empirically based insights into the argument mining task while establishing also new state of the art levels of performance for the three main sub-tasks in argument mining, viz. identification of argument components, classification of the components, and determination of the relation among them, with a leaner approach that dispenses with heavier feature and model engineering.
Stable Natural Language Understanding via Invariant Causal Constraint
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Natural Language Understanding

Natural Language Understanding (NLU) task requires the model to understand the underlying semantics of input text. However, recent analyses demonstrate that NLU models tend to utilize dataset biases to achieve high dataset-specific performances, which always leads to performance degradation on out-of-distribution (OOD) samples. To
increase the performance stability, previous debiasing methods \emph{empirically} capture bias features from data to prevent model from corresponding biases. However, we argue that, the semantic information can form a \emph{causal} relationship with the target labels of the NLU task, while the biases information is only \emph{correlative} to the target labels. Such difference between the semantic information and dataset biases still remains not fully addressed, which limits the effectiveness of debiasing. To address this issue, we analyze the debiasing process under a \emph{causal perspective}, and present a causal invariance based stable NLU framework (CI-sNLU).
Zero-Shot Dependency Parsing with Worst-Case Aware Automated Curriculum Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cross-Lingual Transfer, Dependency Parsing, Multi-Task Learning

Large multilingual pretrained language models such as mBERT and XLM-RoBERTa have been found to be surprisingly effective for cross-lingual transfer of syntactic parsing models Wu and Dredze (2019), but only between related languages. However, source and training languages are rarely related, when parsing truly low-resource languages. To close this gap, we adopt a method from multi-task learning, which relies on automated curriculum learning, to dynamically optimize for parsing performance on {\em outlier} languages. We show that this approach is significantly better than uniform and size-proportional sampling in the zero-shot setting.
Cost-Effective Training in Low-Resource Neural Machine Translation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Active Learning, Domain Adaptation, Low Resource Neural Machine Translation, Low-Resource Neural Machine Translation, Machine Translation, NMT, Translation

While Active Learning (AL) techniques are explored in Neural Machine Translation (NMT), only a few works focus on tackling low annotation budgets where a limited number of sentences can get translated. Such situations are especially challenging and can occur for endangered languages with few human annotators or having cost constraints to label large amounts of data. Although AL is shown to be helpful with large budgets, it is not enough to build high-quality translation systems in these low-resource conditions. In this work, we propose a cost-effective training procedure to increase the performance of NMT models utilizing a small number of annotated sentences and dictionary entries. Our method leverages monolingual data with self-supervised objectives and a small-scale, inexpensive dictionary for additional supervision to initialize the NMT model before applying AL. We show that improving the model using a combination of these knowledge sources is essential to exploit AL strategies and increase gains in low-resource conditions. We also present a novel AL strategy inspired by domain adaptation for NMT and show that it is effective for low budgets. We propose a new hybrid data-driven approach, which samples sentences that are diverse from the labelled data as well as similar to unlabelled data. Finally, we show that initializing the NMT model and further using our AL strategy can achieve gains of up to $13$ BLEU compared to conventional AL methods
Efficient Speech Translation with Pre-trained models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: GPU, Translation

When building state-of-the-art speech translation models, the need for large computational resources is a significant obstacle due to the large training data size and complex models. The availability of pre-trained models is a promising opportunity to build strong speech translation systems efficiently. In a first step, we investigate efficient strategies to build cascaded and end-to-end speech translation systems based on pre-trained models. Using this strategy, we can train and apply the models on a single GPU. While the end-to-end models show superior translation performance to cascaded ones, the application of this technology has a limitation on the need for additional end-to-end training data. In a second step, we proposed an additional similarity loss to encourage the model to generate similar hidden representations for speech and transcript. Using this technique, we can increase the data efficiency and improve the translation quality by 6 BLEU points in scenarios with limited end-to-end training data.
Textual Backdoor Attacks Can Be More Harmful via Two Simple Tricks
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Vocal Bursts Valence Prediction

Backdoor attacks are a kind of emergent security threat in deep learning. After injected into a backdoor, a deep neural model will behave normally on standard inputs but give adversary-specified predictions once the input contains specific backdoor triggers. Current textual backdoor attacks have poor attack performance in some tough situations. In this paper, we find two simple tricks that can make existing textual backdoor attacks much more harmful. The first trick is to add an extra training task to distinguish poisoned and clean data during the training of the victim model, and the second one is to use all the clean training data rather than remove the original clean data corresponding to the poisoned data. These two tricks are universally applicable to different attack models.
We conduct experiments in three tough situations including clean data fine-tuning, low-poisoning-rate, and label-consistent attacks. Experimental results show that the two tricks can significantly improve attack performance. This paper exhibits the great potential harmfulness of backdoor attacks. All the code and data will be made public to facilitate further research.
Combining static and contextualised multilingual embeddings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Retrieval, XLM-R

Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. Contextual language models are more powerful. We combine the strengths of static and contextual models to improve multilingual representations. We extract static embeddings for 40 languages from XLM-R, validate those embeddings with cross-lingual word retrieval, and then align them using VecMap. This results in high-quality, highly multilingual static embeddings. Then we apply a novel continued pre-training approach to XLM-R, leveraging the high quality alignment of our static embeddings to better align the representation space of XLM-R. We show positive results for multiple complex semantic tasks. We will release the static embeddings and the continued pre-training code.
Extracting and Inferring Personal Attributes from Dialogue
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Attribute, Language Modeling, Language Modelling

Personal attributes represent structured information about a person, such as their hobbies, pets, family, likes and dislikes. We introduce the tasks of extracting and inferring personal attributes from human-human dialogue, and analyze the linguistic demands of these tasks. To meet these challenges, we introduce a simple and extensible model that combines an autoregressive language model utilizing constrained attribute generation with a discriminative reranker. Our model outperforms strong baselines on extracting personal attributes as well as inferring personal attributes that are not contained verbatim in utterances and instead requires commonsense reasoning and lexical inferences, which occur frequently in everyday conversation. Finally, we demonstrate the benefit of incorporating personal attributes in social chit-chat and task-oriented dialogue settings.
Discontinuous Constituency and BERT: A Case Study of Dutch
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

In this paper, we set out to quantify the syntactic capacity of BERT in the evaluation regime of non-context free patterns, as occurring in Dutch. We devise a test suite based on a mildly context-sensitive formalism, from which we derive grammars that capture the linguistic phenomena of control verb nesting and verb raising. The grammars, paired with a small lexicon, provide us with a large collection of naturalistic utterances, annotated with verb-subject pairings, that serve as the evaluation test bed for an attention-based span selection probe. Our results, backed by extensive analysis, suggest that the models investigated fail in the implicit acquisition of the dependencies examined.
Speech-to-SQL Parsing: Error Correction with Multi-modal Representations
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition, SQL Parsing, Text to SQL, Text-To-SQL

We study the task of spoken natural language to SQL parsing (speech-to-SQL), where the goal is to map a spoken utterance to the corresponding SQL. Existing work on SQL parsing has focused on text as input (text-to-SQL). To develop a speech-to-SQL parser, we harness progress in text-to-SQL parsing, and automatic speech recognition (ASR). However, ASR is still error-prone, we therefore propose an error correction method that fixes ASR errors in the context of a DB schema. We present a novel multi-modal representation of text, audio, and DB schema with audio attention and a phoneme prediction auxiliary task. Our experiments show that our method yields better performance, is much faster to train, has greater transparency, and is parser-agnostic compared to baselines that seek to adapt to ASR errors.
N-grammer: Augmenting Transformers with latent n-grams
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling

Transformer models have recently emerged as one of the foundational models in natural language processing, and as a byproduct, there has been significant recent interest and investment in scaling these models. However, the training and inference costs of these large Transformer language models are prohibitive, thus necessitating more research in identifying more efficient variants. In this work, we propose a simple yet effective modification to the Transformer architecture inspired by the literature in statistical language modeling, by augmenting the model with n-grams constructed from a discrete latent representation of the text sequence. We evaluate our model, the N-grammer on language modeling on the C4 data-set, and find that it outperforms several strong baselines such as the Transformer and the Primer. We will open-source our model for reproducibility purposes upon acceptance.
Breaking Down Questions for Outside-Knowledge Visual Question Answering
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Graph Neural Network, Question Answering, Visual Question Answering, Visual Question Answering (VQA)

There is a recent trend towards Knowledge-Based VQA (KB-VQA) where different aspects of the question require different sources of knowledge including the image's visual content and external knowledge such as commonsense concepts and factual information. To address this issue, we propose a novel approach that passes knowledge from various sources between different pieces of semantic content in the question. Questions are first segmented into several chunks, and each segment is used to generate queries to retrieve knowledge from ConceptNet and Wikipedia. Then, a graph neural network, taking advantage of the question's syntactic structure, integrates the knowledge for different segments to jointly predict the answer. Our experiments on the OK-VQA dataset show that our approach achieves new state-of-the-art results.