pwc-archive/papers-with-abstracts huggingface dataset:
Relevance in Dialogue: An empirical comparison of existing metrics, and a novel simple metric
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9

In this work, we evaluate various existing dialogue relevance metrics, find strong dependencies on the dataset, often with poor correlation with human scores of relevance, and propose modifications to reduce data requirements while improving correlation. With these changes, our metric achieves a new state-of-the-art on the HUMOD dataset (Merdivan et al., 2020). We achieve this without fine-tuning, using only 3750 unannotated human dialogues and a single negative example. Despite these limitations, we demonstrate competitive performance on three datasets from different domains. Our code including our metric and data processing is open sourced.
Remixers: A Mixer-Transformer Architecture with Compositional Operators for Natural Language Understanding
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Inductive Bias, Natural Language Understanding

Recent work such as MLP-Mixers (Tolstikhin et al.) have demonstrated the promise of All-MLP architectures. While All-MLP architectures have demonstrated reasonable performance in computer vision and garnered recent interest, we argue that making them effective in NLP applications is still an uphill battle. Hence, there may be no solid reason to drop the self-attention modules altogether. In this paper, we propose a new Mixer-Transformer architecture, showing that Transformers and Mixer models can be quite complementary indeed. Fundamentally, we show that Mixer models are capable of acting as persistent global memory (in a similar vein to standard MLPs) while being imbued with global receptive fields at the same time. Hence, interleaving sample-dependent and input-local self-attention with persistent Mixer modules can be an effective strategy. Additionally, we propose compositional remixing, a new way of baking compositional operators (multiplicative and subtractive composition) within the mixing process to improve the expressiveness of the model. This allows us to effectively model relationships between unmixed and mixed representations - an inductive bias that we postulate is powerful for NLU applications. Via extensive experiments on 14 challenging NLU datasets (e.g., SuperGLUE, entailment and compositional generalization), we show that the proposed architecture consistently outperforms a strong T5 baseline (Raffel et al.). We believe this work paves the way for more effective synergies between the two families of models.
Measuring Factual Consistency of Abstractive Summaries
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Abstractive Text Summarization, Re-Ranking

Recent abstractive summarization systems fail to generate factual consistent -- faithful -- summaries, which heavily limits their practical application. 
Commonly, these models tend to mix concepts from the source or hallucinate new content, completely ignoring the source.
Addressing the faithfulness problem is perhaps the most critical challenge for current abstractive summarization systems.
First automatic faithfulness metrics were proposed, but we argue that existing methods do not yet utilize all "machinery" that this field has to offer and introduce new approaches to assess factual correctness.
We evaluate existing and our proposed methods by correlating them with human judgements and find that BERTScore works well.
Next, we conduct a data analysis, which reveals common problems, ways to further improve the metrics and indicates that combining multiple metrics is promising. 
Finally, we exploit faithfulness metrics in pre- and post-processing steps to decrease factual errors made by state-of-the-art summarization systems.
We find that simple techniques like filtering training data and re-ranking generated summaries can increase the faithfulness by a substantial margin.
Should a Bot be Sarcastic? Understanding User Preferences Towards Sarcasm Generation
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9

Previous sarcasm generation research has focused on how to generate text that people perceive as sarcastic to create more human-like interactions. In this paper, we argue that we should first turn our attention to the question of when sarcasm should be generated, finding that human annotators consider many inputs to be unfit for sarcastic responses. Next, we introduce a theory-driven framework for sarcasm generation which allows us to better control the linguistic devices used during the generation process in order to measure their impact on sarcasm perception, finding that pragmatic insincerity and emotional markers are crucial elements in generating recognizable sarcasm.
AMR-to-text Generation with Graph Structure Reconstruction and Coverage
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: AMR-to-Text Generation, Graph-to-Sequence, Text Generation, Translation

Generating text from semantic representations such as AMR is a challenging task. Previous research formalizes this task as a graph-to-sequence learning problem and uses various graph neural networks to model the graph structure. Recently, methods based on pre-trained models improve the performance significantly due to pre-trained on a large text corpus. However, these pre-trained model-based methods take linearized AMR graphs as input and may lose the information of graph structure.  In addition, these methods don't consider the coverage of the AMR graph. Therefore, some nodes in the graph may be lost or repeated in the generated text. To address these problems, we propose a graph structure and coverage enhanced model for this task. To enhance the information of graph structure, we design two auxiliary objectives, relationship prediction and distance prediction of nodes in AMR graphs. To consider the coverage of AMR graphs, we design a coverage mechanism to solve the problem of information under-translation or over-translation in AMR-to-text generation. 
 Experimental results on three standard datasets show that our proposed method outperforms the existing methods significantly.
Learning Emotion-Aware Contextual Representations for Emotion Cause Analysis
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Emotion Cause Extraction, Emotion-Cause Pair Extraction

Emotion Cause Analysis has been a key topic in natural language processing. Previous works focus on Emotion Cause Extraction (ECE), a clause-level classification task aimed at extracting causes of certain given emotion in text. The task has been expanded to Emotion Cause Pair Extraction (ECPE) that focus on extracting both emotions and corresponding causes in the context. Most existing methods for the ECPE task implement a joint model that performs extracting and matching of emotion and cause clauses simultaneously. However, we argue that different input features are needed for the two subtasks, thus sharing contextual representations may be suboptimal. In this work, we propose a pipelined approach that builds on two independent pre-trained encoders, in which the emotion extraction model only provide input features for the cause extraction model. Based on a series of careful experiments, we validate that our model can create distinct contextual representations according to specific emotional texts, and thus achieve state-of-the-art performance in both ECE and ECPE tasks, with the absolute F1 improvements of 1.5% and 4.72% over best previous works respectively. Besides, we apply a set of simple clause selection rules to extract multiple pairs in the document, strengthening the applicability of our approach in real world scenarios.
CoDA21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Natural Language Understanding, World Knowledge

Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks. We introduce CoDA21 (Context Definition Alignment), a challenging benchmark that measures natural language understanding (NLU) capabilities of PLMs: Given a definition and a context each for k words, but not the words themselves, the task is to align the k definitions with the k contexts. CoDA21 requires a deep understanding of contexts and definitions, including complex inference and world knowledge. We find that there is a large gap between human and PLM performance, suggesting that CoDA21 measures an aspect of NLU that is not sufficiently covered in existing benchmarks.
Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Attribute, Benchmarking, Question Answering

Question answering-based summarization evaluation metrics must automatically determine whether the QA model's prediction is correct or not, a task known as answer verification. In this work, we benchmark the lexical answer verification methods which have been used by current QA-based metrics as well as two more sophisticated text comparison methods, BERTScore and LERC. We find that LERC out-performs the other methods in some settings while remaining statistically indistinguishable from lexical overlap in others. However, our experiments reveal that improved verification performance does not necessarily translate to overall QA-based metric quality: In some scenarios, using a worse verification method -- or using none at all -- has comparable performance to using the best verification method, a result that we attribute to properties of the datasets.
Defending Textual Neural Networks against Black-Box Adversarial Attacks with Stochastic Multi-Expert Patcher
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9

Even though several methods have proposed to defend textual neural network (NN) models against black-box adversarial attacks, they often defend against a specific text perturbation strategy and/or require re-training the models from scratch. This leads to a lack of generalization in practice and redundant computation. In particular, the state-of-the-art transformer models (e.g., BERT, RoBERTa) require great time and computation resources. By borrowing an idea from software engineering, in order to address these limitations, we propose a novel algorithm, SHIELD, which modifies and re-trains only the last layer of a textual NN, and thus it "patches" and "transforms" the NN into a stochastic weighted ensemble of multi-expert prediction heads. Considering that most of current black-box attacks rely on iterative search mechanisms to optimize their adversarial perturbations, SHIELD confuses the attackers by automatically utilizing different weighted ensembles of predictors depending on the input. In other words, SHIELD breaks a fundamental assumption of the attack, which is a victim NN model remains constant during an attack. By conducting comprehensive experiments, we demonstrate that all of CNN, RNN, BERT, and RoBERTa-based textual NNs, once patched by SHIELD, exhibit a relative enhancement of 15%--70% in accuracy on average against 14 different black-box attacks, outperforming 6 defensive baselines across 3 public datasets. All codes are to be released.
Context vs Target Word: Quantifying Biases When Applying Models to Lexical Semantic Datasets
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Entity Linking

State-of-the-art contextualized models such as BERT use tasks such as WiC and WSD to evaluate their word-in-context representations. This inherently assumes that performance in these tasks reflect how well a model represents the coupled word and context semantics. This study investigates this assumption by presenting the first quantitative analysis (using probing baselines) on the context-word interaction being tested in major contextual lexical semantic tasks which have dramatically different emphasis. We found that models often exhibit excessive context and target word biases: they solve tasks like WiC as almost purely context classification and rely on target words alone when tackling medical entity linking. In the latter task, where domain reduces ambiguity, context does not improve models and in fact can degrade performance. Our case study on WiC reveals that human subjects do not share models' strong context biases (humans found semantic judgments much more difficult when the target word is missing) and models are learning spurious correlations from context alone. This study demonstrates that models are usually not being tested for word-in-context representations as such in these tasks and results are therefore open to misinterpretation. We recommend our framework as sanity check for context and target word biases of future task design and application in lexical semantics.
Personalized News Recommendation with Candidate-aware User Modeling
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: News Recommendation

News recommendation aims to match news with personalized user interest. Existing methods for news recommendation usually model user interest from historical clicked news without the consideration of candidate news. However, each user usually has multiple interests, and it is difficult for these methods to accurately match a candidate news with a specific user interest. In this paper, we present a candidate-aware user modeling method for personalized news recommendation, which can incorporate candidate news into user modeling for better matching between candidate news and user interest. More specifically, we propose a candidate-aware self-attention network that uses candidate news as guidance to model candidate-aware global user interest. In addition, we propose a candidate-aware CNN network to incorporate candidate news into local behavior context modeling to learn candidate-aware short-term user interest. Besides, we use a candidate-aware attention network to aggregate previously clicked news weighted by their relevance with candidate news to build candidate-aware user representation. The experiments on real-world datasets show the effectiveness of our approach in improving news recommendation performance.
DocEE: A Large-Scale Dataset for Document-level Event Extraction
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Document-level Event Extraction, Event Extraction, Sentence

Event extraction (EE) is the task of identifying events and their types, along with the involved arguments. Despite the great success in sentence-level event extraction, events are more naturally presented in the form of document, with event arguments scattering in multiple sentences. However, a major barrier to promote document-level event extraction has been the lack of large-scale and practical training and evaluation datasets. In this paper, we present DocEE, a new document-level EE dataset including 20,000+ events, 100,000+ arguments. We highlight three features: large-scale annotations, fine-grained event arguments and application-oriented settings. Experiments show that even SOTA models show inferior performance on DocEE, especially in cross-domain settings, indicating that DocEE is still a challenging task. We will publish DocEE upon acceptance.
Meta-Learning with Sparse Experience Replay for Lifelong Language Learning
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Lifelong learning, Meta-Learning, Relation Extraction, text-classification, Text Classification

Lifelong learning requires models that can continuously learn from sequential streams of data without suffering catastrophic forgetting due to shifts in data distributions. Deep learning models have thrived in the non-sequential learning paradigm; however, when used to learn a sequence of tasks, they fail to retain past knowledge and learn incrementally. We propose a novel approach to lifelong learning of language tasks based on meta-learning with sparse experience replay that directly optimizes to prevent forgetting. We show that under the realistic setting of performing a single pass on a stream of tasks and without any task identifiers, our method obtains state-of-the-art results on lifelong text classification and relation extraction. We analyze the effectiveness of our approach and further demonstrate its low computational and space complexity.
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? An Extensive Empirical Study on Language Tasks
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Inductive Bias

There have been a lot of interest in the scaling properties of Transformer models. However, not much has been done on the front of investigating the effect of scaling properties of different inductive biases and model architectures. Do model architectures scale differently? If so, how does inductive bias affect scaling behaviour?  How does this influence upstream (pretraining) and downstream (transfer)? This paper conducts a systematic study of scaling behaviour of ten diverse model architectures such as Transformers, Switch Transformers, Universal Transformers, Dynamic convolutions, Performers, and recently proposed MLP-Mixers. Via extensive experiments, we show that (1) architecture is an indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales. We believe that the findings outlined in this work has significant implications to how model architectures are currently evaluated in the community.
Machine Reading Comprehension: Generative or Extractive Reader?
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Decoder, Language Modeling, Language Modelling, Machine Reading Comprehension, Multi-Task Learning, Question Answering, Reading Comprehension

While both extractive and generative readers have been successfully applied to the Question Answering (QA) task, little attention has been paid toward the comparison of these two readers. Which reader performs better? What are the reasons for the performance differences? In this paper, we aim to answer these questions in the setting of extractive QA tasks. We design multiple transformer-based models and different scenarios to systematically compare these two readers. Our findings characterize the difference of two readers and their pros and cons, which can instruct the optimal selection of the two readers, and open up new research avenues to improve each reader.
Our major findings are:
1) generative readers perform better when the input context is long, whereas extractive readers are better when the context is short;
2) extractive readers generalize better as compared to the generative ones under out-of-domain settings, in both single- and multi-task learning scenarios.  
Our experiments also suggest that, although an encoder-only pre-trained language model (PrLM) is an intuitive choice for extractive readers, the encoder from  encoder-decoder PrLM is a strong alternative that performs competitively.
Towards Better Characterization of Paraphrases
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Data Augmentation, MRPC, Paraphrase Generation, Paraphrase Identification, Position

To effectively characterize the nature of paraphrase pairs without expert human annotation, we proposes two new metrics: word position deviation (WPD) and lexical deviation (LD). WPD measures the degree of structural alteration, while LD measures the difference in vocabulary used. We apply these metrics to better understand the commonly-used MRPC dataset and study how it differs from PAWS, another paraphrase identification dataset. We also perform a detailed study on MRPC and propose improvements to the dataset, showing that it improves generalizability of models trained on the dataset. Lastly, we apply our metrics to filter the output of a paraphrase generation model and show how it can be used to generate specific forms of paraphrases for data augmentation or robustness testing of NLP models.
A Variational Hierarchical Model for Neural Cross-Lingual Summarization
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Machine Translation, Translation

The goal of the cross-lingual summarization (CLS) is to convert a document in one language (e.g., English) to a summary in another one (e.g., Chinese), which is essentially the combination of machine translation (MT) and monolingual summarization (MS). Existing studies on CLS mainly focus on utilizing pipeline methods or jointly training an end-to-end model through an auxiliary MT or MS objective. However, it is very challenging for the model to directly conduct CLS as it requires both the abilities to translate and summarize. Besides, the processes of MT and MS have a hierarchical relationship with CLS. Therefore, we propose a hierarchical model for the CLS task, based on the conditional variational auto-encoder. The hierarchical model contains two kinds of latent variables at the local and global levels, respectively. At the local level, there are two latent variables, one for translation and the other for summarization. As for the global level, there is another latent variable for cross-lingual summarization conditioned on the two local-level variables. Experiments on two language directions (English-Chinese) verify the effectiveness and superiority of the proposed approach, yielding state-of-the-art performances. In addition, we show that our model is able to generate better cross-lingual summaries than comparison models in the few-shot setting.
Towards Full Utilization on Mask Task for Distilling PLMs into NMT
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Knowledge Distillation, Machine Translation, NMT, Translation

Owing to being well-performed in many natural language processing tasks, the application of pre-trained language models (PLMs) in neural machine translation (NMT) is widely concerned. Knowledge distillation (KD) is one of the mainstream methods which could gain considerable promotion for NMT models without extra computational costs. However, previous methods in NMT always distill knowledge at hidden states level and can not make full use of the teacher models. For solving the aforementioned issue, we propose KD based on mask task as a more effective method utilized in NMT which includes encoder input conversion, mask task distillation, and gradient optimization mechanism. Here, we evaluate our translation systems for English→German and Chinese→English tasks and our methods clearly outperform baseline methods. Besides, our framework can get great performances with different PLMs.
Focus on the Target's Vocabulary: Masked Label Smoothing for Machine Translation
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Machine Translation, Translation

Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that jointly adopting these two techniques can be conflicting and even leads to sub-optimal performance, since the soft label produced by label smoothing still considers the source-side words that would not appear at the target side. To address this issue, we propose Masked Label Smoothing (MLS), a new mechanism that masks the soft label probability of source-side words to zero. Simple yet effective, MLS manages to better integrate label smoothing with vocabulary sharing and hence improves the quality of the translation. Our extensive experiments show that MLS consistently yields improvement over original label smoothing on different datasets, including bilingual and multilingual translation in both BLEU and calibration scores.