pwc-archive/papers-with-abstracts huggingface dataset:
Learning Rich Representation of Keyphrases from Text
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Abstractive Text Summarization, Keyphrase Extraction, Keyphrase Generation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Question Answering, Relation Extraction

In this work, we explore how to learn task-specific language models aimed towards learning rich representation of keyphrases from text documents. We experiment with different masking strategies for training transformer language models (LMs) in discriminative as well as generative settings. In the discriminative setting, we introduce a new pre-training objective - Keyphrase Boundary Infilling with Replacement (KBIR), showing large gains in performance (upto 9.26 points in F1) over SOTA, when LM pre-trained using KBIR is fine-tuned for the task of keyphrase extraction. In the generative setting, we introduce a new training setup for BART - KeyBART, that reproduces the keyphrases related to the input text in the CatSeq format, instead of the denoised original input. This also led to gains in performance (upto 4.33 points in F1@M) over SOTA for keyphrase generation. Additionally, we also fine-tune the pre-trained language models on named entity recognition (NER), question answering (QA), relation extraction (RE), abstractive summarization and achieve comparable performance with that of the SOTA, showing that learning rich representation of keyphrases is indeed beneficial for many other fundamental NLP tasks.
Know your tools well: Better $\textit{and}$ faster QA with synthetic examples
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Diversity, Knowledge Distillation, Machine Reading Comprehension, Reading Comprehension

Synthetic training data---commonly used to augment human-labeled examples in supervised learning---are often noisy, but can be generated in very large quantities and diversity. This paper proposes to leverage these unique attributes in a targeted manner to maximize the utility of synthetic examples. Via two novel applications that utilize synthetic data for targeted pre-training and knowledge distillation, we demonstrate the feasibility of this idea for machine reading comprehension (MRC). Using our proposed methods, we are able to train simultaneously $\textbf{\textit{smaller}}$, $\textbf{\textit{faster}}$ and $\textbf{\textit{more accurate}}$ MRC models than existing synthetic augmentation methods. Our methods are generic in nature and can be applied to any task for which synthetic data can be generated.
Finding the Right Recipe for Low Resource Domain Adaptation in Neural Machine Translation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: 8k, Domain Adaptation, es-en, Machine Translation, Translation

Despite the considerable amount of parallel data used to train neural machine translation models, they can still struggle to generate fluent translations in technical domains. In-domain parallel data is often very low resource and synthetic domain data generated via back-translation is frequently lower quality. To guide machine translation practitioners and characterize the effectiveness of domain adaptation methods under different data availability scenarios, we conduct an in-depth empirical exploration of monolingual and parallel data approaches to domain adaptation. We compare mixed domain fine-tuning, traditional back-translation, tagged back-translation, and shallow fusion with domain specific language models in isolation and combination. We study method effectiveness in very low resource (8k parallel examples) and moderately low resource (46k parallel examples) conditions. We demonstrate the advantages of augmenting clean in-domain parallel data with noisy mined in-domain parallel data and propose an ensemble approach to alleviate reductions in original domain translation quality. Our work includes three domains: consumer electronic, clinical, and biomedical and spans four language pairs - Zh-En,  Ja-En, Es-En, and Ru-En. We make concrete recommendations for achieving high in-domain performance. We release our consumer electronic and clinical domain datasets for all languages and make our code publicly available.
On the Sensitivity and Stability of Model Interpretations
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Adversarial Robustness, Dependency Parsing, model, Sensitivity, text-classification, Text Classification

Recent years have witnessed the emergence of a variety of post-hoc interpretations that aim to uncover how natural language processing (NLP) models make predictions. Despite the surge of new interpretation methods, it remains an open problem how to define and quantitatively measure the faithfulness of interpretations, i.e., to what extent interpretations reflect the reasoning process by a model. We propose two new criteria, sensitivity and stability, that provide complementary notions of faithfulness to the existed removal-based criteria. Our results show that the conclusion for how faithful interpretations are could vary substantially based on different notions. Motivated by the desiderata of sensitivity and stability, we introduce a new class of interpretation methods that adopt techniques from adversarial robustness. Empirical results show that our proposed methods are effective under the new criteria and overcome limitations of gradient-based methods on removal-based criteria. Besides text classification, we also apply interpretation methods and metrics to dependency parsing. Our results shed light on understanding the diverse set of interpretations.
Curriculum Data Augmentation for Low-Resource Slides Summarization
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Articles, Data Augmentation, Text Summarization

Data augmentation is commonly used in training in low-resource scenarios. However, there are sometimes large discrepancy between distributions of augmented data and target data. How to bridge the gap between the augmented and target data, especially when target data is harder-to-learn? In this paper, we study improved data augmentation strategies in the scenario of scientific slides text summarization, where we generate a textual summary based on texts of presentation slides. Since slides are messy and difficult to understand by current models, we introduce an easier form of data, i.e., articles in natural language. The basic idea is that we generate the transition data between slides and articles, and all three of them form a curriculum for neural models to learn the distribution transition from article data to slides data. We find that our approach achieves consistent improvements over different backbone summarization models. The curriculum-oriented data augmentation method can generate data that fill the gap between the easy-to-obtain data and the low-resource task data. We show that curriculum learning and data augmentation can be combined to help NLP models learn from otherwise hard-to-learn data.
BotsTalk: Machine-Sourced Framework for Automatic Curation of Large-scale Multi-skill Dialogue Datasets
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Chatbot, Dialogue Generation, Multi-Task Learning

Previous work in open-domain chatbots has introduced dialogue corpora and tasks that aim to inject dialogue systems different communicative skills such as being personable, knowledgeable and empathetic. With the advent of conversational agents grounded to specific skills, a new challenge in open-domain chatbots has been posed: A good open-domain chatbot should retain a well-rounded set of skills and seamlessly blend them into a conversation. To this end, a new dialogue dataset Blended Skill Talk is collected via crowdsourcing and commonly used as a benchmark for multi-skill dialogue generation. However, such data construction approach requires labor intensive manual annotation, which severely limits their utility on large-scale learning. In this work, we propose BotsTalk, a novel machine-sourced framework, where several agents participate in a conversation to automatically annotate multi-skill dialogues. We then present Blended Skill BotsTalk (BS$\mathbb{B}$T), a large-scale multi-skill dialogue dataset of 200K conversations. Experimental results show that our dataset can be effectively used as training data for multi-skill dialogue systems which require an understanding of both skill blending and grounding. We also demonstrate the dataset is orthogonally applicable to diverse learning schemes such as fine-tuning and multi-task learning.
CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Contrastive Learning, Few-shot NER, Named Entity Recognition, Named Entity Recognition (NER)

Named Entity Recognition (NER) in Few-Shot setting is imperative for entity tagging in low resource domains. Existing approaches only learn class-specific semantic features and intermediate representations from source domains. This affects generalizability to unseen target domains, resulting in suboptimal performances. To this end, we present CONTaiNER, a novel contrastive learning technique that optimizes the inter-token distribution distance for Few-Shot NER. Instead of optimizing class-specific attributes, CONTaiNER optimizes a generalized objective of differentiating between token categories based on their Gaussian-distributed embeddings. This effectively alleviates overfitting issues originating from training domains. Our experiments in several traditional test domains (OntoNotes, CoNLL'03, WNUT '17, GUM) and a new large scale Few-Shot NER dataset (Few-NERD) demonstrate that on average, CONTaiNER outperforms previous methods by 3%-13% absolute F1 points while showing consistent performance trends, even in challenging scenarios where previous approaches could not achieve appreciable performance.
ME-GCN: Multi-dimensional Edge-Embedded Graph Convolutional Networks for Semi-supervised Text Classification
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Semi-Supervised Text Classification, text-classification, Text Classification

Compared to sequential learning models, graph-based neural networks exhibit excellent ability in capturing global information and have been used for semi-supervised learning tasks, including citation network analysis or text classification. However, most GCNs are designed with the single-dimensional edge feature and neglected to utilise the rich edge information about graphs. In this paper, we introduce the ME-GCN (Multi-dimensional Edge-enhanced Graph Convolutional Networks) for semi-supervised text classification. A text graph for an entire corpus is firstly constructed to describe the undirected and multi-dimensional relationship of word-to-word, document-document, and word-to-document. The graph is initialised with corpus-trained multi-dimensional word and document node representation, and the relations are represented according to the distance of those words/documents nodes. Then, the generated graph is trained with ME-GCN, which considers the edge features as multi-stream signals, and each stream performs a separate graph convolutional operation. Our ME-GCN can integrate a rich source of graph edge information of the entire text corpus. The results have demonstrated that our proposed model has significantly outperformed the state-of-the-art methods across eight benchmark datasets.
Comparing Apples and Oranges: Recognizing Political Heterogeneity on Reddit and Its Implications for Behavioral Analysis
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10

Reddit is home to a broad spectrum of political activity and users signal their political affiliations in multiple ways—from self-declarations to community participation. Commonly, political studies have assumed political users are a single bloc, both in developing models to infer political leaning and in studying political behavior. Here, test this model assumption of political users. We show that a variety of commonly-used political-inference approaches models do not generalize, indicating heterogeneous types of political users, and remains imprecise at best for most users, regardless of which sources of data or methods are used. Across a 14-year longitudinal analysis, we demonstrate that the choice in definition of a political user has significant implications for behavioral analysis. Controlling for multiple factors, political users are more toxic on the platform and inter-party interactions are even more toxic---but not all political users behave this way.  Last, we identify a subset of political users who repeatedly flip affiliations, showing that these users are the most controversial of all, acting as provocateurs by more frequently bringing up politics, and are more likely to be banned, suspended, or deleted.
Cooperative Semi-Supervised Transfer Learning of Machine Reading Comprehension
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Extractive Question-Answering, Machine Reading Comprehension, Question Answering, Question Generation, Question-Generation, Reading Comprehension, Transfer Learning, Word Embeddings

Pretrained language models have significantly improved the performance of down-stream language understanding tasks, including extractive question answering, by providing high-quality contextualized word embeddings. However, training question answering models still requires large amounts of annotated data for specific domains. In this work, we propose a cooperative, self-play learning framework, REGEX, for automatically generating more non-trivial question-answer pairs to improve model performance. REGEX is built upon a masked answer extraction task with an interactive learning environment containing an answer entity REcognizer, a question Generator, and an answer EXtractor. Given a passage with a masked entity, the generator generates a question around the entity, and the extractor is trained to extract the masked entity with the generated question and raw texts. The framework allows the training of question generation and answering models on any text corpora without annotation. We further leverage a reinforcement learning technique to reward generating high-quality questions and to improve the answer extraction model's performance. Experiment results show that REGEX outperforms the state-of-the-art (SOTA) pretrained language models and transfer learning approaches on standard question-answering benchmarks, and yields the new SOTA performance under given model size and transfer learning settings.
InducT-GCN: Inductive Graph Convolutional Networks for Text Classification
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Classification, Graph Neural Network, text-classification, Text Classification

Text classification aims to assign labels to textual units by making use of global information. Recent studies have applied graph neural network (GNN) techniques to capture the global word co-occurrence in a corpus. Most existing approaches require that all the nodes (training and test) in a graph are present during training, which are transductive and do not naturally generalise to unseen nodes. To make those models \textit{inductive}, previous works use extra resources, like pretrained word embedding. However, high-quality resource is not always available and can be hard to train. Under the extreme settings 
with no extra resource and limited amount of training set, can we still learn an inductive graph-based text classification model? In this paper, we introduce a novel inductive graph-based text classification framework, namely InducT-GCN (InducTive Graph Convolutional Networks for Text classification). 
Compared to transductive models that require test documents in training, we construct a graph based on the statistics of training documents only and represent document vectors with a weighted sum of word vectors. We then conduct one-directional GCN propagation during testing. Across five text classification benchmarks, our InducT-GCN outperformed state-of-the-art methods that are either transductive in nature or pre-trained additional resources. We also conducted scalability testing by gradually increasing the data size and revealed that our InducT-GCN can reduce the time and space complexity.
C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Open-Domain Question Answering, Question Answering

We consider the problem of pretraining a two-stage open-domain question answering (QA) system (retriever + reader) with strong transfer capabilities. The key challenge is how to construct a large amount of high-quality question-answer-context triplets without task-specific annotations. Specifically, the triplets should align well with downstream tasks by: (i) covering a wide range of domains (for open-domain applications), (ii) linking a question to its semantically relevant context with supporting evidence (for training the retriever), and (iii) identifying the correct answer in the context (for training the reader). Previous pretraining approaches generally fall short of one or more of these requirements. In this work, we automatically construct a large-scale corpus that meets all three criteria by consulting millions of references cited within Wikipedia. The well-aligned pretraining signals benefit both the retriever and the reader significantly. Our pretrained retriever leads to 2%-10% absolute gains in top-20 accuracy. And with our pretrained reader, the entire system improves by up to 4% in exact match.
Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuning
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Few-Shot Learning

Prompt-based learning for Pre-trained Language Models (PLMs) has achieved remarkable performance in few-shot learning by exploiting prompts as task guidance and turning downstream tasks into masked language problems. In most existing approaches, the high performance of prompt-based learning heavily relies on handcrafted prompts and verbalizers, which may limit the application of such approaches in real-world scenarios. To solve this issue, we present CP-Tuning, the first end-to-end Contrastive Prompt Tuning framework for PLMs without any manual engineering of task-specific prompts and verbalizers. It is integrated with the task-invariant continuous prompt encoding technique with fully trainable prompt parameters. We further propose a pair-wise cost-sensitive contrastive loss to optimize the model in order to achieve verbalizer-free class mapping and enhance the task-invariance of prompts. Experiments over a variety of NLP tasks show CP-Tuning consistently outperforms state-of-the-art methods.
Attention Temperature Matters in Abstractive Summarization Distillation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Abstractive Text Summarization, Text Summarization

Recent progress of abstractive text summarization largely relies on large pre-trained sequence-to-sequence Transformer models, which are computationally expensive. This paper aims to distill these large models into smaller ones for faster inference and with minimal performance loss. Pseudo-labeling based methods are popular in sequence-to-sequence model distillation. In this paper, we find simply manipulating attention temperatures in Transformers can make pseudo labels easier to learn for student models. Our experiments on three summarization datasets show our proposed method consistently improves vanilla pseudo-labeling based methods. Further empirical analysis shows that both pseudo labels and summaries produced by our students are shorter and more abstractive.
Hate a Little Less, Love a Little More! Proactively Curbing Online Hatred via Hate Speech Normalization
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Hate Speech Normalization, Text Generation

Curbing online hate speech has become the need of the hour; however, a blanket ban on such activities is infeasible due to several political, geographical, and cultural reasons. To reduce the severity of the problem, in this paper, we introduce a novel task, hate speech normalization – weakening the intensity of hatred exhibited by an online post by paraphrasing the original content. The intention of hate speech normalization is to not support hate but instead, provide the users with a stepping stone towards non-hate while giving online platforms more time to monitor any improvement in the user’s behaviour. To this end, we manually curated a parallel corpus of hate texts and their normalized counterparts (a normalized text is less hateful and more benign). We then introduce NACL, a
Neural hAte speeCh normaLizer that operates in three stages – first, it measures the hate intensity of the original sample; second, it identifies the harmful span(s) within it; and finally, it reduces hate intensity by paraphrasing the hate spans. We perform extensive experiments to measure the efficacy of individual components and the overall performance of NACL via three-way evaluation (intrinsic, extrinsic, and human-study). We observe that NACL outperforms its respective baselines – NACL yields a score of 0.683 Pearson correlation for the intensity prediction, 0.6911 F1-score in the span identification, and 67.71 BLEU and 75.83 perplexity for the normalized text generation. We further show the generalizability of NACL across other platforms (Reddit, Facebook, Gab). A scalable prototype of NACL was also deployed for the user study.
Better Sample Efficiency Does Not Imply Out-of-Distribution Robustness
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10

We study the relationship between sample efficiency and out-of-distribution performance---if two models have the same in-distribution performance, does the model trained on fewer labeled training examples (higher sample efficiency) perform better out-of-distribution? First, we find that models with higher sample efficiency can have worse out-of-distribution robustness than models that are less sample-efficient. We then empirically study the correlation between sample efficiency and out-of-distribution robustness across three tasks, 23 total ID-OOD settings, and four broadly-applicable methods that change sample efficiency: (1) changing the pre-training data source; (2) using natural language prompts; (3) increasing model size; and (4) increasing the amount of pre-training data. Given that better sample efficiency does not necessarily give rise to robust models, our results underscore the importance of developing and evaluating whether interventions jointly improve both.