pwc-archive/papers-with-abstracts huggingface dataset:
Continuation is a Sub-Task of Fill in the Blank: Why Not Train for Both?
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR Jun 2021 6
Tasks: Position, Text Generation

The task of inserting text into a specified position in a passage, known as fill in the blank, is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. However, NLG research has mostly focused on continuation models that append text to the end of a passage. Since continuation is in fact a sub-task of fill in the blank, one where the blank is placed at the sequence's end, we propose the training of a single model which can effectively handle both these tasks.
The result is improved efficiency---as only one model needs to be maintained---without any negative impact on performance at either task.
Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR Jun 2021 6
Tasks: nlg evaluation

Recent studies have shown the advantages of evaluating NLG systems using pairwise comparisons as opposed to direct assessment. Given $k$ systems, a naive approach for identifying the top-ranked system would be to uniformly obtain pairwise comparisons from all ${k \choose 2}$ pairs of systems. However, this can be very expensive as the number of human annotations required would grow quadratically with $k$. In this work, we introduce Active Evaluation, a framework to efficiently identify the top-ranked system by actively choosing system pairs for comparison using dueling bandit algorithms. We perform extensive experiments with 13 dueling bandits algorithms on 13 NLG evaluation datasets spanning 5 tasks and show that the number of human annotations can be reduced by 80%. To further reduce the number of human annotations, we propose model-based dueling bandit algorithms which combine automatic evaluation metrics with human evaluations. Specifically, we eliminate sub-optimal systems even before the human annotation process and perform human evaluations only on test examples where the automatic metric is highly uncertain. This reduces the number of human annotations required further by 89%. In effect, we show that identifying the top-ranked system requires only a few hundred human annotations, which grow linearly with $k$. Lastly, we provide practical recommendations and best practices to identify the top-ranked system efficiently.
What is Missing in Existing Multi-hop Datasets? Toward Deeper Multi-hop Reasoning Task
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR Jun 2021 6
Tasks: Entity Linking, Machine Reading Comprehension, Reading Comprehension

Multi-hop machine reading comprehension (MRC) is a task that requires models to read and perform multi-hop reasoning over multiple paragraphs to answer a question. The task can be used to evaluate reasoning skills, as well as to check the explainability of the models, and is useful in applications (e.g., QA system). However, the current definition of hop (alias step) in the multi-hop MRC is ambiguous; moreover, previous studies demonstrated that many multi-hop examples contain reasoning shortcuts where the questions can be solved without performing multi-hop reasoning. In this opinion paper, we redefine multi-hop MRC to solve the ambiguity of its current definition by providing three different definitions of the steps. Inspired by the assessment of student learning in education, we introduce a new term of In-depth multi-hop reasoning task with three additional evaluations: step evaluation, coreference evaluation, and entity linking evaluation. In addition, we also examine the existing multi-hop datasets based on our proposed definitions. We observe that there is potential to extend the existing multi-hop datasets by including more intermediate evaluations to the task.  To prevent reasoning shortcuts, multi-hop MRC datasets should focus more on providing a clear definition for the steps in the reasoning process and preparing gold data to evaluate them.
Topic-independent Detection of Dissonance in Short Stance Text
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR Jun 2021 6
Tasks: Stance Detection

We address dissonance detection, the task of detecting conflicting stance between two input statements.
Computational models for stance detection have typically been trained for a given target topic (e.g. gun control).
In this paper, we aim for building a computational model for dissonance detection without using training data from the topic of test data.
We first build a large-scale dataset of topic-controlled arguments from two sources: (i) an online debate platform, consisting of 15k pairs of statements with support, attack, or no relation from 20 diverse topics, and (ii) Twitter, consisting of 5k pairs of statements from 5 topics.
We then evaluate a BERT-based dissonance detection model on this dataset in a topic-controlled manner.
Our experiments suggest that dissonance detection models learn the topic-independent patterns of language for detecting dissonance and generalize largely to other arguments in unseen topics.
On Evaluation and Improvement of Tail Label Performance for Multi-label Text Classification
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR Jun 2021 6
Tasks: Multi Label Text Classification, Multi-Label Text Classification, text-classification, Text Classification

Extreme multi-label text classification (XMTC) is a task for tagging each document with the most relevant subset of labels from an extremely large label set. The most challenging part for machine learning methods is the skewed label distribution in which a majority of labels receive very few training instances (named as the tail labels). Benchmark evaluations so far have focused on micro-averaging metrics, where the performance on tail labels can be easily overshadowed by high-frequency labels (named as head labels), and hence they are insufficient for evaluating the true success of methods in XMTC.  This paper presents a re-evaluation of state-of-the-art (SOTA) methods based on the binned macro-averaging F1  instead, which reveals new insights into the strengths and weaknesses of representative methods.  Based on the evaluation, we conduct in-depth analysis and experiments on Transformer models with various depths and attention mechanisms to improve the tail label performance. We show that a shallow Transformer model with word-label attentions can effectively leverage word-level features and outperforms previous Transformers on tails labels.
The AI Doctor Is In: A Survey of Task-Oriented Dialogue Systems for Healthcare Applications
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Task-Oriented Dialogue Systems

Task-oriented dialogue systems in healthcare are attracting increased attention, and have been characterized by a diverse range of architectures and objectives. However, although these systems have been surveyed in the medical community from a non-technical perspective, a systematic review from a rigorous computational perspective remains noticeably absent. As a result, many important implementation details of healthcare-oriented dialogue systems remain limited or under-specified, slowing the pace of innovation in this area.  To fill this gap, we investigated an initial pool of 4070 papers from well-known computer science, natural language processing, and artificial intelligence venues, identifying 70 papers that satisfied our defined inclusion criteria. We conducted a comprehensive technical review of the included papers, and present our findings along with identified trends and intriguing directions for future research.
Contrastive Learning of Natural Language and Code Representations for Semantic Code Search
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR Jun 2021 6
Tasks: Code Search, Contrastive Learning

Retrieving semantically relevant code functions given a natural language (NL) or programming language (PL) query is a task of great practical value towards building productivity enhancing tools for software developers. Recent approaches to solve this task involve leveraging transformer based masked language models that are pre-trained on NL and PL and fine-tuned for code search using a contrastive learning objective. However, these approaches suffer from uninformative in-batch negative samples.

We propose DyHardCode: a contrastive learning framework that leverages hard negative examples, which are mined globally from the entire training corpus to improve the quality of code and natural language representations. We experiment with different hard negative mining strategies, and provide explanations to the effectiveness of our method from the perspectives of optimization and adversarial learning. We show that DyHardCode leads to improvements in multiple code search tasks. Our approach achieves an average (across 6 programming languages) mean reciprocal ranking (MRR) score of $0.750$ as opposed to the previous state of the art result of $0.713$ MRR on the CodeSearchNet benchmark.
Best Practices for Noise-Based Augmentation to Improve the Performance of Deployable Speech-Based Emotion Recognition Systems
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR Jun 2021 6
Tasks: Data Augmentation, Emotion Recognition, speech-recognition, Speech Recognition

Emotion recognition models are a key component of several downstream applications, such as mental health assessments. These models are usually trained on small, clean, and synthetically controlled datasets, which leads to high failure rates in presence of `unseen' background noises, promoting noise-overlay based adversarial attacks. Noisy data augmentation has aided robustness of speech recognition and classification models, wherein, the ground truth label remains consistent even in the presence of noise which, isn't always true for subjectively perceived emotion labels. In this work, we create realistic noisy samples of IEMOCAP, using multiple categories of environmental and synthetic noise. We evaluate how ground truth labels (human) and predicted labels (model) change as a function of these noise source introductions. We show that some commonly used noisy augmentation techniques, impact human perception of emotion, thus, falsifying the `clean ground truth label. Our experiments show that the performance of both, baseline, and even denoised emotion recognition models significantly declines on noisy samples as compared to that on the clean set. This performance degradation prevails when model is trained on a combination of clean and test set mismatched noisy samples. We investigate how using the above found `human-perceptible noise overlays can lead to inaccurate metrics when testing the model for robustness or vulnerability to adversarial attacks. Finally, we present a set of recommendations for noise-based augmentation of speech emotion datasets and for deploying the models trained using those datasets.
Privacy-Preserving Graph Convolutional Networks for Text Classification
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Classification, Privacy Preserving, Representation Learning, text-classification, Text Classification

Graph convolutional networks (GCNs) are a powerful architecture for representation learning on documents that naturally occur as graphs, e.g., citation or social networks. However, sensitive personal information, such as documents with people's profiles or relationships as edges, are prone to privacy leaks, as the trained model might reveal the original input. Although differential privacy (DP) offers a well-founded privacy-preserving framework, GCNs pose theoretical and practical challenges due to their training specifics. We address these challenges by adapting differentially-private gradient-based training to GCNs and conduct experiments using two optimizers on five NLP datasets in two languages. We propose a simple yet efficient method based on random graph splits that not only improves the baseline privacy bounds by a factor of 2.7 while retaining competitive $F_1$ scores, but also provides strong privacy guarantees of $\varepsilon = 1.0$. We show that, under certain modeling choices, privacy-preserving GCNs perform up to 90\% of their non-private variants, while formally guaranteeing strong privacy measures.
A Study on Summarizing and Evaluating Long Documents
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Document Summarization, Text Generation, Text Summarization

Text summarization has been a key language generation task for over 60 years. The field has advanced considerably during the past two years, benefiting from the proliferation of pre-trained Language Models (LMs). However, the field is constrained by two factors: 1) the absence of an effective automatic evaluation metric and 2) a lack of effective architectures for long document summarization. Our first contribution is to demonstrate that a set of semantic evaluation metrics (BERTScore, MoverScore and our novel metric, BARTScore) consistently and significantly outperform ROUGE. Using these metrics, we then show that combining transformers with sparse self-attention is a successful method for long document summarization and is very competitive with the state of the art. Finally, we show that sparsifying self-attention does not degrade model performance when using transformers for summarization.
Analysis and Prediction of NLP models via Task Embeddings
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Transfer Learning

 Relatedness between tasks, which is key to transfer learning, is often characterized by measuring the influence of tasks on one another during sequential or simultaneous training, with tasks being treated as black boxes.  In this paper, we propose MetaEval, a set of $101$ NLP tasks. We fit a single transformer to all MetaEval tasks jointly while conditioning it on low-dimensional task embeddings. The resulting task embeddings enable a novel analysis of the relatedness among tasks. We also show that task aspects can be used to predict task embeddings for new tasks without using any annotated examples. Predicted embeddings can modulate the encoder for zero-shot inference and outperform a zero-shot baseline on GLUE tasks. The provided multitask setup can function as a benchmark for future transfer learning research.
Opinion-based Relational Pivoting for Cross-domain Aspect Term Extraction
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Domain Adaptation, Term Extraction

Domain adaptation methods often exploit domain-transferable input features, a.k.a. pivots. 
The task of Aspect and Opinion Term Extraction presents a special challenge for domain transfer: while opinion terms largely transfer across domains, aspects change drastically from one domain to another (e.g. from \textit{restaurants} to \textit{laptops}). 
In this paper, we investigate and establish empirically a prior conjecture, which suggests that the linguistic relations connecting opinion terms to their aspects transfer well across domains and therefore can be leveraged for cross-domain aspect term extraction.
We present several analyses supporting this conjecture, via experiments with four linguistic dependency formalisms to represent relation patterns. 
Following, we present an aspect term extraction method that drives models to consider opinion--aspect relations via explicit multitask objectives. 
This method provides significant performance gains, even on top of a prior state-of-the-art linguistically-informed model, which are shown in analysis to stem from the relational pivoting signal.
BdLAN:BERTdoc Label Attention Networks for Multi-label text classification
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Classification, Information Retrieval, Multi Label Text Classification, Multi-Label Text Classification, Retrieval, TAG, text-classification, Text Classification

Multi-label text classification (MLTC) brings us new challenge in Natural Language Processing (NLP) which aims at assigning multiple labels for a given document. Many real-world tasks can be view as MLTC, such as tag recommendation, information retrieval, etc. However, several flinty problems are placed in the presence of researchers about how to establish connections between labels or distinguish similar sub-labels, which haven't been solved thoroughly by current endeavor. Therefore, we proposed a novel framework named BdLAN, BERTdoc Label Attention Networks in this paper, consist of the BERTdoc layer, the label embeddings layer, the doc encoder layer, the doc-label attention layer and the prediction layer. We apply a powerful technique BERT to pretrain documents to capture their deep semantic features and encode them via Bi-LSTM to obtain a two-directional contextual representation of uniform length. Then we create label embeddings and feed them together with encoded-pretrained-documents to doc-label attention mechanism to obtain interactive information between documents and their corresponding labels, finally using MLP to make prediction.We carry out experiments on three real-world datasets, the empirical results indicating our proposed model outperforms all state-of-the-art MLTC benchmarks. Moreover, we conduct a case study, visualizing real application of our BdLAN model vividly.
SINA-BERT: A Pre-Trained Language Model for Analysis of Medical Texts in Persian
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Language Modeling, Language Modelling, Medical Named Entity Recognition, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Retrieval, Sentiment Analysis

We have released SINA-BERT, a language model pre-trained on BERT to address the lack of a high-quality Persian language model in the medical domain. SINA-BERT utilizes pre-training on a large-scale corpus of medical contents including formal and informal texts collected from various online resources in order to improve the performance on health-care related tasks. We employ SINA-BERT to complete following representative tasks: categorization of medical questions, medical sentiment analysis, medical named entity recognition, and medical question retrieval. For each task, we have developed Persian annotated data sets for training and evaluation and learnt a representation for the data of each task especially complex and long medical questions. With the same architecture being used in each task, SINA-BERT outperforms BERT-based models that were previously made available in the Persian language.
How to Query Language Models?
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5

Large pre-trained language models (LMs) are capable of not only recovering linguistic but also factual and commonsense knowledge. To access the knowledge stored in mask-based LMs, we can use cloze-style questions and let the model fill in the blank. 
The flexibility advantage over structured knowledge bases comes with the drawback of finding the right query for a certain information need. Inspired by human behavior to disambiguate a question, we propose to query LMs by example. To clarify the ambivalent question \textit{Who does Neuer play for?}, a successful strategy is to demonstrate the relation using another subject, e.g., \textit{Ronaldo plays for Portugal. Who does Neuer play for?}. We apply this approach of querying by example to the LAMA probe and obtain substantial improvements of up to 37.8\% for BERT-large on the T-REx data when providing only 10 demonstrations---even outperforming a baseline that queries the model with up to 40 paraphrases of the question. The examples are provided through the model's context and thus require neither fine-tuning nor an additional forward pass. This suggests that LMs contain more factual and commonsense knowledge than previously assumed---if we query the model in the right way.
A Legal Approach to Hate Speech – Operationalizing the EU’s Legal Framework against the Expression of Hatred as an NLP Task
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decision Making, Hate Speech Detection

We propose a 'legal approach' to hate speech detection by operationalization of the decision as to whether a post is subject to criminal law into an NLP task.
Comparing existing regulatory regimes for hate speech, we base our investigation on the European Union's framework as it provides a widely applicable legal minimum standard.
Accurately judging whether a post is punishable or not usually requires legal training.
We show that, by breaking the legal assessment down into a series of simpler sub-decisions, even laypersons can annotate consistently.
Based on a newly annotated dataset, our experiments show that directly learning an automated model of punishable content is challenging.
However, learning the two sub-tasks of `target group' and `targeting conduct' instead of an end-to-end approach to punishability yields better results.
Overall, our method also provides for better explainability and higher transparency, which is a crucial point in legal decision-making.
Detecting Adversarial Text Attacks via SHapley Additive exPlanations
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Adversarial Text, SST-2

State-of-the-art machine learning models are prone to adversarial attacks: maliciously crafted inputs to fool the model into making a wrong prediction, often with high confidence. While defense strategies have been extensively explored in the computer vision domain, research in natural language processing still lacks techniques to make models resilient to adversarial text inputs. We propose an adversarial detector leveraging Shapley additive explanations against text attacks. Our approach outperforms the current state-of-the-art detector by around 19% F1-score on the IMDb and 14% on the SST-2 datasets while also showing competitive performance on AG_News and Yelp Polarity. Furthermore, we prove the detector to only require a low amount of training samples and, in some cases, to generalize to different datasets without needing to retrain.
Coming to its senses: Lessons learned from Approximating Retrofitted BERT representations for Word Sense information
Authors: Anonymous
Date: 2021-05-16 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Word Embeddings

Retrofitting static vector space word representations using external knowledge bases has yielded substantial improvements in their lexical-semantic capacities but is non-trivial to apply to contextual word embeddings (CWE). In this paper, we propose MAKESENSE, a method that 'approximates' retrofitting in CWEs to better infer word sense knowledge from word contexts. We specifically analyze BERT and MAKESENSE-transformed BERT representations over a diverse set of experiments encompassing sense-sensitive similarities, alignment with human-elicited similarity judgments, and probing tasks focusing on sense distinctions and hypernymy. Our findings indicate that MAKESENSE imparts substantial improvements in word sense information over vanilla CWEs but largely preserves more complex usage of sense and directionally sensitive information such as hypernymy.