pwc-archive/papers-with-abstracts huggingface dataset:
An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: counterfactual, Data Augmentation, Language Modeling, Language Modelling

Recent work has shown pre-trained language models capture social biases from the text corpora they are trained on. This has attracted attention to developing techniques that mitigate such biases. In this work, we perform an empirical survey of five recently proposed bias mitigation techniques: Counterfactual Data Augmentation (CDA), Dropout, Iterative Nullspace Projection, Self-Debias,  and SentenceDebias.
We quantify the effectiveness of each technique using three intrinsic bias benchmarks while also measuring the impact of these techniques on a model's language modeling ability, as well as its performance on downstream NLU tasks. We experimentally find that: (1) Self-Debias is the strongest debiasing technique, obtaining improved scores on all bias benchmarks; (2) Current debiasing techniques perform less consistently when mitigating non-gender biases; And (3) improvements on bias benchmarks such as StereoSet and CrowS-Pairs by using debiasing strategies are often accompanied by a decrease in language modeling ability, making it difficult to determine whether the bias mitigation was effective.
DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queries
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Question Answering, Text to SQL, Text-To-SQL

This paper develops the first question answering dataset (DrugEHRQA) containing question-answer pairs from both structured tables and unstructured notes from a publicly available Electronic Health Record (EHR). EHRs contain patient records, stored in structured tables as well as unstructured clinical notes. The information in structured and unstructured EHR records is not strictly disjoint: information may be duplicated, contradictory, or provide additional context between these sources. This presents a rich opportunity to study question answering (QA) models that combine reasoning over both structured and unstructured data. Additionally, we propose a novel methodology that automatically generates a large QA dataset by retrieving answers from both structured and unstructured EHR records. The automatically-generated dataset has medication-related queries, containing over 70,000 question-answer pairs.  Our dataset is validated for both individual modalities using state-of-the-art QA models. In order to address the problem arising from complex, nested queries, this is the first time Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers (RAT-SQL) has been used for EHR data. Finally, we introduce a rule-based method to obtain multi-modal answers, combining the answers from the different modalities. Our goal is to provide a benchmark dataset for multi-modal QA systems, and to open up new avenues of research in improving question answering over EHR structured data by using context from unstructured clinical data.
An Exploration of Prompt-Based Zero-Shot Relation Extraction Method
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Relation, Relation Extraction

Zero-shot relation extraction is an important method for dealing with the newly emerging relations in the real world which lacks labeled data.
However, the current zero-shot methods usually rely on large-scale and in-domain labeled data of predefined relations.
In this work, we view zero-shot relation extraction as a semantic match task optimized by prompt-tuning, which maintains a high generalization ability when using even one labeled data in per predefined relations.
Specifically, we reduce the dependence on labeled data of predefined relation with the help of the existing knowledge in the pretrained language model (PLMs). To induce this knowledge, we fuse the original input with the prompt template to formulate a cloze-style task, which is consistent with the pretraining stage. The induced knowledge facilitates new relation discovery when large-scale labeled data are not available.
Experiment results on two academic datasets show that our method achieves similar performance with only 0.05\% labeled data of pre-defined relations, compared with current state-of-the-art method. Using all labeled data, our method improves the F1 score by nearly 30\% and 9\% on the two datasets respectively.
Can Pre-trained Language Models Interpret Similes as Smart as Human?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentiment Analysis, Sentiment Classification

Simile interpretation is a crucial task in natural language processing. Nowadays, pre-trained language models (PLMs) have achieved state-of-the-art performance on many tasks. However, it remains under-explored whether PLMs can interpret similes or not. In this paper, we investigate the ability of PLMs in simile interpretation by designing a novel task named Simile Property Probing, i.e., to let the PLMs infer the shared properties of similes. We construct our simile property probing datasets from both general textual corpus and human-designed questions, which contain a total of 1,633 examples covering seven main categories. Our empirical study based on the constructed datasets shows that PLMs exhibit the ability to infer shared properties of similes, while they still underperform humans. To bridge the gap with human performance, we additionally design a knowledge-enhanced training objective by incorporating the simile knowledge into PLMs via knowledge embedding methods. Our method brings up to an 8.58% gain in the probing task, and up to a 1.37% gain in the downstream task of sentiment classification. The datasets and code will be publicly available soon.
ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9

Pretrained language models (PLMs), such as BERT and GPT-3, have dominated the majority of NLP tasks. However, relatively little work has been conducted on systematically evaluating the language abilities of PLMs. In this paper, we present a large-scale empirical study on gen\underline{E}ral \underline{l}anguage ab\underline{i}li\underline{t}y \underline{e}valuation of PLMs (ElitePLM). We first design four evaluation dimensions in ElitePLM, including memory, comprehension, reasoning, and composition, and further measure ten widely-used PLMs within five categories. Our empirical results demonstrate that: (1) the pretraining objectives and strategies have significant impacts on PLMs performance in downstream tasks; (2) fine-tuning PLMs in downstream tasks is usually sensitive to the data size and distribution; (3) PLMs have excellent transferability between similar tasks. Our experimental results summarize several important findings, which can guide the future work to choose, apply, and design  PLMs for specific tasks. We have made all the details of experiments publicly available at https://anonymous.4open.science/r/Paper-for-ACL-4FD1.
On the Multilingual Capabilities of Very Large-Scale English Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Extractive Question-Answering, Few-Shot Learning, Language Modeling, Language Modelling, Question Answering, Text Generation, Text Summarization

Generative Pre-trained Transformers (GPTs) have recently been scaled to unprecedented sizes in the history of machine learning. These models, solely trained on the language modeling objective, have been shown to exhibit outstanding zero, one, and few-shot learning capabilities in a number of different tasks. Nevertheless, aside from anecdotal experiences, little is known regarding their multilingual capabilities, given the fact that the pre-training corpus is almost entirely composed of English text. In this work, we investigate its potential and limits in three tasks: extractive question-answering, text summarization and natural language generation for five different languages, as well as the effect of scale in terms of model size. Our results show that GPT-3 can be almost as useful for many languages as it is for English, with room for improvement if optimization of the tokenization is addressed.
Sentence-Level Discourse Parsing as Text-to-Text Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Discourse Parsing, Discourse Segmentation, Segmentation, Sentence, Text Generation

Previous studies have made great advances in RST discourse parsing through neural frameworks or efficient features, but they split the parsing process into two subtasks and heavily depended on gold segmentation. In this paper, we introduce an end-to-end method for sentence-level RST discourse parsing via transforming it into a text-to-text generation task. Our method unifies the traditional two-stage parsing and generates the parsing tree directly from the input text without requiring a complicated model. Moreover, the EDU segmentation can be simultaneously generated and extracted from the parsing tree. Experimental results on the RST Discourse Treebank demonstrate that our proposed method outperforms existing methods in both tasks of sentence-level RST parsing and discourse segmentation. Considering the lack of annotated data in RST parsing, we also create high-quality augmented data based on several filtering strategies, which further improves the performance.
Bootstrapping Text Anonymization Models with Distant Supervision
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Text Anonymization

We propose a novel method to bootstrap text anonymization models based on distant supervision.  Instead of requiring manually labeled training data, the approach relies on a knowledge graph expressing the background information assumed to be publicly available about various individuals. This knowledge graph is employed to automatically annotate text documents including personal data about a subset of those individuals. More precisely, the method determines which text spans ought to be masked in order to guarantee $k$-anonymity, assuming an adversary with access to both the text documents and the background information expressed in the knowledge graph. The resulting collection of labeled documents is then used as training data to fine-tune a pre-trained language model for text anonymization. We illustrate this approach using a knowledge graph extracted from Wikidata and short biographical texts from Wikipedia. Evaluation results with a BERT-based model and a manually annotated collection of 553 summaries showcase the potential of the approach, but also unveil a number of issues that may arise the knowledge graph is noisy or incomplete.  The results also illustrate that, contrary to most sequence labeling problems, the text anonymization task may admit several alternative solutions.
DS-TOD: Efficient Domain Specialization for Task-Oriented Dialog
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: dialog state tracking, Language Modeling, Language Modelling, Masked Language Modeling, Retrieval

Recent work has shown that self-supervised dialog-specific pretraining on large conversational datasets yields substantial gains over traditional language modeling (LM) pretraining in downstream task-oriented dialog (TOD). These approaches, however, exploit general dialogic corpora (e.g., Reddit) and thus presumably fail to reliably embed domain-specific knowledge useful for concrete downstream TOD domains. In this work, we investigate the effects of domain specialization of pretrained language models (PLMs) for TOD. Within our DS-TOD framework, we first automatically extract salient domain-specific terms, and then use them to construct DomainCC and DomainReddit -- resources that we leverage for domain-specific pretraining, based on (i) masked language modeling (MLM) and (ii) response selection (RS) objectives, respectively. We further propose a resource-efficient and modular domain specialization by means of domain adapters -- additional parameter-light layers in which we encode the domain knowledge. Our experiments with prominent TOD tasks -- dialog state tracking (DST) and response retrieval (RR) -- encompassing five domains from the MultiWOZ benchmark demonstrate the effectiveness of DS-TOD. Moreover, we show that the light-weight adapter-based specialization (1) performs comparably to full fine-tuning in single domain setups and (2) is particularly suitable for multi-domain specialization, where besides advantageous computational footprint, it can offer better downstream performance.
A Unified and Efficient Contrastive Learning Framework for Text Summarization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Abstractive Text Summarization, Contrastive Learning, Text Summarization

Both extractive and abstractive summarization systems share a common problem, i.e., there is a  mismatch between the training object and evaluation metrics. To bridge this gap, we introduce a unified and efficient contrastive learning framework called UniCLS for both extractive and abstractive methods. In this framework, the summarization model not only learns how to write summaries but also is required to evaluate the generated summary. Without any additional parameters, our contrastive learning approach can be effortlessly applied to any extractive and abstractive summarization systems. Extensive experiments show that our framework brings substantial improvements on a wide range of datasets and clearly outperforms previous state-of-the-art end-to-end systems by a large margin on CNN/DailyMail benchmark. In Particular,  UniCLS achieves an improvement of 1.75 ROUGE-1 score over the BART-large model without sacrificing inference efficiency.
NewsEdits: A Dataset of News Article Revision Histories and a Novel Approach to Document-Level Edit Reasoning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Articles, Sentence

News article revision histories have the potential to give us novel insights across varied fields of linguistics and social sciences. In this work, we present the first publicly available dataset of news revision histories, \textit{NewsEdits}. 

Our dataset is massive and multilingual; it contains 1.2 million articles with 4.6 million versions from over 22 English- and French-language newspaper sources based in three countries. We develop a highly-scalable sentence-matching algorithm which we use to reliably extract document and sentence-level edit actions: Add, Delete, Edit, Move.

We conduct analyses characterizing the nature of edits. We show that sentences that are added and deleted between article versions are more likely to contain updating events, main content and quotes compared with unchanged sentences. Finally, we introduce three novel tasks aimed at predicting the edit actions given an old version. We show that they are learnable, but challenging for current large NLP models compared with expert human judgement. By offering insights into how news articles grow and update, we hope this can spur research in narrative framing and development and the informational needs of updating news stories.
Metaphor Detection for Low Resource Languages: From Zero-Shot to Few-Shot Learning in Middle High German
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Few-Shot Learning, Word Embeddings

In this work, we present a novel unsupervised
method for adjective-noun metaphor detection
on low resource languages. We propose two
new approaches: First, a way of artificially
generating metaphor training examples and
second, a novel way to find metaphors rely-
ing only on word embeddings. The latter en-
ables application for low resource languages.
Our method is based on a transformation of
word embedding vectors into another vector
space, in which the distance between the ad-
jective word vector and the noun word vec-
tor represents the metaphoricity of the word
pair. We train this method in a zero-shot
pseudo-supervised manner by generating arti-
ficial metaphor examples and show that our
approach can be used to generate a metaphor
dataset with low annotation cost. It can then
be used to finetune the system in a few-shot
manner. In our experiments we show the capa-
bilities of the method in its unsupervised and
in its supervised version. Additionally, we test
it against a comparable unsupervised baseline
method and a supervised variation of it.
E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Explanation Generation, Question Answering

The ability to recognize analogies is fundamental to human cognition. Existing benchmarks to test word analogy does not reveal the underneath process of analogical reasoning of neural models. Holding the belief that models capable of reasoning should be right for the right reasons, we propose a first-of-its-kind Explainable Knowledge-intensive Analogical Reasoning benchmark (E-KAR). Our benchmark consists of 1,665 problems sourced from the Civil Service Exams, which require intensive background knowledge to solve. Besides, we design a free-text explanation scheme to explain how an analogy is drawn, and manually annotate E-KAR with 8,325 knowledge-rich sentences of such explanations. Empirical results suggest that this benchmark is very challenging to some state-of-the-art models for both explanation generation and analogical question answering tasks, which invites further research in this area.
Dynamic Entity Memory Network for Dialogue Relational Triplet Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Information Retrieval, Retrieval, RTE, Triplet

Relational triplet extraction (RTE) is a crucial task in information extraction and has aroused extensive attention. Although advanced studies on RTE have achieved great progress, they are still insufficient for supporting practical applications, such as dialogue system and information retrieval. In this paper, we focus on relational triplet extraction in dialogue scenarios and introduce a new task named dialogue relational triplet extraction (DRTE). Instead of being treated as static texts like sentences or documents, dialogues should be regarded as dynamic ones generated with the progress of conversations. Thus, it imposes three important challenges, including extracting triplets in real-time with incomplete dialogue context, discovering cross-utterance relational triplets, and perceiving the transition of dialogue topics. To tackle these challenges, we propose a Dynamic Entity Memory Network (DEMN). Specifically, the key of our approach is an attentional context encoder and an entity memory network. The attentional context encoder learns dialogue semantics utterance by utterance and dynamically captures salient contexts for each utterance. The entity memory network is devised to store the entities extracted from previous utterances and for cross-utterance triplets extraction. Meanwhile, it also tracks topic transitions in real-time and forgets the semantics of trivial entities. To verify the effectiveness of our model, we manually build three datasets based on KdConv benchmark. Extensive experimental results demonstrate that our model achieves state-of-the-art performances.
D$^4$: A Psychiatrist-proofread Dialogue Dataset for Depression Diagnosis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Diagnostic, Response Generation

Depression has affected large populations and become a significant threat to life expectations globally.
Automatic depression diagnosis methods have been a new research focus. 
In particular, automatic dialogue-based diagnosis systems are desired since depression diagnosis highly relies on clinical consultation.
Based on clinical diagnosis criteria, doctors initiate a conversation with ample emotional support that guides the patients to expose their symptoms. 
Such a dialog is a combination of task-oriented and chitchat, different from traditional single-purpose human-machine dialog systems. 
However, due to the social stigma associated with mental illness, the dialogue data related to the diagnosis of actual patients are rarely disclosed. The lack of data has become one of the major factors restricting the research on the consultation dialogue system of depression.
%Although there are effective methods of diagnosis and treatment for depression, more than 75\% of people in low- and middle-income countries receive no treatment. 
Based on clinical depression diagnostic criteria ICD-11 and DSM-5, we construct a Psychiatrist-proofread Dialogue Dataset for Depression Diagnosis which simulates the dialogue between the doctor and the patient during the diagnosis of depression and provides diagnosis results and symptom summary given by professional psychiatrists for each dialogue.
Finally, we finetune on state-of-art pre-training models and respectively give our dataset baselines on response generation, topic prediction, dialog summary, and severity classification of depression and suicide risk.
Using Structured Content Plans for Fine-grained Syntactic Control in Pretrained Language Model Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Abstract Meaning Representation, Language Modeling, Language Modelling, Sentence, Text Generation

Large pretrained language models offer powerful generation capabilities, but suffer from a lack of interpretability and fine-grained control. We propose an approach to fine-grained control in generating  text directly from a semantic representation, Abstract Meaning Representation (AMR) by augmenting the nodes with syntactic tags. We experiment with English-language generation of three modes of syntax relevant to the framing of a sentence - verb voice (active or passive), verb tense, and realization of human entities - and demonstrate that they can be reliably controlled. Controlling how information is framed is important for applications such as summarization, which aim to highlight salient information.
StableMoE: Stable Routing Strategy for Mixture of Experts
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Machine Translation, Mixture-of-Experts

The Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We point out that existing learning-to-route MoE methods suffer from the routing fluctuation issue, i.e., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference. The routing fluctuation tends to harm sample efficiency because the same input updates different experts but only one is finally used. In this paper, we propose StableMoE with two training stages to address the routing fluctuation problem. In the first training stage, we learn a balanced and cohesive routing strategy and distill it into a lightweight router decoupled from the backbone model. In the second training stage, we utilize the distilled router to determine the token-to-expert assignment and freeze it for a stable routing strategy. We validate our method on language modeling and multilingual machine translation. The results show that StableMoE outperforms existing MoE methods in terms of both convergence speed and performance.