pwc-archive/papers-with-abstracts huggingface dataset:
Logic Traps in Evaluating Attribution Scores
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Modern deep learning models are notoriously opaque, which has motivated the development of methods for interpreting how deep models predict.
This goal is usually approached with attribution method, which assesses the influence of features on model predictions. 
As an explanation method, the evaluation criteria of attribution methods is how accurately it reflects the actual reasoning process of the model (faithfulness). Meanwhile, since the reasoning process of deep models is inaccessible, researchers design various evaluation methods to demonstrate their arguments.
However, some crucial logic traps in these evaluation methods are ignored in most works, causing inaccurate evaluation and unfair comparison.
This paper systematically reviews existing methods for evaluating attribution scores and summarizes the logic traps in these methods.
We further conduct experiments to demonstrate the existence of each logic trap.
Through both theoretical and experimental analysis, we hope to increase attention on the inaccurate evaluation of attribution scores. 
Moreover, with this paper, we suggest stopping focusing on improving performance under unreliable evaluation systems and starting efforts on reducing the impact of proposed logic traps.
A Relation-Attentive 3D Matrix Framework for Relational Triple Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Relation, valid

Extracting relational triples from unstructured text is crucial for information extraction. Recent methods achieve considerable performance, but due to the insufficient consideration of triple global information, there is an obvious performance gap between triple (E1, R, E2) and E1/R/E2, that is, some extracted entities or relations fail to form a valid relational triple. To break this bottleneck,  we propose a relation-attentive 3D matrix  framework (RA3D)  composed of an encoder module, a fusion module, and a 3D matrix module. Instead of using a 2D table to align the subject and object, we integrate clearly encoded relation information to convert the 2D table into a 3D matrix, so that the entries of the 3D matrix can capture the interaction in subjects, objects, and relations completely. To extract relation and entity information required for the 3D matrix reasonably,  we design a transformer-decoder-based fusion module that updates the representation of relations and entities iteratively. Our model achieves state-of-the-art performance with F1 score up to 93.5\% and 94.3\% on two public datasets and delivers consistent performance gain on complex scenarios of overlapping triples.
Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Metamorphic testing has recently been used to check the safety of neural NLP models. Its main advantage is that it does not rely on a ground truth to generate test cases. However, existing studies are mostly concerned with robustness-like metamorphic relations, limiting the scope of linguistic properties they can test. We propose three new classes of metamorphic relations, which address the properties of systematicity, compositionality and transitivity. Unlike robustness, our relations are defined over multiple source inputs, thus increasing the number of test cases that we can produce by a polynomial factor. With them, we test the internal consistency of state-of-the-art NLP models, and show that they do not always behave according to their expected linguistic properties. Lastly, we introduce a novel graphical notation that efficiently summarises the inner structure of metamorphic relations.
Co-VQA : Answering by Interactive Sub Question Sequence
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA), Visual Reasoning

Most existing approaches to Visual Question Answering (VQA) answer questions directly, however, people usually decompose a complex question into a sequence of simple sub questions and finally obtain the answer to the original question after answering the sub question sequence(SQS). By simulating the process, this paper proposes a conversation-based VQA (Co-VQA) framework, which consists of three components: Questioner, Oracle, and Answerer. Questioner raises the sub questions using an extending HRED model, and Oracle answers them one-by-one.  An Adaptive Chain Visual Reasoning Model (ACVRM) for Answerer is also proposed, where the question-answer pair is used to update the visual representation sequentially. To perform supervised learning for each model, we introduce a well-designed method to build a SQS for each question on VQA 2.0 and VQA-CP v2 datasets. Experimental results show that our method achieves state-of-the-art on VQA-CP v2. Further analyses show that SQSs help build direct semantic connections between questions and images, provide question-adaptive variable-length reasoning chains, and with explicit interpretability as well as error traceability.
The Past Mistake is the Future Wisdom: Error-driven Contrastive Probability Optimization for Chinese Spell Checking
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Chinese Spell Checking

Chinese Spell Checking (CSC) aims to detect and correct Chinese spelling errors, which are mainly caused by the phonological or visual similarity. Recently, pre-trained language models (PLMs) promote the progress of CSC task. However, there exists a gap between the learned knowledge of PLMs and the goal of CSC task. PLMs focus on the semantics in text and tend to correct the erroneous characters to semantically proper or commonly used ones, but these aren't the ground-truth corrections. To address this issue, we propose an Error-driven COntrastive Probability Optimization (ECOPO) framework for CSC task. ECOPO refines the knowledge representations of PLMs, and guides the model to avoid predicting these common characters through an error-driven way. Particularly, ECOPO is model-agnostic and it can be combined with existing CSC methods to achieve better performance. Extensive experiments and detailed analyses on SIGHAN datasets demonstrate that ECOPO is simple yet effective.
SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensions
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Neural Architecture Search

Task-agnostic pre-training followed by task-specific fine-tuning is a default approach to train NLU models which need to be deployed on devices with varying resource and accuracy constraints. However, repeating pre-training and fine-tuning across tens of devices is prohibitively expensive. To address this, we propose SuperShaper, a task agnostic pre-training approach wherein we pre-train a single model which subsumes a large number of Transformer models by varying shapes, i.e., by varying the hidden dimensions across layers. 
This is enabled by a backbone network with linear bottleneck matrices around each Transformer layer which are sliced to generate differently shaped sub-networks. Despite its simple design space and efficient implementation, SuperShaper radically simplifies NAS for language models and discovers networks that effectively trade-off accuracy and model size: Discovered networks are more accurate than a range of hand-crafted and automatically searched networks on GLUE benchmarks. Further, we find two critical advantages of shape as a design variable for Neural Architecture Search (NAS): (a) networks found with these heuristics derived for good shapes, match and even improve on carefully searched networks across a range of parameter counts, and (b) the latency of networks across multiple CPUs and GPUs are insensitive to the shape and thus enable device-agnostic search.
Offensive Text Detection Across Languages and Datasets Using Rule-based and Hybrid Methods
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Text Detection, Transfer Learning

We investigate the potential of rule-based systems for the task of offensive text detection in English and German, and demonstrate their effectiveness in low-resource settings, as an alternative or addition to transfer learning across tasks and languages. Task definitions and annotation guidelines used by existing datasets show great variety, hence state-of-the-art machine learning models do not transfer well across datasets or languages. Furthermore, such systems lack explainability and pose a critical risk of unintended bias. We present simple rule systems based on semantic graphs for classifying offensive text in two languages and provide both quantiative and qualitative comparison of their performance with deep learning models on 5 datasets across multiple languages and shared tasks.
Towards Focused and Connected Document-Level Event Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Document-level Event Extraction, Event Extraction

Document-level event extraction (DEE) is indispensable when events are naturally described in the form of a document. Although previous methods have made great success on DEE, they are limited by two bottlenecks: losing focus and losing the connection. In this paper, to break through the above bottlenecks, we annotated a new dataset, named WIKIEVENT++, towards focused and connected DEE.  Besides, we propose two different models to approach this task: the extractive model and the generative model. Experimental results verify the effectiveness of our proposed methods. We further present a promising case study to explore the performance bottleneck for this task. Data and code will be released at \url{http://anonymized} to advance the research on document-level event extraction.
Leveraging Uni-Modal Self-Supervised Learning for Multimodal Audio-visual Speech Recognition
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Audio-Visual Speech Recognition, Language Modelling, Lip Reading, Self-Supervised Learning, Sentence, speech-recognition, Speech Recognition, Visual Speech Recognition

Training Transformer-based models demands a large amount of data, while obtaining parallel aligned and labelled data in multimodality is rather cost-demanding, especially for audio-visual speech recognition (AVSR). Thus it makes a lot of sense to make use of unlabelled uni-modal data. On the other side, although the effectiveness of large-scale self-supervised learning is well established in both audio and visual modalities, how to integrate those pre-trained models into a multimodal scenario remains underexplored. In this work, we successfully leverage uni-modal self-supervised learning to promote the multimodal AVSR. In particular, we first train audio and visual encoders on a large-scale uni-modal dataset, then we integrate components of both encoders into a larger multimodal framework which learns to recognize paired audio-visual data into characters through a combination of CTC and seq2seq decoding. We show that both components inherited from uni-modal self-supervised learning cooperate well, resulting in that the multimodal framework yields competitive results through fine-tuning. Our model is experimentally validated on both word-level and sentence-level AVSR tasks. Especially, even without an external language model, our proposed model raises the state-of-the-art performances on the widely accepted Lip Reading Sentences 2 (LRS2) dataset by a large margin, with a relative improvement of 30%.
NSP-NER: A Prompt-based Learner for Few-shot NER Driven by Next Sentence Prediction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Few-Shot Learning, Few-shot NER, Language Modeling, Language Modelling, Named Entity Recognition (NER), Prompt Learning, Sentence

Recently, prompt-based learning has achieved great success in few-shot learning. This paradigm does better integrate pre-training and downstream tasks and mine the knowledge inherent in the pre-train language model itself. Most research on prompt-learning has been conducted, but the application of prompt-based learning in NER has not been fully explored. The main obstacles of prompt-based learning's application to NER are that, each unit to be predicted is a token or span instead of the whole input text, and the existing prompt-based methods are not sensitive to the boundaries of tokens or spans. To address these issues, we propose a prompt-based learner for few-shot NER driven by Next Sentence Prediction (NSP), reformulating NER as a NSP task with span boundary information enhancement. We conduct experiments on three NER datasets in true few-shot learning setting, which supposes that only a small training set and a small validation set are available. Experimental results show that our method outperforms previous state-of-the-art models and yields promising performance even in such a challenging scenario.
Generative Pretraining for Paraphrase Evaluation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Representation Learning, Text Generation

We introduce ParaBLEU, a paraphrase representation learning model and evaluation metric for text generation. Unlike previous approaches, ParaBLEU learns to understand paraphrasis using generative conditioning as a pretraining objective. ParaBLEU correlates more strongly with human judgements than existing metrics, obtaining new state-of-the-art results on the 2017 WMT Metrics Shared Task. We show that our model is robust to data scarcity, exceeding previous state-of-the-art performance using only $50\%$ of the available training data and surpassing BLEU, ROUGE and METEOR with only $40$ labelled examples. Finally, we demonstrate that ParaBLEU can be used to conditionally generate novel paraphrases from a single demonstration, which we use to confirm our hypothesis that it learns abstract, generalized paraphrase representations.
Plug-Tagger: A Pluggable Sequence Labeling Framework with Pre-trained Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling

Fine-tuning the pre-trained language models (PLMs) on downstream tasks is the de-facto paradigm in NLP. Despite the superior performance on sequence labeling, the fine-tuning requires large-scale parameters and time-consuming deployment for each task, which limits its application in real-world scenarios. To alleviate these problems, we propose a pluggable sequence labeling framework, plug-tagger. By switching the task-specific plugin on the input, plug-tagger allows a frozen PLM to perform different sequence labeling tasks without redeployment. Specifically, the plugin on the input are a few continuous vectors, which manipulates the PLM without modifying its parameters, and each task only needs to store the lightweight vectors rather than a full copy of PLM.  To avoid redeployment, we propose the label word mechanism,  which reuses the language model head to prevent task-specific classifiers from modifying model structures. Experimental results on three sequence labeling tasks show that the proposed method achieves comparable performance with fine-tuning by using 0.1% task-specific parameters.  Experiments show that our method is faster than other lightweight methods under limited computational resources
Empathetic Persuasion: Reinforcing Empathy and Persuasiveness in Dialogue Systems
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Persuasiveness, Reinforcement Learning (RL)

Persuasion is an intricate process involving empathetic connection between two individuals. Plain persuasive responses may make a conversation non-engaging. 
Even the most well-intended and reasoned persuasive conversations can fall through in the absence of empathetic connection between the speaker and listener. In this paper, we propose a novel task of incorporating empathy when generating persuasive responses. We develop an empathetic persuasive dialogue system by fine-tuning a maximum likelihood Estimation (MLE)-based language model in a reinforcement learning (RL) framework. To design feedbacks for our RL-agent, we define an effective and efficient reward function considering consistency, repetitiveness, emotion and persuasion rewards to ensure consistency, non-repetitiveness, empathy and persuasiveness in the generated responses. Due to lack of emotion annotated persuasive data, we first annotate the existing PersuaionForGood dataset with emotions, then build transformer based classifiers to provide emotion based feedbacks to our RL agent. Our experimental results confirm that our proposed model increases the rate of generating persuasive responses as compared to the available state-of-the-art dialogue models while making the dialogues empathetically more engaging and retaining the language quality in responses.
Structure Representation Learning by Jointly Learning to Pool and Represent
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Graph Classification, Representation Learning, Sentence, Sentence Classification

Structure representation learning is a task to provide an overall representation for a given structure (e.g., sequential text, non-sequential graph). This representation characterizes the property of that structure. Previous methods decompose the task into an element representation learning phase and a pooling phase to aggregate element representations. Their pooling phase only considers the final representation of each element without considering the relationship between these elements that are used only to construct representations of elements. In this paper, we conjecture that classification performance suffers from the lack of relation exploitation while pooling and propose the Self-Attention Pooling to dynamically provide centrality scores for pooling based on the self-attention scores from the element representation learning.  Simply applying Self-Attention Pooling improves model performance on $3$ sentence classification tasks ({$\boldsymbol{\uparrow 2.9}$}) and $5$ graph classification tasks ({$\boldsymbol{\uparrow 2.1}$}) on average.
Cross-Document Temporal Relation Extraction with Temporal Anchoring Events
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Articles, Relation, Relation Extraction, Relation Prediction, Temporal Relation Extraction

Automatically extracting a timeline on a certain topic from multiple documents has been a challenge in natural language processing, partly due to the difficulty of collecting large amounts of training data. In this work, we collect a dataset for cross-document timeline extraction from online news that gives access to metadata such as hyperlinks and publication dates. The metadata allows us to define a set of important events while linking them to time anchors, which opens the opportunity to scale up data collection. Furthermore, with this set of linked news articles, we propose a method to enhance the inference process of temporal relation prediction, by utilizing a model to link events to a set of anchoring events that are added to the inference program. We report performance of common neural models and show that our method can boost the performance of all baseline models.
Active Relation Discovery: Towards General and Label-aware OpenRE
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Active Learning, Outlier Detection, Relation, Relation Extraction

Open Relation Extraction (OpenRE) aims to discover and label novel relations from open domains. Previous methods mainly suffer from two problems: (1) Insufficient capacity to discriminate between known and novel relations. When extending conventional test settings to a more general setting where test data might also come from seen classes, existing OpenRE approaches have a significant performance decline. (2) Secondary labeling must be performed before practical application. Existing methods cannot label human-readable and meaningful types for novel relations, which is urgently required by the downstream tasks. To address these issues, we propose the Active Relation Discovery (ARD) framework, which utilizes relational outlier detection for discriminating known and novel relations and involves active learning for labeling novel relations. Extensive experiments\footnote{The source code will be available for reproducibility.} on three real-world datasets show that ARD significantly outperforms state-of-the-art methods on both conventional and our proposed general OpenRE settings.
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Current pre-trained models applied for summarization are prone to factual inconsistencies which misrepresent the source text. Thus, evaluating the factual consistency of summaries is necessary to develop better models. However, the optimal human evaluation setup for factual consistency has not been standardized. To address this issue, we crowdsourced evaluations for factual consistency using the rating-based Likert Scale and ranking-based Best-Worst Scaling to determine the factors that affect the reliability of the human evaluation. Our crowdsourced evaluations are conducted on the summaries of CNN-Daily Mail and XSum datasets generated by four state-of-the-art models. Ranking-based Best-Worst Scaling offers a more reliable measure of summary quality across datasets, and the reliability of Likert ratings highly depends on the target dataset and the evaluation design. To improve the reliability, we extend the scale of the Likert rating to make it more flexible and we present a scoring algorithm for Best-Worst Scaling, called value learning. Our crowdsourcing guidelines and evaluation protocols will be publicly available to facilitate future research on factual consistency in summarization.
$\mathcal{Y}$-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Representation Learning

With the success of large-scale pre-trained models (PTMs), how efficiently adapting PTMs to downstream tasks has attracted tremendous attention, especially for PTMs with billions of parameters. Although some parameter-efficient tuning paradigms have been proposed to address this problem, they still require large resources to compute and store the gradients in the training phase. In this paper, we propose $\mathcal{Y}$-Tuning, an efficient yet effective paradigm to adapt frozen large-scale PTMs to specific downstream tasks. $\mathcal{Y}$-tuning learns dense representations for labels $\mathcal{Y}$ defined in a given task and aligns them to fixed feature representation. Without tuning the features of input text and model parameters, $\mathcal{Y}$-tuning is both parameter-efficient and training-efficient. Although $\mathcal{Y}$-tuning is currently still not comparable with fine-tuning in performance, it has a great advantage in saving computational cost and has the potential to further improve its performance.