pwc-archive/papers-with-abstracts huggingface dataset:
The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Machine Translation, NER, XLM-R

Large pre-trained multilingual models such as mBERT and XLM-R enabled effective cross-lingual zero-shot transfer in many NLP tasks. A cross-lingual adjustment of these models using a small parallel corpus can further improve results. This is a more data efficient method compared to training a machine-translation system or a multi-lingual model from scratch using only parallel data. In this study, we experiment with zero-shot transfer of English models to four typologically different languages (Spanish, Russian, Vietnamese, and Hindi) and three NLP tasks (QA, NLI, and NER). We carry out a cross-lingual adjustment of an off-the-shelf mBERT model. We show that this adjustment makes embeddings of semantically similar words from different languages closer to each other, while keeping unrelated words apart. In contrast, fine-tuning of mBERT on English data (for a specific task such as NER) draws embeddings of both related and unrelated words closer to each other. The cross-lingual adjustment of mBERT improves NLI in four languages and NER in two languages. However, in the case of QA performance never improves and sometimes degrades. In that, the increase in the amount of parallel data is most beneficial for NLI, whereas QA performance peaks at roughly 5K parallel sentences and further decreases as the number of parallel sentences increases.
A Cueing Strategy with Prompt Tuning for Relation Extraction
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Relation, Relation Extraction

Prompt tuning shows great potential to support relation extraction because it is effective to take full use of rich knowledge in pretrained language models (PLMs). However, current prompt tuning models are directly implemented on a raw input. It is weak to encode semantic dependencies of a relation instance. In this paper, we designed a cueing strategy which implants task specific cues into the input. It enables PLMs to learn task specific contextual features and semantic dependencies in a relation instance. Experiments on ReTACRED corpus and ACE 2005 corpus show state-of-the-art performance in terms of F1-score.
LoPE: Learnable Sinusoidal Positional Encoding for Improving Document Transformer Model
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: document understanding

Positional encoding plays a key role in Transformer-based architecture, which is to indicate and embed token sequential order information. Understanding documents with unreliable reading order information is a real challenge for document Transformer model. This paper proposes a new and generic positional encoding method, learnable sinusoidal positional encoding (LoPE), by combining sinusoidal positional encoding function and a learnable feed-forward network. We apply LoPE to document Transformer model and pretrain the model on document datasets. Then we finetune and evaluate the model performance on document understanding tasks in form and receipt domains. Experimental results not only show our proposed method outperforms other baselines and state-of-the-arts, but also demonstrate its robustness and stability on handling noisy data with incorrect order information.
Understand before Answer: Improve Temporal Reading Comprehension via Precise Question Understanding
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Reading Comprehension, Representation Learning

This work studies temporal reading comprehension (TRC), which reads a free-text passage and answers temporal ordering questions. Precise question understanding is critical for temporal reading comprehension. For example, the question "What happened before the victory" and "What happened after the victory" share almost all words except one, while their answers are totally different. Moreover, even if two questions query about similar temporal relations, different varieties might also lead to various answers. For example, although both the question "What usually happened during the press release?" and "What might happen during the press release" query events which happen after "the press release", they convey divergent semantics.To this end, we propose a novel reading comprehension approach with precise question understanding. Specifically, a temporal ordering question is embedded into two vectors to capture the referred event and the temporal relation. Then we evaluate the temporal relation between candidate events and the referred event based on that. Such fine-grained representations offer two benefits. First, it enables a better understanding of the question by focusing on different elements of a question. Second, it provides good interpretability when evaluating temporal relations. Furthermore, we also harness an auxiliary contrastive loss for representation learning of temporal relations, which aims to distinguish relations with subtle but critical changes. The proposed approach outperforms strong baselines and achieves state-of-the-art performance on the TORQUE dataset. It also increases the accuracy of four pre-trained language models (BERT base, BERT large, RoBERTa base, and RoBETRa large), demonstrating its generic effectiveness on divergent models.
On Synthetic Data for Back Translation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: de-en, NMT, Translation

Back translation (BT) is one of the most significant technologies in NMT research fields. Existing attempts on BT share a common characteristic: they employ either beam search or random sampling to generate synthetic data with a backward model but seldom work studies the role of synthetic data in the performance of BT. This motivates us to ask a fundamental question: what kind of synthetic data contributes to BT performance?Through both theoretical and empirical studies, we identify two key factors on synthetic data controlling the back-translation NMT performance, which are quality and importance. Furthermore, based on our findings, we propose a simple yet effective method to generate synthetic data to better trade off both factors so as to yield the better performance for BT. We run extensive experiments on WMT14 DE-EN, EN-DE, and RU-EN benchmark tasks. By employing our proposed method to generate synthetic data, our BT model significantly outperforms the standard BT baselines (i.e., beam and sampling based methods for data generation), which proves the effectiveness of our proposed methods.
LongChecker: Improving scientific claim verification by modeling full-abstract context
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Claim Verification, Domain Adaptation, Fact Checking, Sentence

The spread of scientific mis- and dis-information has motivated the development of datasets and models for the task of scientific claim verification. We address two modeling challenges associated with this task. First, existing claim verification systems make predictions by extracting an evidentiary sentence (or sentences) from a larger context, and then predicting whether this sentence supports or refutes the claim in question. This can be problematic, since the meaning of the selected sentence may change when interpreted outside its original context. Second, given the difficulty of collecting high-quality fact-checking annotations in expert domains, there is an unaddressed need for methods to facilitate zero / few-shot domain adaptation. Motivated by these challenges, we develop LongChecker. Given a claim and evidence-containing abstract, LongChecker predicts a fact-checking label and identifies evidentiary sentences in a multi-task fashion based on a shared encoding of all available context. This approach enables LongChecker to perform domain adaptation by leveraging weakly-supervised in-domain data. We show that LongChecker achieves state-of-the-art performance on three datasets, and conduct analysis to confirm that its strong performance is due to its ability to model full-abstract context.
Exploring Cross-Lingual Guidance in Abstractive Summarization
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Abstractive Text Summarization

Cross-lingual guidance (CLG) as an augmentation method is often applied in cross-lingual summarization (CLS) to improve its performance. In this paper, we empirically study how cross-lingual information of different quality benefits the encoding and decoding procedures for both cross-lingual and mono-lingual abstractive summarization. We specially propose a summarization model DualSum which can utilize CLG in both encoding and decoding, and construct a dataset BiRead with high-quality parallel bilingual document-summary pairs. The empirical experiments will show how CLS and MLS are influenced by CLG.
Few-Shot Semantic Parsing with Language Models Trained On Code
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Semantic Parsing

Large language models can perform semantic parsing with little training data, when prompted with in-context examples. It has been shown that this can be improved by formulating the problem as paraphrasing into canonical utterances, which casts the underlying meaning representation into a controlled natural language-like representation. Intuitively, such models can more easily output canonical utterances as they are closer to the natural language used for pre-training. More recently, models also pre-trained on code, like OpenAI Codex, have risen in prominence. Since semantic parsing requires translating natural language into code, such models may prove more adept at it. In this paper, we test this hypothesis and find that Codex performs better at semantic parsing than equivalent GPT-3 models. We find that unlike GPT-3, Codex performs similarly when targeting meaning representations directly, perhaps because meaning representations used in semantic parsing are structured similar to code.
Building Sequence-to-Sequence Document Revision Models from Matched and Multiple Partially-Matched Datasets
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Sentence

This paper defines the document revision task and proposes a novel modeling method that can utilize not only a matched dataset but also multiple partially-matched datasets. In the document revision task, we aim to simultaneously consider multiple perspectives for writing supports. To this end, it is important not only to correct grammatical errors but also to improve readability and perspicuity, through means such as conjunction insertion and sentence reordering. However, it is difficult to prepare enough the matched dataset for the document revision task since this task has to consider multiple perspectives simultaneously. To mitigate this problem, our idea is to utilize not only a limited matched dataset but also various partially-matched datasets that handles individual perspectives, e.g., correcting grammatical errors or inserting conjunctions. Since suitable partially-matched datasets have either been published or can easily be made, we expect to prepare a large amount of these partially-matched datasets. To effectively utilize these multiple datasets, our proposed modeling method incorporates ``on-off'' switches into sequence-to-sequence modeling to distinguish the matched datasets and individual partially-matched datasets. Experiments using our created document revision datasets demonstrate the effectiveness of the proposed method.
Efficient Weighted Deduction Systems for Earley’s Algorithm
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Sentence

The parsing algorithm of Earley (1970), as presented, has a runtime complexity of $\mathcal{O}(N^3\lvert\mathcal{G}\rvert \lvert\mathcal{R}\rvert)$ where $N$ is the length of the sentence, $\lvert\mathcal{G}\rvert $ is the size of the grammar, and $\lvert\mathcal{R}\rvert$ is the number of productions in the grammar. This is unworkable for the large grammars that arise in natural language processing. Fortunately, the dynamic programming algorithm can be improved to run in time $\mathcal{O}(N^3\lvert\mathcal{G}\rvert)$, matching the complexity of running CKY on a binarized version of $\mathcal{G}$. Some of the necessary speed-ups have been presented in part or in full in various parts of the literature. However, there has been no unified, formal treatment that is written as a deduction system or covers the weighted case. We present such a treatment in terms of five proof rules that can be used in weighted deduction, which refine Earley's \predict, \scan and \complete actions. We also provide a generalization of Earley's algorithm that uses a finite-state automaton to represent the grammar, and whose runtime is proportional to the size of the automaton (and the usual $\mathcal{O}(N^3)$ term), or more precisely the size of the portion of the automaton that is reached while parsing the input sentence.  Further speed-ups can then be achieved by minimizing the automaton so that similar productions share transitions.
A Benchmark for Text Quantification Learning Under Real-World Temporal Distribution Shift
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Sentiment Analysis

Text quantification is a supervised learning task estimating the relative frequency of each class for a collection of uncategorized text documents. Quantification learning has an increasing number of applications in practice and presents unique challenges that are often overlooked in classification problems, such as dealing with distribution shift. Many studies on quantification use artificially re-sampled test sets to evaluate models under varying target label distributions. Despite being a convenient solution, label-based biased sampling changes the underlying test data distribution and makes it hard to rely on the results to deploy models in practice. This paper introduces a text quantification benchmark consisting of 8 datasets across sentiment analysis, document categorization, and toxicity classification. We compare popular quantification baselines on the benchmark and show that there is no model consistently outperforming others. Therefore, we believe the benchmark should enable new community research to tackle text quantification under temporal distribution shift and develop reliable models in real-world applications.
A Meta-transfer Learning framework for Visually Grounded Compositional Concept Learning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Meta-Learning, Transfer Learning

Humans acquire language in a compositional and grounded manner.They can describe their perceptual world using novel compositions from already learnt elementary concepts. However, recent research shows that modern neural networks lack such compositional generalization ability. To address this challenge, in this paper, we propose \textit{MetaVL}, a meta-transfer learning framework to train transformer-based vision-and-language (V\&L) models using optimization-based meta-learning method and episodic training.We carefully created two datasets based on MSCOCO and Flicker30K to specifically target novel compositional concept learning. Our empirical results have shown that \textit{MetaVL} outperforms baseline models in both datasets. Moreover, \textit{MetaVL} has demonstrated higher sample efficiency compared to supervised learning, especially under the few-shot setting.
Modeling Tension in Stories via Commonsense Reasoning and Emotional Word Embeddings
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Word Embeddings

Dramatic tension is crucial for generating interesting stories. This paper aims to model dramatic tension from a story text using neural commonsense-reasoning language models and emotional word embeddings. We also propose a method of converting a categorical emotion word into a numerical value. The evaluation results using human-annotated stories demonstrate that our proposed method is promising in predicting tension development in a story.
CrowdChecked: Detecting Previously Fact-Checked Claims in Social Media
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Fact Checking

While there has been substantial progress in developing systems to automate the process of fact-checking, such systems still lack credibility in the eyes of the users, and thus human fact-checkers remain the main drivers of the process. In view of that, recently, a middle-ground approach has emerged: to do automatic fact-checking by verifying whether the input claim has been previously fact-checked by professional fact-checkers, and to return back an article that explains the verdict on the claim. This is a sensible approach as people trust manual fact-checking, and as many claims are repeated multiple times online.Yet, a major issue when building such kinds of systems is the small number of known input--verified claim pairs available for training. Here, we aim to bridge this gap by making use of crowd fact-checking, i.e., mining claims in social media for which users have responded with a link to a fact-checking article. In particular, we mine a large-scale collection of 330,000 tweets paired with a corresponding fact-checking article. We further propose a new model to learn from this noisy data based on modified self-adaptive training, in a distant supervision scenario. Our experiments on a standard test set show improvements over the state of the art by two points absolute.
δ-SAM: Sharpness-Aware Minimization with Dynamic Reweighting
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Deep neural networks are often overparameterized and may not easily achieve model generalization. Adversarial training has shown effectiveness in improving generalization by regularizing the change of loss on top of adversarially chosen perturbations. The recently proposed sharpness-aware minimization (SAM) algorithm conducts adversarial weight perturbation, encouraging the model to converge to a flat minima. Unfortunately, due to increased computational cost, adversarial weight perturbation can only be efficiently estimated per-batch instead of per-instance by SAM, leading to degraded performance. In this paper, we tackle this efficiency bottleneck and propose the first instance-based weight perturbation method: sharpness-aware minimization with dynamic reweighting (δ-SAM). δ-SAM dynamically reweights perturbation within each batch by estimated guardedness (i.e. unguarded instances are up-weighted), serving as a better approximation to per-instance perturbation. Experiments on various tasks demonstrate the effectiveness of δ-SAM.
A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Machine Translation, Translation

It is difficult for non-autoregressive translation (NAT) models to capture the multi-modal distribution of target translations due to their conditional independence assumption, which is known as the ``multi-modality problem'', including the lexical multi-modality and the syntactic  multi-modality. While the first one has been well studied, the syntactic multi-modality brings severe challenge to the standard cross entropy (XE) loss in NAT and is under studied. In this paper, we conduct a systematic study on the syntactic multi-modality problem. Specifically, we decompose it into short- and long-range syntactic multi-modalities and evaluate several recent NAT algorithms with advanced loss functions on both carefully designed synthesized datasets and real datasets. We find that the Connectionist Temporal Classification (CTC) loss and the Order-Agnostic Cross Entropy (OAXE) loss can better handle short- and long-range syntactic multi-modalities respectively. Furthermore, we take the best of both and design a new loss function to better handle the complicated syntactic multi-modality in real-world datasets. To facilitate practical usage, we provide a guide to use different loss functions for different kinds of syntactic multi-modality.
AutoAttention: Automatic Attention Head Selection Through Differentiable Pruning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Multi-head attention is considered as a driving force and key component behind the state-of-art transformer models. However, recent research reveals that there are many redundant heads with duplicated patterns in each layer. In this work, we propose an automatic pruning strategy using differentiable binary gates to remove redundant heads. We relax the binary head pruning problem into a differentiable optimization by employing Straight Through Estimators (STEs), in which the model weights and head-sparse model structure can be jointly learned through back-propagation. In this way, attention heads can be pruned efficiently and effectively. Experimental results on the General Language Understanding Evaluation  (GLUE)  benchmark are provided using BERT model. We could reduce more than 57% heads on average with zero or minor accuracy drop on all nine tasks and even achieve better results than state-of-the-arts (e.g., Random, HISP, $L0$ Norm, SMP, etc). Furthermore, our proposed method can prune more than 79% heads with only 0.82% accuracy degradation on average. We further illustrate the pruning procedure and parameters change through the head attention visualization and show how the trainable gate parameters determine the head mask and the final attention map.
A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Cross-Lingual Transfer

We show that the choice of pretraining languages affects downstream cross-lingual transfer for BERT based models. We inspect zero-shot performance under balanced data conditions to mitigate data size confounds, classifying pretrain languages that increase downstream performance into donors, and languages that are most improved in zero-shot performance as recipients. We develop a method of quadratic time complexity in the number of pretraining  languages to estimate these inter-language relations, instead of an exponential exhaustive computation of all possible combinations. We find that our method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks.Our findings can inform developers of future large scale multilingual language models in choosing better pretraining configurations.