pwc-archive/papers-with-abstracts huggingface dataset:
Cooperative Self-training of Machine Reading Comprehension
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Extractive Question-Answering, Machine Reading Comprehension, Question Answering, Question Generation, Question-Generation, Reading Comprehension, Transfer Learning, Word Embeddings

Pretrained language models have significantly improved the performance of downstream language understanding tasks, including extractive question answering, by providing high-quality contextualized word embeddings. However, training question answering models still requires large amounts of annotated data for specific domains. In this work, we propose a cooperative self-training framework, RGX, for automatically generating more non-trivial question-answer pairs to improve model performance. RGX is built upon a masked answer extraction task with an interactive learning environment containing an answer entity \textbf{R}ecognizer, a question \textbf{G}enerator, and an answer e\textbf{X}tractor. Given a passage with a masked entity, the generator generates a question around the entity, and the extractor is trained to extract the masked entity with the generated question and raw texts. The framework allows the training of question generation and answering models on any text corpora without annotation. We further leverage a reinforcement learning technique to reward generating high-quality questions and to improve the answer extraction model's performance. Experiment results show that RGX outperforms the state-of-the-art (SOTA) pretrained language models and transfer learning approaches on standard question-answering benchmarks, and yields the new SOTA performance under given model size and transfer learning settings.
Applying SoftTriple Loss for Supervised Language Model Fine Tuning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Language Modeling, Language Modelling

We introduce a new loss function TripleEntropy to improve classification performance for fine-tuninggeneral knowledge pre-trained language models based on cross-entropy and SoftTriple loss. Thisloss function can improve the robust RoBERTa baseline model fine-tuned with cross-entropy loss byabout (0.02% - 2.29%). Thorough tests on popular datasets indicate a steady gain. The fewer samplesin the training dataset, the higher gain – thus, for small-sized dataset it is 0.78%, for medium-sized –0.86% for large – 0.20% and for extra-large 0.04%.
A Dataset for Cross-Domain Reasoning via Template Filling
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Decoder

While several benchmarks exist for reasoning tasks, reasoning across domains is an under-explored area in NLP. Towards this, we present a dataset and a prompt-template-filling approach to enable sequence to sequence models to perform cross-domain reasoning. We also present a case-study with commonsense and health and well-being domains, where we study how prompt-template-filling enables pretrained sequence to sequence models across domains. Our experiments across several pretrained encoder-decoder models show that cross-domain reasoning is challenging for current models. We also show an in-depth error analysis and avenues for future research for reasoning across domains
Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Machine Translation, Translation

Human-translated text displays distinct features from naturally written text in the same language. This phenomena, known as translationese, has been argued to confound the machine translation (MT) evaluation. Yet, we find that existing work on translationese neglects some important factors and the conclusions are mostly correlational but not causal. In this work, we collect CausalMT, a dataset where the MT training data are also labeled with the human translation directions. We inspect two critical factors, the train-test alignment (whether the human translation directions in the training and test sets are aligned), and data-model alignment (whether the model learns in the same direction as the human translation direction in the dataset). We show that these two factors have a large causal effect on the MT performance, in addition to the test-model misalignment highlighted by existing work on the impact of translationese in the test set. In light of our findings, we provide a set of suggestions for MT training and evaluation.
Re2G: Retrieve, Rerank, Generate
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Fact Checking, GPU, Knowledge Distillation, Question Answering, RAG, Reranking, Retrieval, slot-filling, Slot Filling, Zero-shot Slot Filling

As demonstrated by GPT-3 and T5, transformers grow in capability as parameter spaces become larger and larger.  However, for tasks that require a large amount of knowledge, non-parametric memory allows models to grow dramatically with a sub-linear increase in computational cost and GPU memory requirements.  Recent models such as RAG and REALM have introduced retrieval into conditional generation.  These models incorporate neural initial retrieval from a corpus of passages.  We build on this line of research, proposing Re2G, which combines both neural initial retrieval and reranking into a BART-based sequence-to-sequence generation.  Our reranking approach also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of BM25 and neural initial retrieval.  To train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker and generation using only ground truth on the target sequence output.  We find large gains in four diverse tasks: zero-shot slot filling, question answering, fact checking and dialog, with relative gains of 9% to 34% over the previous state-of-the-art on the KILT leaderboard.  We make our code available as open source.
Cross-document Misinformation Detection based on Event Graph Reasoning
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Articles, coreference-resolution, Coreference Resolution, Event Coreference Resolution, Graph Neural Network, Misinformation

For emerging events, human readers are often exposed to both real news and fake news. Multiple news articles may contain complementary or contradictory information that readers can leverage to help detect fake news. Inspired by this process, we propose a novel task of cross-document misinformation detection. Given a cluster of topically related news documents, we aim to detect misinformation at both document level and a more fine-grained level, event level. Due to the lack of data, we generate fake news by manipulating real news, and construct 3 new datasets with 422, 276, and 1,413 clusters of topically related documents, respectively. We further propose a graph-based detector that constructs a cross-document knowledge graph using cross-document event coreference resolution and employs a heterogeneous graph neural network to conduct detection at two levels. We then feed the event-level detection results into the document-level detector. Experimental results show that our proposed method significantly outperforms existing methods by up to 7 F1 points on this new task.
Simple Local Attentions Remain Competitive for Long-Context Tasks
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Many NLP tasks require processing long contexts beyond the length limit of pretrained models. In order to scale these models to longer text sequences, many efficient long-range attention variants have been proposed. Despite the abundance of research along this direction, it is still difficult to gauge the relative effectiveness of these models in practical use cases, e.g., if we apply these models following the pretrain-and-finetune paradigm. In this work, we aim to conduct a thorough analysis of these emerging models with large-scale and controlled experiments. For each attention variant, we pretrain large-size models using the same long-doc corpus and then finetune these models for real-world long-context tasks. Our findings reveal pitfalls of an existing widely-used long-range benchmark and show none of the tested efficient attentions can beat a simple local window attention under standard pretraining paradigms. Further analysis on local attention variants suggests that even the commonly used attention-window overlap is not necessary to achieve good downstream results --- using disjoint local attentions, we are able to build a simpler and more efficient long-doc QA model that matches the performance of Longformer with half of its pretraining compute.
Unsupervised Sentence Simplification via Dependency Parsing
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Dependency Parsing, Sentence, Sentence Embeddings, Text Simplification

Text simplification is the task of rewriting a text so that it is readable and easily understood. In this paper, we propose a simple yet novel unsupervised sentence simplification system that harnesses parsing structures together with sentence embeddings  to produce linguistically effective simplifications. This means our model is capable of introducing substantial modifications to simplify a sentence while maintaining its original semantics and adequate fluency. We establish the unsupervised state-of-the-art at 39.13 SARI on TurkCorpus set and perform competitively against supervised baselines on various quality metrics. Furthermore, we demonstrate our framework's extensibility to other languages via a proof-of-concept on Vietnamese data. Code for reproduction is anonymously published at https://anonymous.4open.science/r/USDP-744B.
Learning Cross-Lingual IR from an English Retriever
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Cross-Lingual Information Retrieval, Information Retrieval, Knowledge Distillation, Language Modeling, Language Modelling, Retrieval, Translation

We present a new cross-lingual information retrieval (CLIR) system trained using multi-stage knowledge distillation (KD). The teacher relies on a highly effective but expensive two-stage process consisting of query translation and monolingual IR, while the student executes a single CLIR step. We teach the student powerful multilingual encoding as well as CLIR by optimizing two corresponding KD objectives. Learning useful non-English representations from an English-only retriever is accomplished through a cross-lingual token alignment algorithm that relies on the representation capabilities of the underlying multilingual language model. In both in-domain and zero-shot evaluation, the proposed method demonstrates far superior accuracy over direct fine-tuning with labeled CLIR data. One of our systems is also the current best single-model system on the XOR-TyDi leaderboard.
MedDistant19: A Challenging Benchmark for Distantly Supervised Biomedical Relation Extraction
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Relation, Relation Extraction

Relation Extraction in the biomedical domain is a challenging task due to the lack of labeled data and the long-tail distribution of the entity mentions. Recent works propose distant supervision as a way to tackle the scarcity of annotated data by automatically pairing knowledge graph relationships with raw textual data. In several benchmarks, Distantly Supervised Biomedical Relation Extraction (Bio-DSRE) models can produce very accurate results. However, given the challenging nature of the task, we set out to investigate the validity of such impressive results. We probed the datasets used by \citet{amin2020data} and \citet{hogan2021abstractified} and found a significant overlap between training and evaluation relationships that, once resolved, reduced the accuracy of the models by up to 71\%. Furthermore, we noticed several inconsistencies along the data construction process, such as the creation of negative samples and improper handling of redundant relationships. To mitigate these issues we present \meddistant, a new benchmark dataset obtained by aligning the MEDLINE abstracts with the widely used SNOMED-Clinical Terms (SNOMED-CT) knowledge base. We experimented with several state-of-the-art models following our methodology, showing that there is still plenty of room for improvement for the task. We release our code and data for reproducibility.
Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Dialogue Generation, Language Modeling, Language Modelling

Knowledge-grounded dialogue systems are challenging to build due to the lack of training data and heterogeneous knowledge sources. Existing systems perform poorly on unseen topics due to limited topics covered in the training data. In addition, it is challenging to generalize to the domains that require different types of knowledge sources. To address the above challenges, we present PLUG, a language model that homogenizes different knowledge sources to a unified knowledge representation for knowledge-grounded dialogue generation tasks. We first retrieve relevant information from heterogeneous knowledge sources (e.g., wiki, dictionary, or knowledge graph); Then the retrieved knowledge is transformed into text and concatenated with dialogue history to feed into the language model for generating responses. PLUG is pre-trained on a large-scale knowledge-grounded dialogue corpus. The empirical evaluation on two benchmarks shows that PLUG generalizes well across different knowledge-grounded dialogue tasks. It achieves comparable performance with state-of-the-art methods in the fully-supervised setting and significantly outperforms other approaches in zero-shot and few-shot settings.
On Leakage in Some Popular Benchmarks on Graphs
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

A number of benchmarks are based on graphs.  Edges are typically split into train, validation and test splits,  using a random partition.  Leakage has been discovered in a number of popular benchmarks; FB15k has been replaced by FB15k-237 and WN18 has been replaced by WN18RR, though leakage has been reported even after these corrections. This paper will report a new type of leakage, $A$-leakage, on benchmarks for synonym-antonym classification.  $A$-leakage infers labels for pairs of words in the test split, $w_i , w_j$, by exploiting labels on paths from $w_i$ to $w_j$ in the training split.  We conclude that it is safer to partition vertices, $V$, than edges, $E$.
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Fact Verification, Memorization, Multi-Task Learning, Open-Domain Question Answering, Open-Ended Question Answering, Question Answering, Retrieval, Retrieval-augmented Generation

Retrieval-augmented generation models have shown state-of-the-art performance across many knowledge-intensive NLP tasks such as open question answering and fact verification. These models are trained to generate a final output given retrieved passages that can be irrelevant to an input query, leading to learning spurious cues or memorization. This work introduces a method to incorporate the evidentiality of passages---whether a passage contains correct evidence to support the output---into training the generator.  We introduce a multi-task learning framework to jointly generate the final output and predict the {\it evidentiality} of each passage. We introduce a new task-agnostic method for obtaining high-quality silver evidentiality labels, addressing the issues of gold evidentiality labels being unavailable in most domains. Our experiments on five datasets across three knowledge-intensive tasks of open-domain question answering, fact verification, and knowledge-enhanced dialogue show that our new evidentiality-guided generator significantly outperforms its direct counterpart on all of them, and advances the state of the art on three of them. Our analysis shows that multi-task learning and silver evidentiality mining played key roles.
Towards Personalized Intelligence at Scale
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Personalized Intelligence (PI) is the problem of providing customized AI experiences tailored to each individual user. In many applications, PI is preferred or even required. Existing personalization approaches involve fine-tuning pre-trained models to create new customized models. However, these approaches require a significant amount of computation to train, scaling with model size and the number of users, inhibiting PI to be realized widely. In this work, we introduce a novel model architecture and training/inference framework to enable Personalized Intelligence at scale. We achieve this by attaching a Personalization Head (PH) and freezing the base pre-trained LM. Since only the parameters in PH are updated during training, this results in a model much smaller than the conventional fine-tuned LM when scaled across users. We evaluate on academia and industry-focused datasets and show that this is much more scalable than traditional fine-tuning and outperforms zeroshot baseline in F1 score. We identify key factors required for effective PH design and training.
End-to-end Dense Video Captioning as Sequence Generation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Dense Video Captioning, Descriptive, Video Captioning

Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event. Previous approaches usually follow a two-stage generative process, which first proposes a segment for each event, then renders a caption for each identified segment. Recent advances in large-scale sequence generation pretraining have seen great success in unifying task formulation for a great variety of tasks, but so far, more complex tasks such as dense video captioning are not able to fully utilize this powerful paradigm.In this work, we show how to model the two subtasks of dense video captioning jointly as one sequence generation task, and simultaneously predict the events and the corresponding descriptions. Experiments on YouCook2 and ViTT show encouraging results and indicate the feasibility of training complex tasks such as end-to-end dense video captioning integrated into large-scale pretrained models.
PROMPT WAYWARDNESS: The Curious Case of Discretized Interpretation of Continuous Prompts
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning. Motivated by these promising results, we investigate the feasibility of extracting a discrete (textual) interpretation of continuous prompts that is faithful to the problem they solve. In practice, we observe a "wayward" behavior between the task solved by continuous prompts and their nearest neighbor discrete projections: We can find continuous prompts that solve a task while being projected to an arbitrary text (e.g., definition of a different or even a contradictory task), while being within a very small (2%) margin of the best continuous prompt of the same size for the task. We provide intuitions behind this odd and surprising behavior, as well as extensive empirical analyses quantifying the effect of various parameters. For instance, for larger model sizes we observe higher waywardness, i.e, we can find prompts that more closely map to any arbitrary text with a smaller drop in accuracy. These findings have important implications relating to the difficulty of faithfully interpreting continuous prompts and their generalization across models and tasks, providing guidance for future progress in prompting language models.
Semantically Informed Slang Interpretation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Machine Translation, Translation

Slang is a predominant form of informal language making flexible and extended use of words that is notoriously hard for natural language processing systems to interpret. Existing approaches to slang interpretation tend to rely on context but ignore semantic extensions common in slang word usage. We propose a semantically informed slang interpretation (SSI) framework that considers jointly the contextual and semantic appropriateness of a candidate interpretation for a query slang.  We perform rigorous evaluation on two large-scale online slang dictionaries and show that our approach not only achieves state-of-the-art accuracy for slang interpretation in English, but also does so in zero-shot and few-shot scenarios where training data is sparse. Furthermore, we show how the same framework can be applied to enhancing machine translation of slang from English to other languages. Our work creates opportunities for the automated interpretation and translation of informal language.
VEE-BERT: Accelerating BERT Inference for Named Entity Recognition via Vote Early Exiting
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Dialogue Understanding, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

Named entity recognition (NER) is of great importance for a wide range of tasks, such as medical health record understanding, document analysis, dialogue understanding. BERT and its variants are the most performing models for NER. However, these models are notorious for being large and slow during inference. Thus their usage in the industry is limited. Pilot experiments exhibit that in the NER task, BERT suffers from the severe over-thinking problem, thus motivating BERT to exit early at intermediate layers. Thus, in this work, we propose a novel method, \underline{V}ote \underline{E}arly \underline{E}xiting BERT (VEE-BERT), for improving the early exiting of BERT on NER tasks. To be able to deal with complex NER tasks with nested entities, we adopt the Biaffine NER model \citep{yu-etal-2020-named}, which converts a sequence labeling task to the table filling task. VEE-BERT makeS early exiting decisions by comparing the predictions of the current layer with those of the previous layers. Experiments on six benchmark NER tasks demonstrate that our method is effective in accelerating the BERT Biaffine model's inference speed with less performance loss compared to the baseline early exiting method.
A Simple But Powerful Graph Encoder for Temporal Knowledge Graph Completion
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Knowledge Graph Completion, Knowledge Graphs, Temporal Knowledge Graph Completion

While knowledge graphs contain rich semantic knowledge about various entities and the relational information among them, temporal knowledge graphs (TKGs) describe and model the interactions of the entities over time. In this context, automatic temporal knowledge graph completion (TKGC) has gained great interest. Recent TKGC methods aim to integrate advanced deep learning techniques, e.g., Transformers, to boost model performance. However, we find that instead of adopting various kinds of complex modules, it is more beneficial to capture more extensive temporal information. In this paper, we propose a simple but powerful graph encoder for TKGC, namely, TARGCN. TARGCN is parameter-efficient, and it extensively utilizes the information from the whole temporal context. We perform experiments on three benchmark datasets. Our model can achieve a more than 46% relative improvement on the GDELT dataset compared with state-of-the-art models. Meanwhile, it outperforms the strongest baseline on the ICEWS05-15 dataset with around 18% fewer parameters.