pwc-archive/papers-with-abstracts huggingface dataset:
SegMix: A Simple Structure-Aware Data Augmentation Method
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Relation Extraction, Structured Prediction

Many Natural Language Processing tasks involve predicting structures, such as Syntax Parsing and Relation Extraction (RE). One central challenge in supervised structured prediction is the lack of high-quality annotated data. The recently proposed interpolation-based data augmentation (DA) algorithms (i.e. mixup) augment the training set via making convex interpolation between training data points. However, current algorithms (e.g. SeqMix, LADA) that apply mixup to language structured prediction tasks are not aware of the syntactic or output structures of the tasks, making their performance unstable and requiring additional heuristic constraints. Furthermore, SeqMix-like algorithms expect a linear encoding scheme of the output structure, such as BIO-Scheme for Named Entity Recognition (NER), restricting its applicability.

To this end, we propose SegMix, a simple framework of interpolation-based algorithms that can adapt to both the syntactic and output structures, making it robust to hyper-parameters and applicable to different tasks. We empirically show that SegMix consistently improves performance over several strong baseline models on two structured prediction tasks (NER and RE). SegMix is a flexible framework that unifies existing rule-based language DA methods, creating interesting mixtures of DA techniques. Furthermore, the method is easy to implement and adds negligible overhead to training and inference.
Composing Structure-Aware Batches for Pairwise Sentence Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Language Modeling, Language Modelling, Sentence, Sentence Classification

Identifying the relation between two sentences requires datasets with pairwise annotations. In many cases, these datasets contain instances that are annotated multiple times as part of different pairs. They constitute a structure that contains additional helpful information about the inter-relatedness of the text instances based on the annotations. This paper investigates how this kind of structural dataset information can be exploited during training.
We propose three batch composition strategies to incorporate such information and measure their performance over 14 heterogeneous pairwise sentence classification tasks. Our results show statistically significant improvements (up to 3.9%) - independent of the pre-trained language model - for most tasks compared to baselines that follow a standard training procedure. Further, we see that even this baseline procedure can profit from having such structural information in a low-resource setting.
Structural Characterization for Dialogue Disentanglement
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Disentanglement, Reading Comprehension

Tangled multi-party dialogue contexts lead to challenges for dialogue reading comprehension, where multiple dialogue threads flow simultaneously within a common dialogue history, increasing difficulties in understanding a dialogue history for both human and machine. Previous studies mainly focus on utterance encoding methods with carefully designed features and pay inadequate attention to characteristic features of the structure of dialogues.  We specially take dialogue structure factors into account and design a novel model for dialogue disentangling. Based on the fact that dialogues are constructed on successive participation of speakers and interactions between users of interest, we extract clues of speaker property and reference of users to model structural information of dialogues. The proposed method achieves new state-of-the-art on benchmark dataset and contributes to dialogue-related comprehension.
Co-training an Unsupervised Constituency Parser with Weak Supervision
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Inductive Bias, Sentence

We introduce a method for unsupervised parsing that relies on bootstrapping classifiers to identify if a node dominates a specific span in a sentence. There are two types of classifiers, an inside classifier that acts on a span, and an outside classifier that acts on everything outside of a given span. Through self-training and co-training with the two classifiers, we show that the interplay between them helps improve the accuracy of both, and as a result, effectively parse. A seed bootstrapping technique prepares the data to train these classifiers. Our analyses further validate that such an approach in conjunction with weak supervision using prior branching knowledge of a known language (left/right-branching) and minimal heuristics injects strong inductive bias into the parser, achieving 63.1 F$_1$ on the English (PTB) test set. In addition, we show the effectiveness of our architecture by evaluating on treebanks for Chinese (CTB) and Japanese (KTB) and achieve new state-of-the-art results.\footnote{For code or data, please contact the authors.}
Textomics: A Dataset for Genomics Data Summary Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Summarizing biomedical discovery from genomics data using natural languages is an essential step in biomedical research but is mostly done manually. Here, we introduce Textomics, a novel dataset of genomics data description, which contains 22,273 pairs of genomics data matrix and its summary. Each summary is written by the researchers who generated the data and associated with a scientific paper. Based on this dataset, we study two novel tasks: generating textual summary from genomics data matrix and vice versa. Inspired by the successful applications of $k$ nearest neighbors in modeling genomics data, We propose a $k$NN-Vec2Text model to address these tasks and observe substantial improvement on our dataset. We further illustrate how Textomics can be used to advance other applications, including evaluating scientific paper embeddings and generating masked templates for scientific paper understanding. Textomics serves as the first benchmark for generating textual summary for genomics data and we envision it will be broadly applied to other biomedical and natural language processing applications.
Self-conditioning pre-trained language models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Text Generation

We present a method to condition pre-trained Transformer-based Language Models without fine-tuning or using additional parameters. Our approach leverages the presence of existing \emph{expert units} in the model that can be used to steer text generation. We describe how to identify such expert units, and propose an inference time intervention upon them at that allows conditioning. Results show that our method is effective for conditioning, even on fine-grained homograph concepts. Furthermore, we use a large corpus of contexts that highlights the presence of inherited gender bias in the output generated by an unconditioned model. Our experiments show that our method can be used to correct this behaviour and to achieve gender parity for all of the contexts. We compare our method with PPLM-BoW (Dathathri et al., 2020), and show that our approach is able to achieve parity at a much lower perplexity. The proposed method is accessible to a wide audience thanks to its simplicity and minimal compute needs.
Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Document Classification, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION

 In document classification for, e.g., legal and biomedical text, we often deal with hundreds of classes, including very infrequent ones, as well as temporal concept drift caused by the influence of real-world events, e.g., policy changes, conflicts, or pandemics. Both class imbalance and drift are often approached by resampling the training data to simulate (or compensate for) a known target distribution, but what if the target distribution is determined by unknown future events? Instead of resampling uniformly to hedge our bets, we focus on the underlying optimization algorithms used to train such document classifiers and evaluate several group-robust optimization algorithms, initially proposed to mitigate group-level disparities. Reframing group-robust algorithms as adaptation algorithms under concept drift, we find that Invariant Risk Minimization and Spectral Decoupling outperform sampling-based approaches to class imbalance and concept drift, and lead to much better performance on minority classes. The effect is more pronounced the larger the label set.
Learning Scalable Representation for Source Code
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Descriptive

This paper presents a scalable distributed code representation (SDCR) learning technique, which addresses the most common sparsity and out-of-vocabulary (OoV) concerns simultaneously. We introduce abstract syntax tree (AST) to reflect the structural information of code snippet and adopt the well-recognized 'bag of AST paths' as its intermediate representation, so that the unique structural and syntactic information of programs can be captured. Our proposed SDCR is supported by two core pillars. First, we provide comprehensive empirical study showing that only 1% of the AST paths can account for approximately 75% of the AST path occurrences. That is, dropping most of unnecessary AST paths still allows SDCR to perform well. Second, all AST paths (without leaf nodes in AST) are made up of a limited number of descriptive path elements, for which a lightweight encoder may produce a good embedding of any AST path. Incorporating these two pillars enables us to represent code snippets with better generalizability and scalability. Based on extensive experiments on two real-world datasets, we show that our SDCR have superior performance against the state-of-the-art with nearly 40% reduction in the number of model parameters.
Sparsifying Transformer Models with Trainable Representation Pooling
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Document Summarization, Text Summarization

We propose a novel method to sparsify attention in the Transformer model by learning to select the most-informative token representations during the training process, thus focusing on the task-specific parts of an input. 
A reduction of quadratic time and memory complexity to sublinear was achieved due to a robust trainable top-$k$ operator.
Our experiments on a challenging long document summarization task show that even our simple baseline performs comparably to the current SOTA, and with trainable pooling we can retain its top quality, while being $1.8\times$ faster during training, $4.5\times$ faster during inference and up to $13\times$ more computationally efficient in the decoder.
Data-adaptive Transfer Learning for Low-resource Translation: A Case Study in Haitian
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cross-Lingual Transfer, Machine Translation, Transfer Learning, Translation

Multilingual transfer techniques often improve low-resource machine translation (MT). Many of these techniques are applied without considering data characteristics. We show in the context of Haitian-to-English translation that transfer effectiveness is correlated with amount of training data and relationships between knowledge-sharing languages. Our experiments suggest that beyond a threshold of authentic data, back-translation augmentation methods are counterproductive, while cross-lingual transfer during training is preferred. We complement this finding by contributing a rule-based French-Haitian orthographic and syntactic engine and a novel method for phonological embedding. When used with multilingual techniques, orthographic transformation significantly improves performance over conventional methods, and phonological transfer greatly improves performance in Jamaican MT.
Global Responses to the COVID-19 Pandemic: A Case Study of Spatiotemporal Evidence Finding and Verification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Fact Verification

This paper explores methods for adapting fact verification models to real-world scenarios that require spatial and temporal inference. As a case study, we search for evidence on governments’ responses to the COVID-19 pandemic. 
We demonstrate that existing fact verification models perform poorly when the verification requires reasoning about spatiotemporal information.
The suggested techniques lead to great improvements and we recommend implementing them for such uses.
Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Chatbot, Retrieval

In this paper, we consider mimicking fictional characters as a promising direction for building engaging conversation models. To this end, we present a new practical task where only a few utterances of each fictional character are available to generate responses mimicking them. Furthermore, we propose a new method named Pseudo Dialog Prompting (PDP) that generates responses by leveraging the power of large-scale language models with prompts containing the target character's utterances. To better reflect the style of the character, PDP builds the prompts in the form of dialog that includes the character's utterances as dialog history. Since only utterances of the characters are available in the proposed task, PDP matches each utterance with an appropriate pseudo-context from a predefined set of context candidates using a retrieval model. Through human and automatic evaluation, we show that PDP generates responses that better reflect the style of fictional characters than baseline methods.
QuoteR: A Benchmark of Quote Recommendation for Writing
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

It is very common to use quotations (quotes) to make our writings more elegant or convincing. To help people find appropriate quotes more efficiently, the task of quote recommendation is presented, aiming to recommend quotes that fit the current context of writing. There have been various quote recommendation approaches, but they are evaluated on different unpublished datasets. To facilitate the research on this task, we build a large and fully open quote recommendation dataset called QuoteR, which comprises three parts including English, standard Chinese and classical Chinese. Any part of it is larger than previous unpublished counterparts. We conduct an extensive evaluation of existing quote recommendation methods on QuoteR. Furthermore, we propose a new quote recommendation model that significantly outperforms previous methods on all three parts of QuoteR. All the code and data of this paper will be released.
Towards Robust Passage Re-Ranking Model by Mitigating Lexical Match Bias
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Information Retrieval, Passage Re-Ranking, Re-Ranking, Retrieval

While deep learning models can overcome the limitations of traditional machine learning algorithms that use hand-crafted features, recent studies have shown that these models often achieve high dataset-specific accuracy by exploiting several bias without understanding deeper semantics of intended task. In this paper, we show that neural re-ranking models in information retrieval field are easily deceived by passage and query which exist some lexical match but are semantically irrelevant or less relevant. Then we create a challenging evaluation dataset to expose model's inability in fully capturing contextual semantic information to learn sequence representation for relevance match. After that in order to encourage model to focus on semantic match between passage and query, we explore adversarial removal method to mitigate model's tendency in learning shortcut of lexical match bias in training corpus. Experiments on two benchmarks show that our debiased models can both achieve certain gains on original and challenging test sets compared with baseline models.
Challenges for Open-domain Targeted Sentiment Analysis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentence, Sentiment Analysis

Since previous studies on open-domain targeted sentiment analysis are limited in dataset domain variety and sentence level,  we propose a novel dataset consisting of 6,013 human-labeled data to extend the data domains in topics of interest and document level. Furthermore, we offer a nested target annotation schema to extract the complete sentiment information in documents, boosting the practicality and effectiveness of open-domain targeted sentiment analysis. Moreover, we leverage the pre-trained model BART in a sequence-to-sequence generation method for the task. Benchmark results show that there exists large room for improvement of open-domain targeted sentiment analysis. Meanwhile, experiments have shown that challenges remain in the effective use of open-domain data, long documents, the complexity of target structure, and domain gaps.
Unsupervised multiple-choice question generation for out-of-domain Q\&A fine-tuning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Multiple-choice, Question Answering, Question Generation, Question-Generation

Pre-trained models have shown very good performances on a number of question answering benchmarks especially when fine-tuned on multiple question answering datasets at once. In this work, we propose an approach for generating a fine-tuning dataset thanks to a rule-based algorithm that generates questions and answers from unannotated sentences. We show that the state-of-the-art model UnifiedQA can greatly benefit from such a system on a multiple-choice benchmark about physics, biology and chemistry it has never been trained on. We further show that improved performances may be obtained by selecting the most challenging distractors (wrong answers), with a dedicated ranker based on a pretrained RoBERTa model.
Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling

Incorporating stronger syntactic biases into neural language models (LMs) is a long-standing goal, but research in this area often focuses on modeling English text, where constituent treebanks are readily available. Extending constituent tree-based LMs to the multilingual setting, where dependency treebanks are more common, is possible via dependency-to-constituency conversion methods. However, this raises the question of which tree formats are best for learning the model, and for which languages. We investigate this question by training recurrent neural network grammars (RNNGs) using various conversion methods, and evaluating them empirically in a multilingual setting. We examine the effect on LM performance across nine conversion methods and five languages through seven types of syntactic tests. On average, the performance of our best model represents a 19 \% increase in accuracy over the worst choice across all languages. Our best model shows the advantage over sequential/overparameterized LMs, suggesting the positive effect of syntax injection in a multilingual setting. Our experiments highlight the importance of choosing the right tree formalism, and provide insights into making an informed decision.
Letters from the past: modeling historical sound change through diachronic character embeddings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Change Detection

While a great deal of work has been done on NLP approaches to Lexical Semantic Change detection,  other  aspects  of  language  change have received less attention from the NLP community. In this paper, we address the detection of  sound  change  through  historical  spelling. We propose that a sound change, a→b / c, can be captured by comparing the relative distance through time between the distributions of the corresponding characters, a and b.  We model these  distributions  using  PPMI  character  embeddings.   We  verify  this  hypothesis  in  synthetic  data  and  then  test  the  method’s  ability to trace the well-known historical change of lenition of plosives in Danish historical sources. We show that the models are able to identify several of the changes under consideration and to uncover meaningful contexts in which they appeared.  The methodology has the potential to  contribute  to  the  study  of  open  questions such as the relative chronology of sound shifts and their geographical distribution.
Negative-aware Entity Set Expansion
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Retrieval

Entity Set Expansion (ESE) aims to find all entities of one target semantic class with a few seed entities describing it. However, existing ESE methods cannot express what entities we explicitly dislike, and thus hinder its application in real-world scenarios.
In this paper, to endow models with the capability of understanding the ``dislike'' relationship among seed entities, we express the target semantic class with both positive and negative seed entities.
To this end, we propose an efficient and learnable negative-aware entity set expansion framework, which is essentially a retrieval model. To facilitate this study, a large-scale Negative-aware ESE Dataset NED with more than 1M entities is further collected and annotated. Extensive experiments on NED show that the proposed framework can effectively understand the dislike relations expressed by the negative seeds and expand fewer dislike entities than baseline methods.
Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Machine Translation, Retrieval, Sentence, Translation

Multi-encoder models are a broad family of context-aware neural machine translation systems that aims to improve translation quality by encoding document-level contextual information alongside the current sentence. The context encoding is undertaken by contextual parameters, trained on document-level data. In this work, we discuss the difficulty of training these parameters effectively, due to the sparsity of the words in need of context (i.e., the training signal), and their relevant context. We propose to pre-train the contextual parameters over split sentence pairs, which makes an efficient use of the available data for two reasons. Firstly, it increases the contextual training signal by breaking intra-sentential syntactic relations, and thus pushing the model to search the context for disambiguating clues more frequently. Secondly, it eases the retrieval of relevant context, since context segments become shorter. We propose four different splitting methods, and evaluate our approach with BLEU and contrastive test sets. Results show that it consistently improves learning of contextual parameters, both in low and high resource settings.