pwc-archive/papers-with-abstracts huggingface dataset:
Discrete Structural Planning for Neural Machine Translation
Authors: Anonymous
Date: 2018-05-23 00:00:00
Tasks: Machine Translation, Sentence, Text Generation, Translation

Structural planning is important for producing long sentences, which is a missing part in current language generation models. In this work, we add a planning phase in neural machine translation to control the coarse structure of output sentences. The model first generates some planner codes, then predicts real output words conditioned on them. The codes are learned to capture the coarse structure of the target sentence. In order to learn the codes, we design an end-to-end neural network with a discretization bottleneck, which predicts the simplified part-of-speech tags of target sentences. Experiments show that the translation performance are generally improved by planning ahead. We also find that translations with different structures can be obtained by manipulating the planner codes.
How to Seq2seq for SQL
Authors: Anonymous
Date: 2018-04-23 00:00:00
Tasks: Question Answering, Table-based Question Answering

Translating natural language to SQL queries for table-based question answering has recently attracted more research attention. Previous approaches develop models whose architecture is specifically tuned to the structure of the task, such as separately predicting the arguments of the SELECT clause. In this work, we show that a more general attention-based sequence-to-sequence model outperforms more specialized state-of-the-art approaches by only adapting the input and output layers. In particular, we extend it with on-the-fly embedding and output vectors as well as an input copying mechanism, which are used throughout the whole decoding process. We also investigate the potential order-matters problem that could arise due to having multiple correct decoding paths and investigate the use of a dynamic oracle in this context.
Distributional Representation Clusters Complement Part-of-Speech Tags
Authors: Anonymous
Date: 2018-05-22 00:00:00

Many works have successfully co-opted word clusters derived from distributional information, such as Brown clusters, as features in language processing tasks. We note that not only do such clusters make poor proxies for part-of-speech tags; these clusters are in fact quite different from part-of-speech tags. This paper investigates the gap between Brown clusters, clusterings in word embedding space, and part-of-speech tags, across a range of languages. We find that, while word types clustered together may seem at a glance to be cohesive, distributionally derived clusters in fact strongly complement part-of-speech tags across many languages, suggesting a surprising amount of difference between the information contained in these representations.
Linguistically-Informed Self-Attention for Semantic Role Labeling
Authors: Anonymous
Date: 2018-04-08 00:00:00
Tasks: Dependency Parsing, Multi-Task Learning, Predicate Detection, Semantic Role Labeling

The current state-of-the-art end-to-end semantic role labeling (SRL) model is a deep neural network architecture with no explicit linguistic features. 
However, prior work has shown that gold syntax trees can dramatically improve SRL, suggesting that neural network models could see great improvements from explicit modeling of syntax.
In this work, we present linguistically-informed self-attention (LISA): a new neural network model that combines 
multi-head self-attention with multi-task learning across dependency parsing, part-of-speech, predicate detection and SRL. For example, syntax is incorporated by training one of the attention heads to attend to syntactic parents for each token. Our model can predict all of the above tasks, but it is also trained such that if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model.
In experiments on the CoNLL-2005 SRL dataset LISA achieves an increase of 2.5 F1 absolute over the previous state-of-the-art on newswire with predicted predicates and more than 2.0 F1 on out-of-domain data. On ConLL-2012 English SRL we also show an improvement of more than 3.0 F1, a 13% reduction in error.
Maximally Consistent Sampling and the Jaccard Index of Probability Distributions
Authors: Anonymous
Date: 2018-03-29 00:00:00

We introduce simple, efficient algorithms for computing a MinHash of a probability distribution, suitable for both sparse and dense data, with equivalent running times to the state of the art for both cases. The collision probability of these algorithms is a new measure of the similarity of positive vectors which we investigate in detail. We describe the sense in which this collision probability is optimal for any Locality Sensitive Hash based on sampling. We argue that this similarity measure is more useful for probability distributions than the similarity pursued by other algorithms for weighted MinHash, and is the natural generalization of the Jaccard index.
Multi-time-horizon Solar Forecasting Using Recurrent Neural Network
Authors: Anonymous
Date: 2018-03-25 00:00:00
Tasks: Prediction

The non-stationarity characteristic of the solar power renders traditional point forecasting methods to be less useful due to large prediction errors. This results in increased uncertainties in the grid operation, thereby negatively affecting the reliability and resulting in increased cost of operation. This research paper proposes a unified architecture for multi-time-horizon solar forecasting for short and long-term predictions using Recurrent Neural Networks (RNN). The paper describes an end-to-end pipeline to implement the architecture along with methods to test and validate the performance of the prediction model. The results demonstrate that the proposed method based on the unified architecture is effective for multi-horizon solar forecasting and achieves a lower root-mean-squared prediction error compared to the previous best performing methods which use one model for each time-horizon. The proposed method enables multi-horizon forecasts with real-time inputs, which have a high potential for practical applications in the evolving smart grid.
RFC-0000 - RFC on RFCs
Authors: Frédéric Kaplan, Kevin Baumer, Mike Kestemont, Daniel Jeller
Date: 2020-08-11 00:00:00
Proceeding: TimeMachine RFC 2031 1

Reaching consensus on the technology options to pursue in a programme as large as Time Machine is a complex issue. To ensure the open development and evaluation of work, a process inspired by the Request for Comments (RFC) that was used for the development of the Internet protocol1 is being adapted to the needs of Time Machine. Time Machine Requests for Comments are freely accessible publications, identified with a unique ID, that constitute the main process for establishing rules, recommendations and core architectural choices for Time Machine components.
BERT Learns to Teach: Knowledge Distillation with Meta Learning
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Distillation, Meta-Learning

We present Knowledge Distillation with Meta Learning (MetaDistil), a simple yet effective alternative to traditional knowledge distillation (KD) methods where the teacher model is fixed during training. We show the teacher network can learn to better transfer knowledge to the student network (i.e., learning to teach) with the feedback from the performance of the distilled student network in a meta learning framework. Moreover, we introduce a pilot update mechanism to improve the alignment between the inner-learner and meta-learner in meta learning algorithms that focus on an improved inner-learner. Experiments on various benchmarks show that MetaDistil can yield significant improvements compared with traditional KD algorithms and is less sensitive to the choice of different student capacity and hyperparameters, facilitating the use of KD on different tasks and models.
Quantifying the Task-Specific Information in Text-Based Classifications
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8

Recently, neural natural language models have attained state-of-the-art performance on a wide variety of tasks, but the high performance can result from superficial, surface-level cues (Bender and Koller, 2020; Niven and Kao, 2020). These surface cues, as the ``shortcuts'' inherent in the datasets, do not contribute to the task-specific information (TSI) of the classification tasks. While it is essential to look at the model performance, it is also important to understand the datasets. In this paper, we consider this question: Apart from the information introduced by the shortcut features, how much task-specific information is required to classify a dataset? We formulate this quantity in an information-theoretic framework. While this quantity is hard to compute, we approximate it with a fast and stable method. TSI quantifies the amount of linguistic knowledge modulo a set of predefined shortcuts -- that contributes to classifying a sample from each dataset. This framework allows us to compare across datasets, saying that, apart from a set of ``shortcut features'', classifying the Multi-NLI task involves around 0.4 nats more TSI than the Quora Question Pair.
An Empirical Survey of Data Augmentation \\for Limited Data Learning in NLP
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Data Augmentation, News Classification, Sentence


NLP has achieved great progress in the past decade through the use of neural models and large labeled datasets. The dependence on abundant data prevents NLP models from being applied to low-resource settings or novel tasks where significant time, money, or expertise is required to label massive amounts of textual data. Recently, data augmentation methods have been explored as a means of improving data efficiency in NLP. To date, there has been no systematic empirical overview of data augmentation for NLP in the limited labeled data setting, making it difficult to understand which methods work in which settings. In this paper, we provide an empirical survey of recent progress on data augmentation for NLP in the limited labeled data setting, summarizing the landscape of methods (including token-level augmentations, sentence-level augmentations, adversarial augmentations, and hidden-space augmentations) and carrying out experiments on 11 datasets covering topics/news classification, inference tasks, paraphrasing tasks, and single-sentence tasks. Based on the results, we draw several conclusions to help practitioners choose appropriate augmentations in different settings and discuss the current challenges and future directions for limited data learning in NLP.
Reliability and Robustness of Transformers for Automated Short-Answer Grading
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Fairness, Transfer Learning

Short-Answer Grading (SAG) is an application for NLP in education where student
answers to open questions are graded. This task places high demands both on the
reliability (accuracy and fairness) of label predictions and model robustness
against strategic, "adversarial"  input. Neural approaches are powerful tools for
many problems in NLP, and transfer learning for Transformer-based models
specificially promises to support data-poor tasks as this. We analyse the
performance of a Transfomer-based SOTA model, zooming in on class- and item type
specific behavior in order to gauge reliability; we use adversarial testing to
analyze the the model's robustness towards strategic answers. We find a strong
dependence on the specifics of training and test data, and recommend that model
performance be verified for each individual use case.
On the Diversity and Limits of Human Explanations
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Diversity

A growing effort in NLP aims to build datasets of human explanations. However, the term explanation encompasses a broad range of notions, each with different properties and ramifications. Our goal is to provide an overview of diverse types of explanations and human limitations, and discuss implications for collecting and using explanations in NLP. Inspired by prior work in psychology and cognitive sciences, we group existing human explanations in NLP into three categories: proximal mechanism, evidence, and procedure. These three types differ in nature and have implications for the resultant explanations. For instance, procedure is not considered explanations in psychology and connects with a rich body of work on learning from instructions. The diversity of explanations is further evidenced by proxy questions that are needed for annotators to interpret and answer open-ended why questions. Finally, explanations may require different, often deeper, understandings than predictions, which casts doubt on whether humans can provide useful explanations in some tasks.
QAConv: Question Answering on Informative Conversations
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Question Answering

This paper introduces QAConv, a new question answering (QA) dataset that uses conversations as a knowledge source. We focus on informative conversations, including business emails, panel discussions, and work channels. Unlike open-domain and task-oriented dialogues, these conversations are usually long, complex, asynchronous, and involve strong domain knowledge. In total, we collect 34,608 QA pairs, including span-based and unanswerable questions, from 10,259 selected conversations with both human-written and machine-generated questions. We use a question generator and a dialogue summarizer as auxiliary tools to collect multi-hop questions. The dataset has two testing scenarios, chunk mode and full mode, depending on whether the grounded partial conversation is provided or retrieved. Experimental results show that state-of-the-art pretrained QA systems have limited zero-shot performance and tend to predict our questions as unanswerable. Our dataset provides a new training and evaluation testbed to facilitate QA on conversations research.
Tailor: Generating and Perturbing Text with Semantic Controls
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation, Natural Language Inference, Style Transfer

Making controlled perturbations is essential for various tasks (e.g., data augmentation), but building task-specific generators can be expensive. We introduce Tailor, a task-agnostic generation system that perturbs text in a semantically-controlled way. With unlikelihood training, Tailor's generator is designed to follow a series of control codes derived from semantic roles. Through modifications of these control codes, Tailor can produce fine-grained perturbations. We implement a set of operations on control codes that can be composed into complex perturbation strategies, and demonstrate their effectiveness in three applications. First, Tailor facilitates the construction of high-quality contrast sets that are lexically diverse and less biased than original task test data. Second, paired with automated labeling heuristics, Tailor helps improve model generalization through data augmentation: we obtain an average gain of 1.73 on an (natural language inference) NLI challenge set by perturbing just $\sim5\%$ of training data. Third, without any finetuning overhead, Tailor's perturbations effectively improve compositionality in fine-grained style transfer, outperforming fine-tuned baselines on 5 transfers.
VALSE: A Task-Independent Benchmark for Vision and Language Models centered on Linguistic Phenomena
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA), Visual Reasoning

We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for specific visio-linguistic grounding capabilities. Currently, V&L models are evaluated on tasks such as visual question answering or visual reasoning, which do not address their fine-grained linguistic capabilities. VALSE addresses this gap by offering a suite of six tests targeting specific linguistic phenomena. Solving these tests requires models to ground these  phenomena in the visual modality, allowing more fine-grained evaluations than hitherto possible. We build VALSE using methods that support the construction of reliable foils, and report results from evaluating five widely-used V&L models. Our experiments suggest that current models have considerable difficulty addressing most phenomena. Hence, we expect VALSE to serve as an important benchmark to measure future progress of pretrained V&L models from a linguistic perspective, complementing the canonical task-centred V&L evaluations.
Deduplicating Training Data Makes Language Models Better
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Language Modeling, Language Modelling, Sentence

We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings.
As a result, over $1\%$ of the unprompted output of language models trained on these datasets is copied verbatim from the training data.
We develop two tools that allow us to deduplicate training datasets---for example removing from C4 a single 61 word English sentence that is repeated over $60{,}000$ times.
Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer training steps to achieve the same or better accuracy.
We can also reduce train-test overlap, which affects over $4\%$ of the validation set of standard datasets, thus allowing for more accurate evaluation.
A Comparison of Strategies for Source-Free Domain Adaptation
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Active Learning, Data Augmentation, Domain Adaptation, Source-Free Domain Adaptation

Data sharing restrictions are common in NLP, especially in the clinical domain, but there is limited research on adapting models to new domains without access to the original training data, a setting known as source-free domain adaptation. We take algorithms that traditionally assume access to the source-domain training data—active learning, self-training, and data augmentation—and adapt them for source free domain adaptation. Then we systematically compare these different strategies across multiple tasks and domains. We find that active learning yields consistent gains across all SemEval 2021 Task 10 tasks and domains, but though the shared task saw successful self-trained and data augmented models, our systematic comparison finds these strategies to be unreliable for source-free domain adaptation.
Multimodal Knowledge Learning for Named Entity Disambiguation
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Entity Disambiguation, Meta-Learning, Transfer Learning

With the popularity of online social medias in recent years, massive-scale multimodal information has brought new challenges to traditional Named Entity Disambiguation (NED) tasks. Recently, Multimodal Named Entity Disambiguation (MNED) is proposed to link ambiguous mentions with the textual and visual contexts to a predefined knowledge graph.  Recent attempts handle these issues mainly by annotating multimodal mentions and adding multimodal features to traditional NED models. These methods still suffer from 1) lack of multimodal annotation data against the huge scale of unlabeled corpus  and 2) failing to model multimodal information at knowledge level. In this paper, we explore a pioneer study on leveraging multimodal knowledge learning to address the MNED task. Specifically, we propose a knowledge-guided transfer learning strategy to extract unified representation from different modalities and enrich multimodal lnowledge in a Meta Learning way which is much easier than collecting ambiguous mention corpus. Then we propose an Interactive Multimodal Learning Network (IMN), which is capable of fully utilizing the multimodal information in both mention and knowledge side. To verify the validity of the proposed method, we implemented comparisons on a public large-scale MNED dataset based on Twitter KB. Experimental results show that our method is superior to the state-of-the-art multimodal methods
Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Classification, Few-Shot Text Classification, Language Modeling, Language Modelling, text-classification, Text Classification

Prompt-based learning (i.e., prompting) is an emerging paradigm for exploiting knowledge learned by a pretrained language model. In this paper, we propose Automatic Multi-Label Prompting (AMuLaP), a simple yet effective method to automatically select label mappings for few-shot text classification with prompting. Our method exploits one-to-many label mappings and a statistics-based algorithm to select label mappings given a prompt template. Our experiments demonstrate that AMuLaP achieves competitive performance on the GLUE benchmark without human effort or external resources.
A Latent-Variable Model for Intrinsic Probing
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Attribute, model

The success of pre-trained contextualized representations has prompted researchers to analyze them for the presence of linguistic information. Indeed, it is natural to assume that these pre-trained representations do encode some level of linguistic knowledge as they have brought about large empirical improvements on a wide variety of NLP tasks, which suggests they are learning true linguistic generalization.
In this work, we focus on intrinsic probing, an analysis technique where the goal is not only to identify whether a representation encodes a linguistic attribute, but also to pinpoint where this attribute is encoded. We propose a novel latent-variable formulation for constructing intrinsic probes and derive a tractable variational approximation to the log-likelihood. Our results show that our model is versatile and outperforms two intrinsic probes previously proposed in the literature. Finally, we find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of  morphosyntax.
Semantics-aware Attention Improves Neural Machine Translation
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Decoder, Machine Translation, Translation

The integration of syntactic structure into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel parameter-free methods for injecting semantic information into Transformers, both rely on semantics-aware masking of (some of) the attention heads. One such method operates on the encoder, through a Scene-Aware Self-Attention (SASA) head. Another on the decoder, through a Scene-Aware Cross-Attention (SACrA) head. We show a consistent improvement over the vanilla Transformer and syntax-aware models for four language pairs.
We further show an additional gain when using both semantic and syntactic structures in some language pairs.