pwc-archive/papers-with-abstracts huggingface dataset:
VinAI at ChEMU 2020: An accurate system for named entity recognition in chemical reactions from patents
Authors: Mai Hoang Dao, Dat Quoc Nguyen
Date: 2020-09-01 00:00:00
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Word Embeddings

This paper describes our VinAI system for the ChEMU task 1 of named entity recognition (NER) in chemical reactions. Our system employs a BiLSTM-CNN-CRF architecture with additional contextualized word embeddings. It achieves very high performance, officially ranking second with regards to both exact- and relaxed-match F1 scores at 94.33% and 96.84%, respectively. In a post-evaluation phase, fixing a mapping bug which converts the column-based format into the brat standoff format helps our system to obtain higher results. In particular, we obtain an exact-match F1 score at 95.21% and especially a relaxed-match F1 score at 97.26%, thus achieving the highest relaxed-match F1 compared to all other participating systems. We believe our system can serve as a strong baseline for future research and downstream applications of chemical NER over chemical reactions from patents.
On Nondeterminism and Instability in Optimizing Neural Networks
Authors: Anonymous
Date: 2020-08-21 00:00:00
Tasks: Data Augmentation, Diversity

Nondeterminism in optimization causes uncertainty when improving neural networks, with small gains in performance difficult to discern from run-to-run variability. While this uncertainty can be reduced by training multiple copies of a model with different random seeds, doing so can be time-consuming, costly, and is energy-inefficient. Despite this, little attention has been paid towards establishing an understanding of this problem. In this work, we establish an experimental protocol for understanding the effect of optimization nondeterminism on model diversity, which allows us to study the independent effects of nondeterminism sources, including random parameter initialization, data augmentation, data shuffling, and even low-level nondeterminism in popular accelerator libraries like cuDNN. Surprisingly, we find that changes to each source of nondeterminism all have similar effects on measures of model diversity. To explain this intriguing fact, we examine and identify the instability of model training, when taken as an end-to-end procedure, as the key determinant. We show that even one-bit changes in initial model parameters result in models that converge to vastly different values, and even place a lower bound of $10^7$ on the condition number of model training for example architectures considered.
Supersymmetric Artificial Neural Network
Authors: Anonymous
Date: 2019-03-28 00:00:00
Tasks: Deep Learning

The “Supersymmetric Artificial Neural Network” in deep learning (denoted (x; θ, bar{θ})Tw), espouses the importance of considering biological constraints in the aim of further generalizing backward propagation. 

Looking at the progression of ‘solution geometries’; going from SO(n) representation (such as Perceptron like models) to SU(n) representation (such as UnitaryRNNs) has guaranteed richer and richer representations in weight space of the artificial neural network, and hence better and better hypotheses were generatable. The Supersymmetric Artificial Neural Network explores a natural step forward, namely SU(m|n) representation. These supersymmetric biological brain representations (Perez et al.) can be represented by supercharge compatible special unitary notation SU(m|n), or (x; θ, bar{θ})Tw parameterized by θ, bar{θ}, which are supersymmetric directions, unlike θ seen in the typical non-supersymmetric deep learning model. Notably, Supersymmetric values can encode or represent more information than the typical deep learning model, in terms of “partner potential” signals for example.
Plug-and-Play Conversational Models
Authors: Anonymous
Date: 2020-07-23 00:00:00
Tasks: Attribute, Language Modeling, Language Modelling, Large Language Model, Response Generation

There has been considerable progress made towards conversational models that generate coherent and fluent responses; however, this often involves training large language models on large dialogue datasets, such as Reddit. These large conversational models provide little control over the generated responses, and this control is further limited in the absence of annotated conversational datasets for attribute specific generation that can be used for fine-tuning the model. In this paper, we first propose and evaluate plug-and-play methods for controllable response generation, which does not require dialogue specific datasets and does not rely on fine-tuning a large model. While effective, the decoding procedure induces considerable computational overhead, rendering the conversational model unsuitable for interactive usage. To overcome this, we introduce an approach that does not require further computation at decoding time, while also does not require any fine-tuning of a large language model. We demonstrate, through extensive automatic and human evaluation, a high degree
of control over the generated conversational responses with regard to multiple desired attributes, while being fluent.
Decentralized Attribution of Generative Models
Authors: Anonymous
Date: 2020-07-12 00:00:00

There have been growing concerns regarding the fabrication of contents through generative models. 
This paper investigates the feasibility of decentralized attribution of such models. 
Given a group of models derived from the same dataset and published by different users, attributability is achieved when a public verification service associated with each model (a linear classifier) returns positive only for outputs of that model. 
Attribution allows tracing of machine-generated contents back to its source model, thus facilitating IP-protection and content regulation. 
Decentralized attribution prevents forgery of source models by only allowing users to have access to their own classifiers, which are parameterized by keys distributed by a registry.
Our major contribution is the development of design rules for the keys, which are derived from first-order sufficient conditions for decentralized attribution. Through validation on MNIST, CelebA and Cityscapes, we show that keys need to be (1) orthogonal or opposite to each other and (2) belonging to a subspace dependent on the data distribution and the architecture of the generative model. 
We also empirically examine the trade-off between generation quality and robust attributability against adversarial post-processes of model outputs.
NLP Service APIs and Models for Efficient Registration of New Clients
Authors: Anonymous
Date: 2020-07-07 00:00:00
Tasks: GPU, Language Modeling, Language Modelling, NER

State-of-the-art NLP inference uses enormousneural architectures and models  trained  for GPU-months, well beyond the reach of most consumers  of  NLP. This has led to one-size-fits-all public API-based NLP service models by major AI companies, serving large numbers of clients. Neither (hardware deficient) clients nor (heavily subscribed) servers can afford traditional fine tuning. Many clients own little or no labeled data. We initiate a study of adaptation  of centralized  NLP services to clients, and present one practical and lightweight approach. Each client uses an unsupervised, corpus-based sketch to register to the service. The server uses an auxiliary network to map the sketch to an abstract vector representation, which then informs the main labeling network. When a new client registers with its sketch, it gets immediate accuracy benefits. We demonstrate the success of the proposed architecture using sentiment labeling, NER, and predictive language modeling.
Building Production-Quality NLG Models with Minimal Labelled Data
Authors: Anonymous
Date: 2020-06-29 00:00:00
Tasks: Text Generation

Natural language generation (NLG) plays an important role in task-oriented dialog systems to provide meaningful and natural responses to user's requests. However, training a NLG model that could surface production-ready quality responses usually requires a large amount of training data. In this paper, we propose two novel data-efficient approaches to bootstrap the model. We first propose a template-based approach that leverages a scenario generation framework to create full coverage of possible scenarios and their corresponding synthetic annotations. Secondly, we leverage the pretrained BART model with a bucketing method that groups scenarios based on their dialog act structures. Extensive experiments on three datasets show our approaches achieve production-quality with 10 times less labelled data than a standard NLG dataset.
Large image datasets: A pyrrhic win for computer vision?
Authors: Anonymous
Date: 2020-06-23 00:00:00

vision datasets. We examine broad issues such as the question of consent and justice as well as specific concerns such as the inclusion of verifiably pornographic images in datasets. Taking the ImageNet-ILSVRC-2012 dataset as an example, we perform a cross-sectional model-based quantitative census covering factors such as age, gender, NSFW content scoring, class-wise accuracy, human-cardinality-analysis, and the semanticity of the image class information in order to statistically investigate the extent and subtleties of ethical transgressions. We then use the census to help hand-curate a look-up-table of images in the ImageNet-ILSVRC-2012 dataset that fall into the categories of verifiably pornographic: shot in a non-consensual setting (up-skirt), beach voyeuristic, and exposed private parts. We survey the landscape of harm and threats both society broadly and individuals face due to uncritical and ill-considered dataset curation practices. We then propose possible courses of correction and critique the pros and cons of these. We have duly open-sourced all of the code and the census meta-datasets generated in this endeavor for the computer vision community to build on.
KARL: Knowledge-Aware Reasoning Memory Modeling with Reinforcement Learning of Vector Space
Authors: Jiazheng Chen
Date: 2019-12-16 00:00:00
Tasks: Decoder, Language Modeling, Language Modelling, Question Answering, reinforcement-learning, Reinforcement Learning (RL)

Founded in Atkinson-Shiffrin Memory Model's three-stage theory, the cognitive process of answering a question with stored knowledge can be seen as a reasoning process that goes from the external sensory memory via short-term knowledge-aware reasoning towards the internal knowledge storage. While in the machine, this process can be interpreted as a pipeline from encoding the question’s information, through decoding the reasoning query of triples, towards the knowledgebase. How to encode and decode the varying semantic information of question into accurate triple query, and how to adjust the query generation with an evolving knowledgebase, are two inevitable problems in this cognitive process. Our model KARL, provides a solution by designing the three memory spaces as, an encoder to handle the language modeling, a decoder for query generation, and a self-calibration with reinforcement learning of the knowledge representation vector space. We evaluate the model's reasoning ability in knowledge-based question answering against the Question Answering over Linked Data (QALD) benchmark, and achieve significant improvements in answers accuracy as compared to other neural models.
Record-to-Text Generation with Style Imitation
Authors: Anonymous
Date: 2020-06-17 00:00:00
Tasks: Sentence, Text Generation

Recent neural approaches to record-to-text generation have mostly focused on improving content ﬁdelity while lacking explicit control over writing styles (e.g., sentence structures, word choices). More traditional systems use templates to determine the realization of text. Yet manual or automatic construction of high-quality templates is difﬁcult, and a template acting as hard constraints could harm content ﬁdelity when it does not match the record perfectly. We study a new way of stylistic control by using existing sentences as “soft” templates. That is, a model learns to imitate the writing style of any given exemplar sentence, with automatic adaptions to faithfully describe the record. The problem is challenging due to the lack of parallel data. We develop a neural approach that includes a hybrid attention-copy mechanism, learns with weak supervisions, and is enhanced with a new content coverage constraint. We conduct experiments in restaurants and sports domains. Results show our approach achieves stronger performance than a range of comparison methods. Our approach balances well between content ﬁdelity and style control given exemplars that match the records to varying degrees.
Summarizing Text on Any Aspects: A Knowledge-Informed Weakly-Supervised Approach
Authors: Anonymous
Date: 2020-06-17 00:00:00
Tasks: Abstractive Text Summarization

Given a document and a target aspect (e.g., a topic of interest), aspect-based abstractive summarization attempts to generate a summary with respect to the aspect. Previous studies have assumed a small pre-defined set of aspects and fall short of summarizing on other diverse topics. In this work, we study summarizing on arbitrary aspects relevant to the document, which significantly expands the application of the task in practice. Due to the lack of supervision data, we develop a new weak supervision construction method and an aspect modeling scheme, both of which integrate rich external knowledge sources such as ConceptNet and Wikipedia. Experiments show our approach achieves performance boosts on summarizing real and synthetic documents given pre-defined or arbitrary aspects.
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems
Authors: Anonymous
Date: 2020-06-08 00:00:00
Tasks: Dialogue State Tracking, Management, Task-Oriented Dialogue Systems

Task-oriented dialogue systems are either modularized with separate dialogue state tracking (DST) and management steps or end-to-end trainable. In either case, the knowledge base (KB) plays an essential role in fulfilling user requests. Modularized systems rely on DST to interact with the KB, which is expensive in terms of annotation and inference time. End-to-end systems use the KB directly as input, but they cannot scale when the KB is larger than a few hundred entries. In this paper, we propose a method to embed the KB, of any size, directly into the model parameters. The resulting model does not require any DST or template responses, nor the KB as input, and it can dynamically update its KB via fine-tuning. We evaluate our solution in five task-oriented dialogue datasets with small, medium, and large KB size. Our experiments show that end-to-end models can effectively embed knowledge bases in their parameters and achieve competitive performance in all evaluated datasets.
Learning Helpful Inductive Biases from Self-Supervised Pretraining
Authors: Anonymous
Date: 2020-06-06 00:00:00
Tasks: Diagnostic

Large pretrained language models demonstrate strong, language-specific biases during fine-tuning that allow them to solve language tasks better than models without pretraining. We aim to characterize these biases, and identify the amount of pretraining that is necessary to acquire them. We introduce a new English language diagnostic set called MSGS (Mixed Signals Generalization Set) which contains two types of data: mixed data in which the labels are consistent with both a linguistic classification (e.g., Is the main verb in the progressive form?) and a superficial surface one (e.g., Does "the" precede "a"?); and unmixed data in which the labels align only with the linguistic feature. We fine-tune RoBERTa on mixed data (with and without small amounts of inoculating unmixed data) and test on unmixed data to see which feature it has bias in favor of. We pretrain RoBERTa from scratch on quantities of data ranging from 1M to 1B words and compare their performance on MSGS to the publicly available RoBERTa-Base. We find steady growth in linguistic bias with increased pretraining data. The models we test can usually represent the linguistic features, but they only learn to prefer to generalize based on these features with significant pretraining. In the absence of inoculating data, only RoBERTa-Base consistently demonstrates a linguistic bias with any regularity.
Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension
Authors: Anonymous
Date: 2020-06-03 00:00:00
Tasks: Machine Reading Comprehension, Question Answering, Reading Comprehension

While neural networks with attention mechanisms have achieved superior performance on many natural language processing tasks, it remains unclear to which extent learned attention resembles human visual attention.
We study the similarity between human visual and neural attention and analyze if neural attention-based methods perform better if they mimic human attention.
To this end, we compare state-of-the-art networks based on long short-term memory (LSTM), convolutional neural (CNN) and XLNet Transformer architectures on a question answering task.
We evaluate all methods on a novel 23-participant dataset of eye tracking data recorded while reading movie plots.
We find that while higher similarity to human attention and performance significantly correlates to the LSTM and CNN this does not hold true for the XLNet -- despite the fact that the XLNet performs best on this challenging task.
Our work not only shows that different architectures seem to learn rather different neural attention but also that similarity of neural to human attention is not necessarily helpful and hence desirable.
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Authors: Anonymous
Date: 2020-06-03 00:00:00
Tasks: Cross-Lingual Transfer, Question Answering, XLM-R, Zero-Shot Cross-Lingual Transfer

Intermediate-task training has been shown to substantially improve pretrained model performance on many language understanding tasks, at least in monolingual English settings. Here, we investigate whether English intermediate-task training is still helpful on non-English target tasks in a zero-shot cross-lingual setting. Using a set of 7 intermediate language understanding tasks, we evaluate intermediate-task transfer in a zero-shot cross-lingual setting on 9 target tasks from the XTREME benchmark. Intermediate-task training yields large improvements on the BUCC and Tatoeba tasks that use model representations directly without training, and moderate improvements on question-answering target tasks. Using SQuAD for intermediate training achieves the best results across target tasks, with an average improvement of 8.4 points on development sets. Selecting the best intermediate task model for each target task, we obtain a 6.1 point improvement over XLM-R Large on the XTREME benchmark, setting a new state of the art. Finally, we show that neither multi-task intermediate-task training nor continuing multilingual MLM during intermediate-task training offer significant improvements.
Segatron: Segment-aware Transformer for Language Modeling and Understanding
Authors: Anonymous
Date: 2020-06-02 00:00:00
Tasks: Language Modeling, Language Modelling, Masked Language Modeling, Position, Sentence

Transformers are powerful for sequence modeling and have the potential of learning long-term dependency. Nearly all state-of-the-art language models and pre-trained language models are based on the Transformer architecture. However, it distinguishes sequential tokens only with the token position index. We hypothesize that better contextual representations can be generated from the Transformer with richer positional information. To verify this, we propose a segment-aware Transformer~(Segatron), by replacing the original token position encoding with a combined position encoding of paragraph, sentence, and token. We first introduce the segment-aware mechanism to the Transformer-XL, which is a popular Transformer model based on the relative position encoding with memory extension. Our proposed method outperforms the Transformer-XL base model and large model on the Wiki103 dataset over 1.5 and 1.2 perplexities, respectively, which is comparable to the state-of-the-art result. We further pre-trained our model on the masked language modeling task in BERT but without any affiliated tasks. Experimental results show that our pre-trained model can outperform the original BERT model on various NLP tasks.
Improving Distantly-Supervised Relation Extraction through BERT-based Label & Instance Embeddings
Authors: Anonymous
Date: 2020-06-01 00:00:00
Tasks: Relation, Relation Extraction

Distantly-supervised relation extraction (RE) is an effective method to scale RE to large corpora but suffers from noisy labels. Existing approaches try to alleviate noise through multi-instance learning and by providing additional information, but they treat labels as independent and seldom consider the relationship between labels and entities. Consequently, potentially valuable information goes unused. We propose REDSandT (Relation Extraction with Distant Supervision and Transformers), a novel distantly-supervised transformer-based RE method that manages to capture highly informative instance and label embeddings for RE by exploiting BERT's pre-trained model. We guide REDSandT to focus solely on relational tokens by fine-tuning BERT on a structured input, including the sub-tree connecting an entity pair and the entities' types. Using the extracted informative vectors, we shape label embeddings, which we also use as attention mechanism over instances to further reduce noise. Finally, we represent sentences by concatenating relation and instance embeddings. Experiments in the NYT-10 dataset show that REDSandT captures a broader set of relations with higher confidence,  achieving state-of-the-art AUC (0.424).
Apply Lean Thinking With Industrial Experience To Professors' Careers Opinion And Survey
Authors: Anonymous
Date: 2020-06-01 00:00:00
Tasks: Survey

This paper describes how professors could apply lean thinking and industrial experience to their careers. There is also a survey of the usefulness of this method after professors read this paper. The entire paper is divided into two parts, with the first part covering the author’s opinions and the second part covering the survey result. The novel idea presented in the first part combines lean thinking, the author’s academic and industrial experience to optimize the professors’ career development process. This paper should be of interest to university professors and postdocs. Students who are in graduate programs will also benefit from learning how their professors think.
An algorithm for controlled text analysis on Wikipedia
Authors: Anonymous
Date: 2020-05-28 00:00:00
Tasks: Articles, Attribute

While numerous work has examined bias on Wikipedia, most approaches fail to control for possible confounding variables. In this work, given a target corpus for analysis (e.g. biography pages about women), we present a method for constructing a control corpus that matches the target corpus in as many attributes as possible, except the target attribute (e.g. the gender of the subject). This methodology can be used to analyze specific types of bias in Wikipedia articles, for example, gender or racial bias, while minimizing the influence of confounding variables.