pwc-archive/papers-with-abstracts huggingface dataset:
Semantic Diversity by Phonetics for Accurate and Robust Machine Translation
Authors: Anonymous
Date: 2019-05-28 00:00:00
Tasks: Diversity, Machine Translation, NMT, Translation

Neural Machine Translation (NMT) learns from examples, and thus often lacks robustness against noise. Previous work has shown that integrating noise into the training process is effective at improving such robustness, but this solution can be inefficient due to the exponential number of string perturbations, i.e., exponential in the number of words or characters. To robustify the translation input, we treat human phonetic interaction throughout history as a pre-compiled computational device. This device implements a many-to-one function that converts text into phonetics.  To the best of our knowledge, we are the first in Machine Translation, to apply the phonetic algorithms Soundex, NYSIIS, and MetaPhone to foreign word/character sequences.  We also apply another linguistic representation, the logogram inference, Wubi, for Chinese. To explain why phonetic encodings improve NMT, we introduce, quantify, and empirically verify our hypothesis: "one phonetic representation usually corresponds to words that are semantically diverse." Driven by our hypothesis, we simulate this "natural" phonetic device and introduce an artificial method called random clustering. We achieved significant and consistent improvements overall language pairs and datasets we experimented with: French-English, German-English, and Chinese-English in IWSLT'17, with up to nearly 2 BLEU points over the state-of-the-art. Moreover, our approaches are more robust than baselines when evaluated on unknown noisy or out-of-domain test sets, with up to about 5 BLEU point increase. Upon acceptance, all software source code and experiments will be available as Open Source.
Modelling Retinal Vessel Flicker Response
Authors: Anonymous
Date: 2019-07-18 00:00:00

We propose a parsimonious mathematical model for the human retinal vessel flicker response. The model is derived from data recorded by Dynamical Vessel Analyzer (DVA, Imedos GMBH, Germany).
An Effective Multi-Stage Approach For Question Answering
Authors: Anonymous
Date: 2019-07-10 00:00:00
Tasks: Machine Reading Comprehension, Natural Language Understanding, Question Answering, Reading Comprehension

Machine reading comprehension is a key part
of natural language understanding. Due to
wide range of applications, machine reading
comprehension has attracted considerable in-
terest from both commercial and academic en-
tities. Recently, deep neural network based
models have been able to achieve near human
performance on certain types of reading com-
prehension datasets (Rajpurkar et al., 2016;
Devlin et al., 2018; Hermann et al., 2015; Seo
et al., 2016). Newer and much harder datasets
have been proposed that require more sophisti-
cated reasoning capabilities. One such dataset
was proposed by Khashabi et al.(2018) called
MultiRC. In this work, we propose a multi-
stage approach which significantly improves
the previous state of the art results on MultiRC
(Khashabi et al., 2018).
Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
Authors: Anonymous
Date: 2019-06-24 00:00:00
Tasks: Negation, Open-Ended Question Answering, Sentence

Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical knowledge. We explore five experimental methods inspired by prior work evaluating pretrained sentence representation models. We use a single linguistic phenomenon, negative polarity item (NPI) licensing, as a case study for our experiments. NPIs like 'any' are grammatical only if they appear in a licensing environment like negation ('Sue doesn't have any cats' vs. '*Sue has any cats'). This phenomenon is challenging because of the variety of NPI licensing environments that exist. We introduce an artificially generated dataset that manipulates key features of NPI licensing for the experiments. We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods. We conclude that a variety of methods is necessary to reveal all relevant aspects of a model's grammatical knowledge in a given domain.
Text classification can distinguish mainstream and fringe scientific papers
Authors: Anonymous
Date: 2019-06-19 00:00:00
Tasks: Classification, text-classification, Text Classification

In this work, I explore the use of supervised learning in distinguishing mainstream and fringe scientific papers. This work has two goals. The first is to determine whether mainstream and fringe scientific papers can be reliably distinguished through automated means. The second is to determine whether classifiers trained using stylometric features, such as word count, average word and sentences lengths, and frequencies of part-of-speech sequences, can outperform conventional n-gram document models in classifying papers across scientific topics. I conduct a systematic study of the ability of classifiers to distinguish mainstream and fringe scientific papers across topics, for example by training a classifier on biophysics papers and testing it against cosmology papers. The term-based and style-based approaches both perform significantly better than chance, with neither approach consistently outperforming the other. Classifiers trained using the combined feature set (i.e., n-gram frequencies and stylometric features) perform little better than those trained only on one or the other feature set, suggesting that the two feature sets are, in aggregate, highly correlated. Overall, the results of this work suggest that mainstream and fringe scientific papers are readily distinguishable by conventional text classification methods.
Mastering emergent language: learning to guide in simulated navigation
Authors: Anonymous
Date: 2019-06-09 00:00:00
Tasks: Navigate

To cooperate with humans effectively, virtual agents need to be able to understand and execute language instructions. A typical setup to achieve this is with a scripted teacher which guides a virtual agent using language instructions. However, such setup has clear limitations in scalability and, more importantly, it is not interactive. Here, we introduce an autonomous agent that uses discrete communication to interactively guide other agents to navigate and act on a simulated environment. The developed communication protocol is trainable, emergent and requires no additional supervision. The emergent language speeds up learning of new agents, it generalizes across incrementally more difficult tasks and, contrary to most other emergent languages, it is highly interpretable. We demonstrate how the emitted messages correlate with particular actions and observations, and how new agents become less dependent on this guidance as training progresses. By exploiting the correlations identified in our analysis, we manage to successfully address the agents in their own language.
Beyond Lexical: A Semantic Retrieval Framework for Textual Search Engine
Authors: Anonymous
Date: 2019-06-09 00:00:00
Tasks: Retrieval, Semantic Retrieval

Search engine has become a fundamental component in various web and mobile applications. Retrieving relevant documents from the massive datasets is challenging for a search engine system, especially when faced with verbose or tail queries. In this paper, we explore a vector space search framework for document retrieval. Specifically, we trained a deep semantic matching model so that each query and document can be encoded as a low dimensional embedding. Our model was trained based on BERT architecture. We deployed a fast k-nearest-neighbor index service for online serving. Both offline and online metrics demonstrate that our method improved retrieval performance and search quality considerably, particularly for tail queries
Internal and External Pressures on Language Emergence: Least Effort, Object Constancy and Frequency
Authors: Anonymous
Date: 2019-06-07 00:00:00
Tasks: Diagnostic, Position

In previous work, artificial agents were shown to achieve almost perfect accuracy in referential games where they have to communicate to identify images. Nevertheless, the resulting communication protocols rarely display salient features of natural languages, such as compositionality. In this paper, we propose some realistic sources of pressure on communication that avert this outcome. More specifically, we formalise the principle of least effort through an auxiliary objective. Moreover, we explore several game variants, inspired by the principle of object constancy, in which we alter the frequency, position, and luminosity of the objects in the images. We perform an extensive analysis on their effect through compositionality metrics, diagnostic classifiers, and zero-shot evaluation. Our findings reveal that the proposed sources of pressure result in emerging languages with less redundancy, more focus on high-level conceptual information, and better abilities of generalisation. Overall, our contributions reduce the gap between emergent and natural languages.
How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations
Authors: Anonymous
Date: 2019-06-07 00:00:00
Tasks: Question Answering

Bidirectional Encoder Representations from Transformers (BERT) reach state-of-the-art results in a variety of Natural Language Processing tasks. However, understanding of their internal functioning is still insufficient and unsatisfactory. In order to better understand BERT and other Transformer-based models, we present a layer-wise analysis of BERT's hidden states. Unlike previous research, which mainly focuses on explaining Transformer models by their \hbox{attention} weights, we argue that hidden states contain equally valuable information. Specifically, our analysis focuses on models fine-tuned on the task of Question Answering (QA) as an example of a complex downstream task. We inspect how QA models transform token vectors in order to find the correct answer. To this end, we apply a set of general and QA-specific probing tasks that reveal the information stored in each representation layer. Our qualitative analysis of hidden state visualizations provides additional insights into BERT's reasoning process. Our results show that the transformations within BERT go through phases that are related to traditional pipeline tasks. The system can therefore implicitly incorporate task-specific information into its token representations. Furthermore, our analysis reveals that fine-tuning has little impact on the models' semantic abilities and that prediction errors can be recognized in the vector representations of even early layers.
Using Clinical Notes for ICU Management
Authors: Anonymous
Date: 2019-05-31 00:00:00
Tasks: Decompensation, Management, Mortality Prediction, Time Series, Time Series Analysis

Monitoring patients in ICU is a challenging and high-cost task. Hence, predicting the condition of patients during their ICU stay can help provide better acute care and plan the hospital's resources. There has been continuous progress in machine learning research for ICU management, and most of this work has focused on using time series signals recorded by ICU instruments. In our work, we show that adding clinical notes as another modality improves the performance of the model for three benchmark tasks: in-hospital mortality prediction, modeling decompensation, and length of stay forecasting that play an important role in ICU management. While the time-series data is measured at regular intervals, doctor notes are charted at irregular times, making it challenging to model them together. We propose a method to model them jointly, achieving considerable improvement across benchmark tasks over baseline time-series model.
Clinical Text Generation through Leveraging Medical Concept and Relations
Authors: Anonymous
Date: 2019-05-30 00:00:00
Tasks: Text Generation

With a neural sequence generation model, this study aims to develop a method writing patient clinical texts given brief medical history. As a proof-of-a-concept, we have demonstrated that it can be workable to use medical concept embedding in clinical text generation. Our model was based on the Sequence-to-Sequence architecture and trained with a large set of de-identified clinical text data. The quantitative result shows that our concept embedding method decreased the perplexity of the baseline architecture. Also, we discuss the analyzed results from human evaluation performed by medical doctors.
CATS: Customizable Abstractive Topic-based Summarization
Authors: Anonymous
Date: 2019-05-24 00:00:00
Tasks: Abstractive Text Summarization

Neural sequence-to-sequence models are a recently proposed family of approaches used in abstractive summarization of text documents, useful for producing condensed versions of source text narratives without being restricted to using only words from the original text. Despite the advances in abstractive summarization, custom generation of summaries (e.g. towards a user's preference) remains unexplored. In this paper, we present CATS, an abstractive neural summarization model, that summarizes content in a sequence-to-sequence fashion but also introduces a new mechanism to control the underlying latent topic distribution of the produced summaries. Our experimental results on the well-known CNN/DailyMail dataset show that our model achieves state-of-the-art performance.
How to learn from unlabeled volume data: Self-Supervised 3D Context Feature Learning
Authors: Anonymous
Date: 2019-05-23 00:00:00
Tasks: One-Shot Segmentation, Self-Supervised Learning

The vast majority of 3D medical images lacks detailed image-based expert annotations. The ongoing advances of deep convolutional neural networks clearly demonstrate the benefit of supervised learning to successfully extract relevant anatomical information and aid image-based analysis and interventions, but it heavily relies on labeled data. Self-supervised learning, that requires no expert labels, provides an appealing way to discover data-inherent patterns and leverage anatomical information freely available from medical images themselves.
In this work, we propose a new approach to train effective convolutional feature extractors based on a new concept of image-intrinsic spatial offset relations with an auxiliary heatmap regression loss. The learned features successfully capture semantic, anatomical information and enable state-of-the-art accuracy for a k-NN based one-shot segmentation task without any subsequent fine-tuning.
Co-evolution of language and agents in referential games
Authors: Anonymous
Date: 2019-05-23 00:00:00

Referential games offer a grounded learning environment for neural agents which accounts for the fact that language is functionally used to communicate. However, they do not take into account a second constraint considered to be fundamental for the shape of human language: that it must be learnable by new language learners and thus has to overcome a transmission bottleneck.
In this work, we insert such a bottleneck in a referential game, by introducing a changing population of agents in which new agents learn by playing with more experienced agents. We show that mere cultural transmission results in a substantial improvement in language efficiency and communicative success, measured in convergence speed, degree of structure in the emerged languages and within-population consistency of the language. However, as our core contribution, we show that the optimal situation is to co-evolve language and agents. When we allow the agent population to evolve through genotypical evolution, we achieve across the board improvements on all considered metrics. These results stress that for language emergence studies cultural evolution is important, but also the suitability of the architecture itself should be considered.
Pretraining for Conditional Generation with Pseudo Self Attention
Authors: Anonymous
Date: 2019-05-21 00:00:00
Tasks: Conditional Text Generation, Natural Language Understanding, Text Generation, Transfer Learning

Large pretrained language representation models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that finetune a pretrained model. However, such transfer learning approaches have not seen the same success for natural language generation. In this work, we explore transfer learning for conditional generation with large pretrained language models. We propose a simple modification to a pretrained unconditional transformer model to inject arbitrary conditioning into the self attention layer, an approach we call pseudo self attention. Through experiments on four long-form conditional text generation tasks, we show that this technique outperforms strong baselines and other transfer learning approaches, and produces coherent generations.
A Co-guided Neural Network for Person Name Recognition in Academic Homepages
Authors: Anonymous
Date: 2019-05-14 00:00:00
Tasks: NER

Academic homepages are important channels for learning researchers' profiles. Knowing the person names in academic homepages is essential to the extraction of other entities such as contacts, publications, and biography. Traditional NER models are trained on newswire corpora such as CoNLL-2003, which contain well-formed names in consistent and complete syntax. However, academic homepages often contain text with incomplete syntax and names of various forms. Few studies have addressed person name recognition in this context. To fill this gap, we propose a new name annotation scheme which provides detailed name form information including first, middle, or last name, and a full name word or a name initial. To take full advantage of this annotation scheme, we propose a Co-guided Neural Network (CogNN) model that can accurately recognise person names in academic homepages using the fine-grained annotations. CogNN uses co-attention mechanism to co-guide two jointly trained neural networks. Experimental results on real datasets show that CogNN significantly outperforms state-of-the-art NER models in extracting person names from academic homepages, while achieving comparable performance on a traditional NER benchmark dataset.
Models in the Wild: On Corruption Robustness of NLP Systems
Authors: Anonymous
Date: 2019-05-05 00:00:00
Tasks: NER, Sentiment Analysis, Word Embeddings

Natural Language Processing models lack a unified approach to robustness testing. In this paper we introduce WildNLP - a framework for testing model stability in a natural setting where text corruptions such as keyboard errors or misspelling occur. We compare robustness of models from 4 popular NLP tasks: Q&A, NLI, NER and Sentiment Analysis by testing their performance on aspects introduced in the framework. In particular, we focus on a comparison between recent state-of-the- art text representations and non-contextualized word embeddings. In order to improve robust- ness, we perform adversarial training on se- lected aspects and check its transferability to the improvement of models with various cor- ruption types. We find that the high perfor- mance of models does not ensure sufficient robustness, although modern embedding tech- niques help to improve it. We release cor- rupted datasets and code for WildNLP frame- work for the community.
Pretrained Transformers for Simple Question Answering
Authors: Anonymous
Date: 2019-05-02 00:00:00
Tasks: Knowledge Graphs, Prediction, Question Answering, Relation Prediction, Word Embeddings

Answering simple questions over knowledge graphs is a well-studied problem in question answering. Previous approaches for this task built on recurrent and convolutional neural networks (RNNs and CNNs) based architectures that use pretrained word embeddings. It was recently shown that a pretrained transformer network (BERT) can outperform RNN- and CNN based approaches on various natural language processing tasks. In this work, we investigate how well network BERT performs on the entity span prediction and relation prediction subtasks of simple QA. In addition, we provide an evaluation of both BERT and BiLSTM-based models in datasparse scenarios.
Robust Authorship Verification with Transfer Learning
Authors: Anonymous
Date: 2019-02-27 00:00:00
Tasks: Authorship Verification, Generative Adversarial Network, Language Modeling, Language Modelling, Text Augmentation, Transfer Learning

We address the problem of open-set authorship verification, a classification task that consists of attributing texts of unknown authorship to a given author when the unknown documents in the test set are excluded from the training set. We present an end-to-end model-building process that is universally applicable to a wide variety of corpora with little to no modification or fine-tuning. It relies on transfer learning of a deep language model and uses a generative adversarial network and a number of text augmentation techniques to improve the model's generalization ability. The language model encodes documents of known and unknown authorship into a domain-invariant space, aligning document pairs as input to the classifier, while keeping them separate. The resulting embeddings are used to train to an ensemble of recurrent and quasi-recurrent neural networks. The entire pipeline is bidirectional; forward and backward pass results are averaged. We perform experiments on four traditional authorship verification datasets, a collection of machine learning papers mined from the web, and a large Amazon-Reviews dataset. Experimental results surpass baseline and current state-of-the-art techniques, validating the proposed approach.
Integrating and Evaluating Extra-linguistic Context in Neural Machine Translation
Authors: Anonymous
Date: 2019-01-27 00:00:00
Tasks: Machine Translation, Translation

In Machine Translation (MT), taking into account information related to the setting in which a text is produced can be crucial. We investigate the impact of different extra-linguistic factors (speaker gender, speaker age, film genre and film year) on the MT of subtitles. Our starting point is the pseudo-token approach (Sennrich, 2016). We explore the simultaneous addition of multiple factors in various orders in order to assess the limits of treating these factors as a sequence of words. We compare this approach to the encoding of the same factors using an additional, separate encoder. We evaluate both using BLEU and a targeted evaluation of how well the context is used. Our results show that both strategies are well adapted to exploiting such context.  Contrary to our intuitions, the pseudo-token approach appears unperturbed by the use of multiple values in various orders, and also results in significant improvements in BLEU score (p<0.01).  The multi-encoder approach proves more effective at integrating context but results in lower overall BLEU scores.