pwc-archive/papers-with-abstracts huggingface dataset:
Evaluating Textual Representations through Image Generation
Authors: Graham Spinks, Marie-Francine Moens
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Image Generation

We present a methodology for determining the quality of textual representations through the ability to generate images from them. Continuous representations of textual input are ubiquitous in modern Natural Language Processing techniques either at the core of machine learning algorithms or as the by-product at any given layer of a neural network. While current techniques to evaluate such representations focus on their performance on particular tasks, they don{'}t provide a clear understanding of the level of informational detail that is stored within them, especially their ability to represent spatial information. The central premise of this paper is that visual inspection or analysis is the most convenient method to quickly and accurately determine information content. Through the use of text-to-image neural networks, we propose a new technique to compare the quality of textual representations by visualizing their information content. The method is illustrated on a medical dataset where the correct representation of spatial information and shorthands are of particular importance. For four different well-known textual representations, we show with a quantitative analysis that some representations are consistently able to deliver higher quality visualizations of the information content. Additionally, we show that the quantitative analysis technique correlates with the judgment of a human expert evaluator in terms of alignment.
Linguistic representations in multi-task neural networks for ellipsis resolution
Authors: Ola R{\o}nning, Daniel Hardt, Anders S{\o}gaard
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Multi-Task Learning

Sluicing resolution is the task of identifying the antecedent to a question ellipsis. Antecedents are often sentential constituents, and previous work has therefore relied on syntactic parsing, together with complex linguistic features. A recent model instead used partial parsing as an auxiliary task in sequential neural network architectures to inject syntactic information. We explore the linguistic information being brought to bear by such networks, both by defining subsets of the data exhibiting relevant linguistic characteristics, and by examining the internal representations of the network. Both perspectives provide evidence for substantial linguistic knowledge being deployed by the neural networks.
Unsupervised Token-wise Alignment to Improve Interpretation of Encoder-Decoder Models
Authors: Shun Kiyono, Sho Takase, Jun Suzuki, Naoaki Okazaki, Kentaro Inui, Masaaki Nagata
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Decoder, Machine Translation, Sentence, Text Generation, Translation

Developing a method for understanding the inner workings of black-box neural methods is an important research endeavor. Conventionally, many studies have used an attention matrix to interpret how Encoder-Decoder-based models translate a given source sentence to the corresponding target sentence. However, recent studies have empirically revealed that an attention matrix is not optimal for token-wise translation analyses. We propose a method that explicitly models the token-wise alignment between the source and target sequences to provide a better analysis. Experiments show that our method can acquire token-wise alignments that are superior to those of an attention mechanism.
Interpretable Neural Architectures for Attributing an Ad's Performance to its Writing Style
Authors: Reid Pryzant, Sugato Basu, Kazoo Sone
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Interpretable Machine Learning

How much does {``}free shipping!{''} help an advertisement{'}s ability to persuade? This paper presents two methods for \textit{performance attribution}: finding the degree to which an outcome can be attributed to parts of a text while controlling for potential confounders. Both algorithms are based on interpreting the behaviors and parameters of trained neural networks. One method uses a CNN to encode the text, an adversarial objective function to control for confounders, and projects its weights onto its activations to interpret the importance of each phrase towards each output class. The other method leverages residualization to control for confounds and performs interpretation by aggregating over learned word vectors. We demonstrate these algorithms{'} efficacy on 118,000 internet search advertisements and outcomes, finding language indicative of high and low click through rate (CTR) regardless of who the ad is by or what it is for. Our results suggest the proposed algorithms are high performance and data efficient, able to glean actionable insights from fewer than 10,000 data points. We find that quick, easy, and authoritative language is associated with success, while lackluster embellishment is related to failure. These findings agree with the advertising industry{'}s emperical wisdom, automatically revealing insights which previously required manual A/B testing to discover.
`Indicatements' that character language models learn English morpho-syntactic units and regularities
Authors: Yova Kementchedjhieva, Adam Lopez
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Feature Engineering, Language Modeling, Language Modelling, Machine Translation, Morphological Tagging, Speech Recognition

Character language models have access to surface morphological patterns, but it is not clear whether or \textit{how} they learn abstract morphological regularities. We instrument a character language model with several probes, finding that it can develop a specific unit to identify word boundaries and, by extension, morpheme boundaries, which allows it to capture linguistic properties and regularities of these units. Our language model proves surprisingly good at identifying the selectional restrictions of English derivational morphemes, a task that requires both morphological and syntactic awareness. Thus we conclude that, when morphemes overlap extensively with the words of a language, a character language model can perform morphological abstraction.
Introspection for convolutional automatic speech recognition
Authors: Andreas Krug, Sebastian Stober
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), image-classification, Image Classification, Machine Translation, speech-recognition, Speech Recognition

Artificial Neural Networks (ANNs) have experienced great success in the past few years. The increasing complexity of these models leads to less understanding about their decision processes. Therefore, introspection techniques have been proposed, mostly for images as input data. Patterns or relevant regions in images can be intuitively interpreted by a human observer. This is not the case for more complex data like speech recordings. In this work, we investigate the application of common introspection techniques from computer vision to an Automatic Speech Recognition (ASR) task. To this end, we use a model similar to image classification, which predicts letters from spectrograms. We show difficulties in applying image introspection to ASR. To tackle these problems, we propose normalized averaging of aligned inputs (NAvAI): a data-driven method to reveal learned patterns for prediction of specific classes. Our method integrates information from many data examples through local introspection techniques for Convolutional Neural Networks (CNNs). We demonstrate that our method provides better interpretability of letter-specific patterns than existing methods.
What do RNN Language Models Learn about Filler--Gap Dependencies?
Authors: Ethan Wilcox, Roger Levy, Takashi Morita, Richard Futrell
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Language Modeling, Language Modelling, Machine Translation

RNN language models have achieved state-of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn. Here we investigate whether state-of-the-art RNN language models represent long-distance \textbf{filler{--}gap dependencies} and constraints on them. Examining RNN behavior on experimentally controlled sentences designed to expose filler{--}gap dependencies, we show that RNNs can represent the relationship in multiple syntactic positions and over large spans of text. Furthermore, we show that RNNs learn a subset of the known restrictions on filler{--}gap dependencies, known as \textbf{island constraints}: RNNs show evidence for wh-islands, adjunct islands, and complex NP islands. These studies demonstrates that state-of-the-art RNN models are able to learn and generalize about empty syntactic positions.
Closing Brackets with Recurrent Neural Networks
Authors: Natalia Skachkova, Thomas Trost, Dietrich Klakow
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Language Modeling, Language Modelling

Many natural and formal languages contain words or symbols that require a matching counterpart for making an expression well-formed. The combination of opening and closing brackets is a typical example of such a construction. Due to their commonness, the ability to follow such rules is important for language modeling. Currently, recurrent neural networks (RNNs) are extensively used for this task. We investigate whether they are capable of learning the rules of opening and closing brackets by applying them to synthetic Dyck languages that consist of different types of brackets. We provide an analysis of the statistical properties of these languages as a baseline and show strengths and limits of Elman-RNNs, GRUs and LSTMs in experiments on random samples of these languages. In terms of perplexity and prediction accuracy, the RNNs get close to the theoretical baseline in most cases.
Interpreting Word-Level Hidden State Behaviour of Character-Level LSTM Language Models
Authors: Avery Hiebert, Cole Peterson, Alona Fyshe, Nishant Mehta
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Clustering, Language Modeling, Language Modelling, Word Embeddings

While Long Short-Term Memory networks (LSTMs) and other forms of recurrent neural network have been successfully applied to language modeling on a character level, the hidden state dynamics of these models can be difficult to interpret. We investigate the hidden states of such a model by using the HDBSCAN clustering algorithm to identify points in the text at which the hidden state is similar. Focusing on whitespace characters prior to the beginning of a word reveals interpretable clusters that offer insight into how the LSTM may combine contextual and character-level information to identify parts of speech. We also introduce a method for deriving word vectors from the hidden state representation in order to investigate the word-level knowledge of the model. These word vectors encode meaningful semantic information even for words that appear only once in the training text.
Importance of Self-Attention for Sentiment Analysis
Authors: Ga{\"e}l Letarte, Fr{\'e}d{\'e}rik Paradis, Philippe Gigu{\`e}re, Fran{\c{c}}ois Laviolette
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Classification, Decision Making, General Classification, Image Captioning, Language Modeling, Language Modelling, Machine Translation, Sentiment Analysis, text-classification, Text Classification, Topic Classification, Word Embeddings

Despite their superior performance, deep learning models often lack interpretability. In this paper, we explore the modeling of insightful relations between words, in order to understand and enhance predictions. To this effect, we propose the Self-Attention Network (SANet), a flexible and interpretable architecture for text classification. Experiments indicate that gains obtained by self-attention is task-dependent. For instance, experiments on sentiment analysis tasks showed an improvement of around 2{\%} when using self-attention compared to a baseline without attention, while topic classification showed no gain. Interpretability brought forward by our architecture highlighted the importance of neighboring word interactions to extract sentiment.
An Analysis of Encoder Representations in Transformer-Based Machine Translation
Authors: Aless Raganato, ro, J{\"o}rg Tiedemann
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Feature Engineering, Machine Translation, Transfer Learning, Translation

The attention mechanism is a successful technique in modern NLP, especially in tasks like machine translation. The recently proposed network architecture of the \textit{Transformer} is based entirely on attention mechanisms and achieves new state of the art results in neural machine translation, outperforming other sequence-to-sequence models. However, so far not much is known about the internal properties of the model and the representations it learns to achieve that performance. To study this question, we investigate the information that is learned by the attention mechanism in Transformer models with different translation quality. We assess the representations of the encoder by extracting dependency relations based on self-attention weights, we perform four probing tasks to study the amount of syntactic and semantic captured information and we also test attention in a transfer learning scenario. Our analysis sheds light on the relative strengths and weaknesses of the various encoder representations. We observe that specific attention heads mark syntactic dependency relations and we can also confirm that lower layers tend to learn more about syntax while higher layers tend to encode more semantics.
Evaluating Grammaticality in Seq2seq Models with a Broad Coverage HPSG Grammar: A Case Study on Machine Translation
Authors: Johnny Wei, Khiem Pham, Brendan O{'}Connor, Brian Dillon
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Machine Translation, Translation

Sequence to sequence (seq2seq) models are often employed in settings where the target output is natural language. However, the syntactic properties of the language generated from these models are not well understood. We explore whether such output belongs to a formal and realistic grammar, by employing the English Resource Grammar (ERG), a broad coverage, linguistically precise HPSG-based grammar of English. From a French to English parallel corpus, we analyze the parseability and grammatical constructions occurring in output from a seq2seq translation model. Over 93{\%} of the model translations are parseable, suggesting that it learns to generate conforming to a grammar. The model has trouble learning the distribution of rarer syntactic rules, and we pinpoint several constructions that differentiate translations between the references and our model.
How much should you ask? On the question structure in QA systems.
Authors: Barbara Rychalska, Dominika Basaj, Anna Wr{\'o}blewska, Przemyslaw Biecek
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Question Answering, valid

Datasets that boosted state-of-the-art solutions for Question Answering (QA) systems prove that it is possible to ask questions in natural language manner. However, users are still used to query-like systems where they type in keywords to search for answer. In this study we validate which parts of questions are essential for obtaining valid answer. In order to conclude that, we take advantage of LIME - a framework that explains prediction by local approximation. We find that grammar and natural language is disregarded by QA. State-of-the-art model can answer properly even if {'}asked{'} only with a few words with high coefficients calculated with LIME. According to our knowledge, it is the first time that QA model is being explained by LIME.
Language Models Learn POS First
Authors: Naomi Saphra, Adam Lopez
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Language Modeling, Language Modelling, Part-Of-Speech Tagging, POS

A glut of recent research shows that language models capture linguistic structure. Such work answers the question of whether a model represents linguistic structure. But how and when are these structures acquired? Rather than treating the training process itself as a black box, we investigate how representations of linguistic structure are learned over time. In particular, we demonstrate that different aspects of linguistic structure are learned at different rates, with part of speech tagging acquired early and global topic information learned continuously.
Probing sentence embeddings for structure-dependent tense
Authors: Geoff Bacon, Terry Regier
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Sentence, Sentence Embeddings

Learning universal sentence representations which accurately model sentential semantic content is a current goal of natural language processing research. A prominent and successful approach is to train recurrent neural networks (RNNs) to encode sentences into fixed length vectors. Many core linguistic phenomena that one would like to model in universal sentence representations depend on syntactic structure. Despite the fact that RNNs do not have explicit syntactic structural representations, there is some evidence that RNNs can approximate such structure-dependent phenomena under certain conditions, in addition to their widespread success in practical tasks. In this work, we assess RNNs{'} ability to learn the structure-dependent phenomenon of main clause tense.
Interpretable Word Embedding Contextualization
Authors: Kyoung-Rok Jang, Sung-Hyon Myaeng, Sang-Bum Kim
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Word Embeddings

In this paper, we propose a method of calibrating a word embedding, so that the semantic it conveys becomes more relevant to the context. Our method is novel because the output shows clearly which senses that were originally presented in a target word embedding become stronger or weaker. This is possible by utilizing the technique of using sparse coding to recover senses that comprises a word embedding.
Extracting Syntactic Trees from Transformer Encoder Self-Attentions
Authors: David Mare{\v{c}}ek, Rudolf Rosa
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Machine Translation, Sentence

This is a work in progress about extracting the sentence tree structures from the encoder{'}s self-attention weights, when translating into another language using the Transformer neural network architecture. We visualize the structures and discuss their characteristics with respect to the existing syntactic theories and annotations.
Portable, layer-wise task performance monitoring for NLP models
Authors: Tom Lippincott
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Machine Translation, Sentence, Sentence Embeddings

There is a long-standing interest in understanding the internal behavior of neural networks. Deep neural architectures for natural language processing (NLP) are often accompanied by explanations for their effectiveness, from general observations (e.g. RNNs can represent unbounded dependencies in a sequence) to specific arguments about linguistic phenomena (early layers encode lexical information, deeper layers syntactic). The recent ascendancy of DNNs is fueling efforts in the NLP community to explore these claims. Previous work has tended to focus on easily-accessible representations like word or sentence embeddings, with deeper structure requiring more ad hoc methods to extract and examine. In this work, we introduce Vivisect, a toolkit that aims at a general solution for broad and fine-grained monitoring in the major DNN frameworks, with minimal change to research patterns.
Explicitly modeling case improves neural dependency parsing
Authors: Clara Vania, Adam Lopez
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Dependency Parsing, Multi-Task Learning

Neural dependency parsing models that compose word representations from characters can presumably exploit morphosyntax when making attachment decisions. How much do they know about morphology? We investigate how well they handle morphological case, which is important for parsing. Our experiments on Czech, German and Russian suggest that adding explicit morphological case{---}either oracle or predicted{---}improves neural dependency parsing, indicating that the learned representations in these models do not fully encode the morphological knowledge that they need, and can still benefit from targeted forms of explicit linguistic modeling.
Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis
Authors: Kelly Zhang, Samuel Bowman
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: CCG Supertagging, Language Modeling, Language Modelling, Machine Translation, Part-Of-Speech Tagging, Transfer Learning, Translation

Recently, researchers have found that deep LSTMs trained on tasks like machine translation learn substantial syntactic and semantic information about their input sentences, including part-of-speech. These findings begin to shed light on why pretrained representations, like ELMo and CoVe, are so beneficial for neural language understanding models. We still, though, do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn. With this in mind, we compare four objectives{---}language modeling, translation, skip-thought, and autoencoding{---}on their ability to induce syntactic and part-of-speech information, holding constant the quantity and genre of the training data, as well as the LSTM architecture.
Representation of Word Meaning in the Intermediate Projection Layer of a Neural Language Model
Authors: Steven Derby, Paul Miller, Brian Murphy, Barry Devereux
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11
Tasks: Language Modeling, Language Modelling, Time Series Analysis

Performance in language modelling has been significantly improved by training recurrent neural networks on large corpora. This progress has come at the cost of interpretability and an understanding of how these architectures function, making principled development of better language models more difficult. We look inside a state-of-the-art neural language model to analyse how this model represents high-level lexico-semantic information. In particular, we investigate how the model represents words by extracting activation patterns where they occur in the text, and compare these representations directly to human semantic knowledge.
Interpretable Structure Induction via Sparse Attention
Authors: Ben Peters, Vlad Niculae, Andr{\'e} F. T. Martins
Date: 2018-11-01 00:00:00
Proceeding: WS 2018 11

Neural network methods are experiencing wide adoption in NLP, thanks to their empirical performance on many tasks. Modern neural architectures go way beyond simple feedforward and recurrent models: they are complex pipelines that perform soft, differentiable computation instead of discrete logic. The price of such soft computing is the introduction of dense dependencies, which make it hard to disentangle the patterns that trigger a prediction. Our recent work on sparse and structured latent computation presents a promising avenue for enhancing interpretability of such neural pipelines. Through this extended abstract, we aim to discuss and explore the potential and impact of our methods.