pwc-archive/papers-with-abstracts huggingface dataset:
Multi-view Subword Regularization
Authors: Anonymous
Date: 2020-11-28 00:00:00
Tasks: Cross-Lingual Transfer, Segmentation

Multilingual pretrained representations generally rely on subword segmentation algorithms to create a shared multilingual vocabulary. However, standard heuristic algorithms often lead to sub-optimal segmentation, especially for languages with limited amounts of data. In this paper, we take two major steps towards alleviating this problem. First, we demonstrate empirically that applying existing subword regularization methods (Kudo, 2018; Provilkov et al., 2020) during fine-tuning of pre-trained multilingual representations improves the effectiveness of cross-lingual transfer. Second, to take full advantage of different possible input segmentations, we propose Multi-view Subword Regularization (MVR), a method that enforces the consistency between predictions of using inputs tokenized by the standard and probabilistic segmentations. Results on the XTREME multilingual benchmark (Hu et al., 2020) show that MVR brings consistent improvements of up to 2.5 points over using standard segmentation algorithms.
AMR Parsing with Action-Pointer Transformer
Authors: Anonymous
Date: 2020-11-24 00:00:00
Tasks: Abstract Meaning Representation, AMR Parsing, Hard Attention, Sentence

Abstract Meaning Representation parsing belongs to a category of sentence-to-graph prediction tasks where the target graph is not explicitly linked to the sentence tokens. However, nodes or subgraphs are semantically related to subsets of the sentence tokens, and locality between words and related nodes is often preserved. Transition-based approaches have recently shown great progress in capturing these inductive biases but still suffer from limited expressiveness. In this work we propose a transition-based system that combines hard-attention over sentences with a target-side action pointer mechanism to decouple source tokens from node representations. We model the transitions as well as the pointer mechanism using a single Transformer model. Parser state and graph structure information is efficiently encoded using attention heads. We show that our approach leads to increased expressiveness while capitalizing inductive biases and attains new state-of-the Smatch scores on AMR 1.0 (78.5) and AMR 2.0 (81.8).
Increasing Robustness to Spurious Correlations using Forgettable Examples
Authors: Anonymous
Date: 2020-11-16 00:00:00
Tasks: Out-of-Distribution Generalization, QQP

Neural NLP models tend to rely on spurious correlations between labels and input features to perform their tasks. Minority examples, i.e., examples that contradict the spurious correlations present in the majority of data points, have been shown to increase the out-ofdistribution generalization of pre-trained language models. In this paper, we first propose using example forgetting to find minority examples without prior knowledge of the spurious correlations present in the dataset. Forgettable examples are instances either learned and then forgotten during training or never learned. We show empirically how these examples are related to minorities in our training sets. Then, we introduce a new approach to robustify models by fine-tuning our models twice, first on the full training data and second on the minorities only. We obtain substantial improvements in out-of-distribution generalization when applying our approach to the MNLI, QQP and FEVER datasets.
Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Approaches
Authors: Keerthiram Murugesan, Mattia Atzeni, Pavan Kapanipathi, Pushkar Shukla, Sadhana Kumaravel, Gerald Tesauro, Kartik Talamadupula, Mrinmaya Sachan, Murray Campbell
Date: 2020-07-12 00:00:00
Tasks: Decision Making, Reinforcement Learning (RL), Sequential Decision Making, text-based games

Text-based games have emerged as an important test-bed for Reinforcement Learning (RL) research, requiring RL agents to combine grounded language understanding with sequential decision making. In this paper, we examine the problem of infusing RL agents with commonsense knowledge. This allows agents to efficiently act in the world by pruning out implausible actions, and to perform look-ahead planning to determine how current actions might affect future world states. We design a new text-based gaming environment called TextWorld Commonsense (TWC) for training and evaluating RL agents with a specific kind of commonsense knowledge about objects, their attributes, and affordances.  We introduce a number of RL agents that combine the sequential context with a dynamic graph representation of their beliefs of the world and commonsense knowledge from ConceptNet in different ways. We show that our agents act efficiently (fewer moves) and achieve better scores, and that learned policies can be transferred to other instances in TWC.
Biomedical Network Link Prediction using Neural Network Graph Embedding
Authors: Sumit Kumar, Raj Ratn Pranesh, Ambesh Shekhar
Date: 2020-10-19 00:00:00
Tasks: Classification, Graph Embedding, Link Prediction, Prediction

In this paper, we aim at Graph embedding learning for automatic grasping of low-dimensional node representation on biomedical networks. The purpose is to use different neural Graph embedding methods for conducting analysis on 3 major biomedical link prediction tasks: drug-disease association (DDA) prediction, drug-drug interactions (DDI) classification, and protein-protein interaction (PPI)classification. We observe that the graph embedding method achieve a promising result without the use of any biological features.
M2D: A Multi-modal Framework for Automatic Medical Diagnosis
Authors: Raj Ratn Pranesh, Ambesh Shekhar, Sumit Kumar
Date: 2020-10-19 00:00:00
Tasks: Language Modeling, Language Modelling, Medical Diagnosis, Multimodal Deep Learning, Transfer Learning

In this paper, we present M2D: a multimodal deep learning framework for automatic medical condition diagnosis via transfer learning. M2D leverages acoustic and textual features extracted from the audio utterance and the corresponding transcription describing a patient’s medical symptoms. Our model utilizes ResNet-34 to learn audio feature via log mel-spectrogram and BioBERT language model to learn textual feature. We conducted a comparative performance analysis of M2D with baseline models based on textual or acoustic feature.
Improving Neural Text Summarization using Knowledge Graphs
Authors: Ambesh Shekhar, Raj Ratn Pranesh, Sumit Kumar
Date: 2020-10-24 00:00:00
Tasks: Extractive Text Summarization, Knowledge Graphs, Text Summarization

In this paper, we propose a method for extractive text summarization using auto-regressive transformers. For better learning procedure we adopt the knowledge graph method to convert our textual data to more informative text and unsupervised training methods for wide use. We feed the informative text to our pre-trained generative model to summarize the text more properly and infer on generating a proper summary. The model is able to summarize input text into adequate information and is capable of performing several Natural Language Processing(NLP) tasks.
RMHIDD: A Reddit Mental Health Intervention Dialogue Dataset
Authors: Anonymous
Date: 2020-10-24 00:00:00
Tasks: Dialogue Generation, Machine Translation

In modern human society, mental health is one of the most critical concern. Over the past many years, a large proportion of the population has been affected by serious mental disorders. People with mental illness require effective mental health intervention and treatment as early as possible to decrease the chances of any further mental defilement. In this paper, we present RMHIDD, a new dialogue corpus for automated mental health intervention. The dataset is consists of over 200K Reddit posts collected from 18 different sub-Reddit groups with each post consisting of a sequential conversation between the users. On this dataset,
we also trained various models for dialogue generation task, namely-’Seq2Seq’, ’BART’ and ’DialoGPT’. In our analysis, we found that the BART model outperformed other models with a higher Perplexity score of 19.7. We also found that the DialoGPT model surpasses other models on various machine translation evaluation metrics. The results generated from various language models were promising and showed the possibility of building automated mental health intervention.
Pretrain-KGEs: Learning Knowledge Representation from Pretrained Models for Knowledge Graph Embeddings
Authors: Zhiyuan Zhang, Xiaoqian Liu, Yi Zhang, Qi Su, Xu sun, Bin He
Date: 2019-12-01 00:00:00
Tasks: Knowledge Graph Completion, Knowledge Graph Embeddings, Knowledge Graphs, Link Prediction, Relation, Triplet

Learning knowledge graph embeddings (KGEs) is an efficient approach to knowledge graph completion. Conventional KGEs often suffer from limited knowledge representation, which causes less accuracy especially when training on sparse knowledge graphs. To remedy this, we present Pretrain-KGEs, a training framework for learning better knowledgeable entity and relation embeddings, leveraging the abundant linguistic knowledge from pretrained language models. Specifically, we propose a unified approach in which we first learn entity and relation representations via pretrained language models and use the representations to initialize entity and relation embeddings for training KGE models. Our proposed method is model agnostic in the sense that it can be applied to any variant of KGE models. Experimental results show that our method can consistently improve results and achieve state-of-the-art performance using different KGE models such as TransE and QuatE, across four benchmark KG datasets in link prediction and triplet classification tasks.
Towards Automatic Sentiment-based Topic Phrase Generation
Authors: Raj Ratn Pranesh, Ambesh Shekhar, Sumit Kumar
Date: 2020-10-18 00:00:00
Tasks: Decoder, Language Modeling, Language Modelling

For obtaining a comprehensive understanding and knowledge of customers’ expectations and demands, analysis of user-generated online product and service reviews is very important. Utilizing crucial insights from customers' feedback for developing business strategies can significantly improve product quality.
In this paper, we have proposed a pretrained language model based encoder-decoder framework which generates a topic that describes the concise meaning of a customer's review text based on its corresponding polarity. We performed a comparative performance analysis of three language models, namely, BERT, ALBERT and GPT2 for the topic generation task on a dataset containing 8,124 customer reviewers along with topics and it's associates sentiment. In our experiment, we found that GPT2 model outperformed the other two models by achieving lower perplexity of 1.82.
Towards Automatic Online Hate Speech Intervention Generation using Pretrained Language Model
Authors: Raj Ratn Pranesh, Ambesh Shekhar, Anish Kumar
Date: 2020-10-19 00:00:00
Tasks: Dialogue Generation, Language Modeling, Language Modelling, Text Generation

Social media harbours substantial toxic and hateful conversations today. Curbing them has emerged as a critical challenge for governments and organizations globally. Prior research has primarily concentrated on the detection of online hate speech while ignoring further action needed to discourage individuals from using hate speech in the future. Counterspeech is an effective way to tackle online hate, leaving freedom of speech untouched. The focus is to directly intervene in the conversation with textual responses that counter the hate content and prevent it from further spreading. In this paper, we propose a novel natural language generation task for hate speech intervention, where the goal is to automatically generate responses to intervene during online conversations that contain hate speech. We sequentially analyzed the performance and capability of various state-of-the-art pretrained language models dialogue generation model for automated hate speech intervention system using automatic metric and
manual human evaluation. The results indicate that the generated intervention responses are very promising in terms of relevance and contextual meaning
S_Covid: An Engine to Explore COVID-19 Scientific Literature
Authors: Mehrdad Farokhnejad, Raj Ratn Pranesh, Genoveva Vargas-Solar, Davoud Amiri Mehr
Date: 2020-10-21 00:00:00
Tasks: Information Retrieval, Question Answering, Retrieval

This paper introduces S_Covid, an end-to-end unsupervised learning based question-answering engine for exploring COVID-19 scientific literature collections. S_Covid enables documents exploration for finding relevant research literature that most possibly
contains information that can answer a user query. Thus, S_Covid pinpoints sentences out of research papers that can be
possible answers to complex COVID-19 related user queries. We conducted experiments on 80,000 COVID-19 related papers collection. The paper shows statistically how the model performs but also through the feedback of real users. It also compares
S_Covid with existing search engines addressing information retrieval of COVID-19 scientific literature.
QuesBELM: A BERT based Ensemble Language Model for Natural Questions
Authors: Raj Ratn Pranesh, Ambesh Shekhar, Smita Pallavi
Date: 2020-10-18 00:00:00
Tasks: Language Modeling, Language Modelling, Natural Questions, Question Answering

A core goal in artificial intelligence is to build systems that can read the web and then answer complex questions related to random searches about any topic. These question-answering (QA) systems could have a big impact on the way that we access information. In this paper, we addressed the task of question-answering (QA) systems on Google’s Natural Questions (NQ) dataset containing real user questions issued to Google search and the answers found from Wikipedia by annotators. 
In our work, we systematically compare the performance of powerful variant models of Transformer architectures- ’BERTbase, BERT large-WWM and ALBERT-XXL’ over Natural Questions dataset. We also propose a state-of-the-art BERT based ensemble language model- QuesBELM. QuesBELM leverages the power of existing BERT variants combined together to build a more accurate stacking ensemble model for question answering (QA) system. The model integrates top-K predictions from single language models to determine the best answer out of all. Our model surpassed the baseline language models with the Harmonic mean score of 0.731 and 0.582 for the long answer(LA) and short answer(SA) tasks respectively, reporting an average of 10% improvement over the baseline models.
Unified Embedding for Universal Multilingual Neural Machine Translation
Authors: Anonymous
Date: 2020-10-16 00:00:00
Tasks: Decoder, Machine Translation, Translation

In this paper, we propose a novel neural machine translation method that enables translation of multiple languages using a single universal model. Our approach introduces a small shared character space across languages and a language-specific linear transformation after the decoder output. Through mapping each token of language difference to a fixed-length sequence of characters and then building corresponding embeddings, our approach enables a subword system using a small shared pseudo vocabulary. A dictionary-specific inference is represented by a specific component to project the decoder output onto a language-specific token, which eliminates the effect of the off-target problem described in many previous studies, and improves the language discrimination ability of the model. Experiments on the International Workshop on Spoken Language Translation dataset showed that our approach achieves improvement of 3-4 BLEU for zero-shot translation, and without lag on supervised translation under separate byte-pair encoding. Statistical analysis indicated that, with our model, there is no off-target problem for low-resource multilingual translation. Our experiments demonstrate the feasibility of sharing vocabulary about characters rather than at the word/subword or character/byte levels mentioned in previous studies.
Weight Squeezing: Reparameterization for Compression and Fast Inference
Authors: Anonymous
Date: 2020-05-30 00:00:00
Tasks: Knowledge Distillation, Model Compression, text-classification, Text Classification, Transfer Learning

In this work, we present a novel approach for simultaneous knowledge transfer and model compression called Weight Squeezing. With this method, we perform knowledge transfer from a pre-trained teacher model by learning the mapping from its weights to smaller student model weights, without significant loss of model accuracy.

We applied Weight Squeezing combined with Knowledge Distillation to a pre-trained text classification model, and compared it to various knowledge transfer and model compression methods on several downstream text classification tasks. We observed that our approach produces better results than Knowledge Distillation methods without any loss in inference speed. We also compared Weight Squeezing with Low Rank Factorization methods and observed that our method is significantly faster at inference while being competitive in terms of accuracy.
Spatially Structured Recurrent Modules
Authors: Anonymous
Date: 2020-09-28 00:00:00
Tasks: Starcraft II, Video Prediction

Capturing the structure of a data-generating process by means of appropriate inductive biases can help in learning models that generalize well and are robust to changes in the input distribution. While methods that harness spatial and temporal structures find broad application, recent work has demonstrated the potential of models that leverage sparse and modular structure using an ensemble of sparingly interacting modules. In this work, we take a step towards dynamic models that are capable of simultaneously exploiting both modular and spatiotemporal structures. We accomplish this by abstracting the modeled dynamical system as a collection of autonomous but sparsely interacting sub-systems. The sub-systems interact according to a topology that is learned, but also informed by the spatial structure of the underlying real-world system. This results in a class of models that are well suited for modeling the dynamics of systems that only offer local views into their state, along with corresponding spatial locations of those views. On the tasks of video prediction from cropped frames and multi-agent world modeling from partial observations in the challenging Starcraft2 domain, we find our models to be more robust to the number of available views and better capable of generalization to novel tasks without additional training, even when compared against strong baselines that perform equally well or better on the training distribution.
Population-Guided Imitation Learning
Authors: Anonymous
Date: 2020-09-27 00:00:00
Tasks: Atari Games, Imitation Learning, MuJoCo

Learning to imitate expert behavior is a challenging problem, especially in environments with high-dimensional, continuous observations and unknown dynamics. It includes imitation learning from demonstrations (ILfD) and imitation learning from observations (ILfO). The simplest methods in imitation learning are behavior cloning (BC) and behavior cloning from observations (BCO), for ILfD and ILfO respectively. But BC suffers from the problem of distribution shift, while in ILfO, the inverse dynamic model heavily depends on the current policy, without sufficient generalization to the expert state distribution. Since there is no easily-specified reward function available, exploration is more important for imitation learning than regular RL. In this paper, and we propose population-based exploration techniques for imitation learning, which are simple to implement and improve sample efficiency significantly. And we find that population-based exploration can have much more performance improvement than that in regular RL problems. In ILfD, to enlarge the overall search region, we propose to use Stein Variational Gradient Descent (SVGD) to generate the multiple policies, and attenuate distribution shift by RL with intrinsic rewards. In ILfO, additionally, in order to produce more diverse state-action pairs to make the inverse dynamic model generalize better, we introduce neuro-evolution (NE) to further augment the exploration capability of learning policies. We find these population-based exploration techniques can have more performance improvement in imitation learning, than regular RL problems. Here the intrinsic rewards are simply generated by random network distillation (RND), trained over expert states. The proposed frameworks provide the imitation agent both the intrinsic intention of the demonstrator and better exploration ability, which is critical for the agent to outperform the demonstrator. With experiments of ILfD and ILfO over various difficult Atari games and MuJoCo environments, the proposed exploration-augmented methods show significant performance improvement, especially achieving 5X better sampling efficiency, compared with previous popular imitation learning methods.
Adversarial Stein Training for Graph Energy Models
Authors: Anonymous
Date: 2020-09-21 00:00:00
Tasks: Graph Generation

Learning distributions over graph-structured data is a challenging task. In this work we present an energy-based model (EBM) using graph neural networks (GNN) to learn permutation invariant unnormalized density functions on graphs. The model is trained via minimizing adversarial stein discrepancy. Samples from the model can be obtained via Langevin dynamics. We find that this approach achieves better results on graph generation compared to benchmark models.
Understanding Attention Training via Output Relevance
Authors: Charlie Snell, Ruiqi Zhong, Jacob Steinhardt, Dan Klein
Date: 2020-08-16 00:00:00
Tasks: Translation

In recurrent models with attention, the learned attention weights sometimes correlate with individual token importance, even though the training objective does not explicitly reward this. To understand why, we study the training dynamics of attention for sequence classification and translation. We identify a quantity in the model, which we call the \emph{output relevance}, and show that it drives the learning of the attention. If we ablate attention by fixing it to uniform, the output relevance still correlates with the attention of a normally trained model; but if we instead ablate output relevance, attention cannot be learned. Using output relevance, we explain why attention correlates with gradient-based interpretation; and perhaps surprisingly, a Seq2Seq with attention model sometimes fails to learn a simple permutation copying task. Finally, we discuss evidence that multi-head attention improves not only expressiveness but also learning dynamics.
Examining the rhetorical capacities of neural language models
Authors: Anonymous
Date: 2020-09-01 00:00:00
Tasks: Philosophy

Recently, neural language models (LMs) have demonstrated impressive abilities in generating high-quality discourse. While many recent papers have analyzed the syntactic aspects encoded in LMs, there has been no analysis to date of the inter-sentential, rhetorical knowledge. 
In this paper, we propose a method that quantitatively evaluates the rhetorical  capacities of neural LMs. We examine the capacities of neural LMs understanding the rhetoric of discourse by evaluating their abilities to encode a set of linguistic features derived from Rhetorical Structure Theory (RST). Our experiments show that BERT-based LMs outperform other Transformer LMs, revealing the richer discourse knowledge in their intermediate layer representations. In addition, GPT-2 and XLNet apparently encode less rhetorical knowledge, and we suggest an explanation drawing from linguistic philosophy. Our method shows an avenue towards quantifying the rhetorical capacities of neural LMs.