pwc-archive/papers-with-abstracts huggingface dataset:
TechNet: Technology Semantic Network Based on Patent Data
Authors: Serhad Sarica, Jianxi Luo, Kristin L. Wood
Date: 2019-06-02 00:00:00
Tasks: Knowledge Graphs, Retrieval, Semantic Textual Similarity

The growing developments in general semantic networks, knowledge graphs and ontology databases have motivated us to build a large-scale comprehensive semantic network of technology-related data for engineering knowledge discovery, technology search and retrieval, and artificial intelligence for engineering design and innovation. Specially, we constructed a technology semantic network (TechNet) that covers the elemental concepts in all domains of technology and their semantic associations by mining the complete U.S. patent database from 1976. To derive the TechNet, natural language processing techniques were utilized to extract terms from massive patent texts and recent word embedding algorithms were employed to vectorize such terms and establish their semantic relationships. We report and evaluate the TechNet for retrieving terms and their pairwise relevance that is meaningful from a technology and engineering design perspective. The TechNet may serve as an infrastructure to support a wide range of applications, e.g., technical text summaries, search query predictions, relational knowledge discovery, and design ideation support, in the context of engineering and technology, and complement or enrich existing semantic databases. To enable such applications, the TechNet is made public via an online interface and APIs for public users to retrieve technology-related terms and their relevancies.
Mining Data from the Congressional Record
Authors: Zhengyu Ma, Tianjiao Qi, James Route, Amir Ziai
Date: 2019-06-03 00:00:00

We propose a data storage and analysis method for using the US Congressional record as a policy analysis tool. We use Amazon Web Services and the Solr search engine to store and process Congressional record data from 1789 to the present, and then query Solr to find how frequently language related to tax increases and decreases appears. This frequency data is compared to six economic indicators. Our preliminary results indicate potential relationships between incidence of tax discussion and multiple indicators. We present our data storage and analysis procedures, as well as results from comparisons to all six indicators.
Unsupervised Neural Generative Semantic Hashing
Authors: Casper Hansen, Christian Hansen, Jakob Grue Simonsen, Stephen Alstrup, Christina Lioma
Date: 2019-06-03 00:00:00
Tasks: Code Generation, Document Ranking, Information Retrieval, Retrieval

Fast similarity search is a key component in large-scale information retrieval, where semantic hashing has become a popular strategy for representing documents as binary hash codes. Recent advances in this area have been obtained through neural network based models: generative models trained by learning to reconstruct the original documents. We present a novel unsupervised generative semantic hashing approach, \textit{Ranking based Semantic Hashing} (RBSH) that consists of both a variational and a ranking based component. Similarly to variational autoencoders, the variational component is trained to reconstruct the original document conditioned on its generated hash code, and as in prior work, it only considers documents individually. The ranking component solves this limitation by incorporating inter-document similarity into the hash code generation, modelling document ranking through a hinge loss. To circumvent the need for labelled data to compute the hinge loss, we use a weak labeller and thus keep the approach fully unsupervised. Extensive experimental evaluation on four publicly available datasets against traditional baselines and recent state-of-the-art methods for semantic hashing shows that RBSH significantly outperforms all other methods across all evaluated hash code lengths. In fact, RBSH hash codes are able to perform similarly to state-of-the-art hash codes while using 2-4x fewer bits.
Contextually Propagated Term Weights for Document Representation
Authors: Casper Hansen, Christian Hansen, Stephen Alstrup, Jakob Grue Simonsen, Christina Lioma
Date: 2019-06-03 00:00:00
Tasks: Word Embeddings

Word embeddings predict a word from its neighbours by learning small, dense embedding vectors. In practice, this prediction corresponds to a semantic score given to the predicted word (or term weight). We present a novel model that, given a target word, redistributes part of that word's weight (that has been computed with word embeddings) across words occurring in similar contexts as the target word. Thus, our model aims to simulate how semantic meaning is shared by words occurring in similar contexts, which is incorporated into bag-of-words document representations. Experimental evaluation in an unsupervised setting against 8 state of the art baselines shows that our model yields the best micro and macro F1 scores across datasets of increasing difficulty.
Deep Learning Recommendation Model for Personalization and Recommendation Systems
Authors: Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, Misha Smelyanskiy
Date: 2019-05-31 00:00:00
Tasks: Deep Learning, Recommendation Systems

With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks. These networks differ significantly from other deep learning networks due to their need to handle categorical features and are not well studied or understood. In this paper, we develop a state-of-the-art deep learning recommendation model (DLRM) and provide its implementation in both PyTorch and Caffe2 frameworks. In addition, we design a specialized parallelization scheme utilizing model parallelism on the embedding tables to mitigate memory constraints while exploiting data parallelism to scale-out compute from the fully-connected layers. We compare DLRM against existing recommendation models and characterize its performance on the Big Basin AI platform, demonstrating its usefulness as a benchmark for future algorithmic experimentation and system co-design.
Emotional Embeddings: Refining Word Embeddings to Capture Emotional Content of Words
Authors: Armin Seyeditabari, Narges Tabari, Shafie Gholizade, Wlodek Zadrozny
Date: 2019-05-31 00:00:00
Tasks: Word Embeddings

Word embeddings are one of the most useful tools in any modern natural language processing expert's toolkit. They contain various types of information about each word which makes them the best way to represent the terms in any NLP task. But there are some types of information that cannot be learned by these models. Emotional information of words are one of those. In this paper, we present an approach to incorporate emotional information of words into these models. We accomplish this by adding a secondary training stage which uses an emotional lexicon and a psychological model of basic emotions. We show that fitting an emotional model into pre-trained word vectors can increase the performance of these models in emotional similarity metrics. Retrained models perform better than their original counterparts from 13% improvement for Word2Vec model, to 29% for GloVe vectors. This is the first such model presented in the literature, and although preliminary, these emotion sensitive models can open the way to increase performance in variety of emotion detection techniques.
Sparse Bayesian Learning Approach for Discrete Signal Reconstruction
Authors: Jisheng Dai, An Liu, Hing Cheung So
Date: 2019-06-01 00:00:00
Tasks: Bayesian Inference

This study addresses the problem of discrete signal reconstruction from the perspective of sparse Bayesian learning (SBL). Generally, it is intractable to perform the Bayesian inference with the ideal discretization prior under the SBL framework. To overcome this challenge, we introduce a novel discretization enforcing prior to exploit the knowledge of the discrete nature of the signal-of-interest. By integrating the discretization enforcing prior into the SBL framework and applying the variational Bayesian inference (VBI) methodology, we devise an alternating optimization algorithm to jointly characterize the finite-alphabet feature and reconstruct the unknown signal. When the measurement matrix is i.i.d. Gaussian per component, we further embed the generalized approximate message passing (GAMP) into the VBI-based method, so as to directly adopt the ideal prior and significantly reduce the computational burden. Simulation results demonstrate substantial performance improvement of the two proposed methods over existing schemes. Moreover, the GAMP-based variant outperforms the VBI-based method with i.i.d. Gaussian measurement matrices but it fails to work for non i.i.d. Gaussian matrices.
Learning Semantic Annotations for Tabular Data
Authors: Jiaoyan Chen, Ernesto Jimenez-Ruiz, Ian Horrocks, Charles Sutton
Date: 2019-05-30 00:00:00
Tasks: Column Type Annotation, Prediction, Table annotation, Type prediction

The usefulness of tabular data such as web tables critically depends on understanding their semantics. This study focuses on column type prediction for tables without any meta data. Unlike traditional lexical matching-based methods, we propose a deep prediction model that can fully exploit a table's contextual semantics, including table locality features learned by a Hybrid Neural Network (HNN), and inter-column semantics features learned by a knowledge base (KB) lookup and query answering algorithm.It exhibits good performance not only on individual table sets, but also when transferring from one table set to another.
Training Detection-Range-Frugal Cooperative Collision Avoidance Models for Quadcopters via Neuroevolution
Authors: Amir Behjat, Krushang Gabani, Souma Chowdhury
Date: 2019-05-31 00:00:00
Tasks: Collision Avoidance, Evolutionary Algorithms

Cooperative autonomous approaches to avoiding collisions among small Unmanned Aerial Vehicles (UAVs) is central to safe integration of UAVs within the civilian airspace. One potential online cooperative approach is the concept of reciprocal actions, where both UAVs take pre-trained mutually coherent actions that do not require active online coordination (thereby avoiding the computational burden and risk associated with it). This paper presents a learning based approach to train such reciprocal maneuvers. Neuroevolution, which uses evolutionary algorithms to simultaneously optimize the topology and weights of neural networks, is used as the learning method -- which operates over a set of sample approach scenarios. Unlike most existing work (that minimize travel distance, energy or risk), the training process here focuses on the objective of minimizing the required detection range; this has important practical implications w.r.t. alleviating the dependency on sophisticated sensing and their reliability under various environments. A specialized design of experiments and line search is used to identify the minimum detection range for each sample scenarios. In order to allow an efficient training process, a classifier is used to discard actions (without simulating them) where the controller would fail. The model obtained via neuroevolution is observed to generalize well to (i.e., successful collision avoidance over) unseen approach scenarios.
Multi-Objective Pruning for CNNs Using Genetic Algorithm
Authors: Chuanguang Yang, Zhulin An, Chao Li, Boyu Diao, Yongjun Xu
Date: 2019-06-02 00:00:00

In this work, we propose a heuristic genetic algorithm (GA) for pruning convolutional neural networks (CNNs) according to the multi-objective trade-off among error, computation and sparsity. In our experiments, we apply our approach to prune pre-trained LeNet across the MNIST dataset, which reduces 95.42% parameter size and achieves 16$\times$ speedups of convolutional layer computation with tiny accuracy loss by laying emphasis on sparsity and computation, respectively. Our empirical study suggests that GA is an alternative pruning approach for obtaining a competitive compression performance. Additionally, compared with state-of-the-art approaches, GA is capable of automatically pruning CNNs based on the multi-objective importance by a pre-defined fitness function.
SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikes
Authors: Johannes Christian Thiele, Olivier Bichler, Antoine Dupret
Date: 2019-06-03 00:00:00
Proceeding: ICLR 2020 1

Event-based neuromorphic systems promise to reduce the energy consumption of deep learning tasks by replacing expensive floating point operations on dense matrices by low power sparse and asynchronous operations on spike events. While these systems can be trained increasingly well using approximations of the back-propagation algorithm, these implementations usually require high precision errors for training and are therefore incompatible with the typical communication infrastructure of neuromorphic circuits. In this work, we analyze how the gradient can be discretized into spike events when training a spiking neural network. To accelerate our simulation, we show that using a special implementation of the integrate-and-fire neuron allows us to describe the accumulated activations and errors of the spiking neural network in terms of an equivalent artificial neural network, allowing us to largely speed up training compared to an explicit simulation of all spike events. This way we are able to demonstrate that even for deep networks, the gradients can be discretized sufficiently well with spikes if the gradient is properly rescaled. This form of spike-based backpropagation enables us to achieve equivalent or better accuracies on the MNIST and CIFAR10 dataset than comparable state-of-the-art spiking neural networks trained with full precision gradients. The algorithm, which we call SpikeGrad, is based on accumulation and comparison operations and can naturally exploit sparsity in the gradient computation, which makes it an interesting choice for a spiking neuromorphic systems with on-chip learning capacities.
Signal Coding and Perfect Reconstruction using Spike Trains
Authors: Anik Chattopadhyay, Arunava Banerjee
Date: 2019-05-31 00:00:00

In many animal sensory pathways, the transformation from external stimuli to spike trains is essentially deterministic. In this context, a new mathematical framework for coding and reconstruction, based on a biologically plausible model of the spiking neuron, is presented. The framework considers encoding of a signal through spike trains generated by an ensemble of neurons via a standard convolve-then-threshold mechanism. Neurons are distinguished by their convolution kernels and threshold values. Reconstruction is posited as a convex optimization minimizing energy. Formal conditions under which perfect reconstruction of the signal from the spike trains is possible are then identified in this setup. Finally, a stochastic gradient descent mechanism is proposed to achieve these conditions. Simulation experiments are presented to demonstrate the strength and efficacy of the framework
Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains
Authors: Elliot Meyerson, Risto Miikkulainen
Date: 2019-05-31 00:00:00
Proceeding: NeurIPS 2019 12
Tasks: Multi-Task Learning

As deep learning applications continue to become more diverse, an interesting question arises: Can general problem solving arise from jointly learning several such diverse tasks? To approach this question, deep multi-task learning is extended in this paper to the setting where there is no obvious overlap between task architectures. The idea is that any set of (architecture,task) pairs can be decomposed into a set of potentially related subproblems, whose sharing is optimized by an efficient stochastic algorithm. The approach is first validated in a classic synthetic multi-task learning benchmark, and then applied to sharing across disparate architectures for vision, NLP, and genomics tasks. It discovers regularities across these domains, encodes them into sharable modules, and combines these modules systematically to improve performance in the individual tasks. The results confirm that sharing learned functionality across diverse domains and architectures is indeed beneficial, thus establishing a key ingredient for general problem solving in the future.
Neural Bug Finding: A Study of Opportunities and Challenges
Authors: Andrew Habib, Michael Pradel
Date: 2019-06-01 00:00:00

Static analysis is one of the most widely adopted techniques to find software bugs before code is put in production. Designing and implementing effective and efficient static analyses is difficult and requires high expertise, which results in only a few experts able to write such analyses. This paper explores the opportunities and challenges of an alternative way of creating static bug detectors: neural bug finding. The basic idea is to formulate bug detection as a classification problem, and to address this problem with neural networks trained on examples of buggy and non-buggy code. We systematically study the effectiveness of this approach based on code examples labeled by a state-of-the-art, static bug detector. Our results show that neural bug finding is surprisingly effective for some bug patterns, sometimes reaching a precision and recall of over 80%, but also that it struggles to understand some program properties obvious to a traditional analysis. A qualitative analysis of the results provides insights into why neural bug finders sometimes work and sometimes do not work. We also identify pitfalls in selecting the code examples used to train and validate neural bug finders, and propose an algorithm for selecting effective training data.
Budgeted Policy Learning for Task-Oriented Dialogue Systems
Authors: Zhirui Zhang, Xiujun Li, Jianfeng Gao, Enhong Chen
Date: 2019-06-02 00:00:00
Proceeding: ACL 2019 7
Tasks: Scheduling, Task-Oriented Dialogue Systems

This paper presents a new approach that extends Deep Dyna-Q (DDQ) by incorporating a Budget-Conscious Scheduling (BCS) to best utilize a fixed, small amount of user interactions (budget) for learning task-oriented dialogue agents. BCS consists of (1) a Poisson-based global scheduler to allocate budget over different stages of training; (2) a controller to decide at each training step whether the agent is trained using real or simulated experiences; (3) a user goal sampling module to generate the experiences that are most effective for policy learning. Experiments on a movie-ticket booking task with simulated and real users show that our approach leads to significant improvements in success rate over the state-of-the-art baselines given the fixed budget.
Discovering Neural Wirings
Authors: Mitchell Wortsman, Ali Farhadi, Mohammad Rastegari
Date: 2019-06-03 00:00:00
Proceeding: NeurIPS 2019 12
Tasks: Feature Engineering, Network Pruning, Neural Architecture Search

The success of neural networks has driven a shift in focus from feature engineering to architecture engineering. However, successful networks today are constructed using a small and manually defined set of building blocks. Even in methods of neural architecture search (NAS) the network connectivity patterns are largely constrained. In this work we propose a method for discovering neural wirings. We relax the typical notion of layers and instead enable channels to form connections independent of each other. This allows for a much larger space of possible networks. The wiring of our network is not fixed during training -- as we learn the network parameters we also learn the structure itself. Our experiments demonstrate that our learned connectivity outperforms hand engineered and randomly wired networks. By learning the connectivity of MobileNetV1we boost the ImageNet accuracy by 10% at ~41M FLOPs. Moreover, we show that our method generalizes to recurrent and continuous time networks. Our work may also be regarded as unifying core aspects of the neural architecture search problem with sparse neural network learning. As NAS becomes more fine grained, finding a good architecture is akin to finding a sparse subnetwork of the complete graph. Accordingly, DNW provides an effective mechanism for discovering sparse subnetworks of predefined architectures in a single training run. Though we only ever use a small percentage of the weights during the forward pass, we still play the so-called initialization lottery with a combinatorial number of subnetworks. Code and pretrained models are available at https://github.com/allenai/dnw while additional visualizations may be found at https://mitchellnw.github.io/blog/2019/dnw/.
Improving Minimal Gated Unit for Sequential Data
Authors: Kazuki Takamura, Satoshi Yamane
Date: 2019-05-21 00:00:00
Tasks: Machine Translation, speech-recognition, Speech Recognition, Translation

In order to obtain a model which can process sequential data related to machine translation and speech recognition faster and more accurately, we propose adopting Chrono Initializer as the initialization method of Minimal Gated Unit. We evaluated the method with two tasks: adding task and copy task. As a result of the experiment, the effectiveness of the proposed method was confirmed.
Adaptive Model Refinement with Batch Bayesian Sampling for Optimization of Bio-inspired Flow Tailoring
Authors: Payam Ghassemi, Sumeet Sanjay Lulekar, Souma Chowdhury
Date: 2019-05-31 00:00:00
Tasks: Bayesian Optimization

This paper presents an advancement to an approach for model-independent surrogate-based optimization with adaptive batch sampling, known as Adaptive Model Refinement (AMR). While the original AMR method provides unique decisions with regards to "when" to sample and "how many" samples to add (to preserve the credibility of the optimization search process), it did not provide specific direction towards "where" to sample in the design variable space. This paper thus introduces the capability to identify optimum location to add new samples. The location of the infill points is decided by integrating a Gaussian Process-based criteria ("q-EI"), adopted from Bayesian optimization. The consideration of a penalization term to mitigate interaction among samples (in a batch) is crucial to effective integration of the q-EI criteria into AMR. The new AMR method, called AMR with Penalized Batch Bayesian Sampling (AMR-PBS) is tested on benchmark functions, demonstrating better performance compared to Bayesian EGO. In addition, it is successfully applied to design surface riblets for bio-inspired passive flow control (where high-fidelity samples are given by costly RANS CFD simulations), leading to a 10% drag reduction over the corresponding baseline (i.e., riblet-free aerodynamic surface).