pwc-archive/papers-with-abstracts huggingface dataset:
Federated Neural Architecture Search
Authors: Jinliang Yuan, Mengwei Xu, Yuxin Zhao, Kaigui Bian, Gang Huang, Xuanzhe Liu, Shangguang Wang
Date: 2020-02-15 00:00:00
Tasks: Neural Architecture Search

To preserve user privacy while enabling mobile intelligence, techniques have been proposed to train deep neural networks on decentralized data. However, training over decentralized data makes the design of neural architecture quite difficult as it already was. Such difficulty is further amplified when designing and deploying different neural architectures for heterogeneous mobile platforms. In this work, we propose an automatic neural architecture search into the decentralized training, as a new DNN training paradigm called Federated Neural Architecture Search, namely federated NAS. To deal with the primary challenge of limited on-client computational and communication resources, we present FedNAS, a highly optimized framework for efficient federated NAS. FedNAS fully exploits the key opportunity of insufficient model candidate re-training during the architecture search process, and incorporates three key optimizations: parallel candidates training on partial clients, early dropping candidates with inferior performance, and dynamic round numbers. Tested on large-scale datasets and typical CNN architectures, FedNAS achieves comparable model accuracy as state-of-the-art NAS algorithm that trains models with centralized data, and also reduces the client cost by up to two orders of magnitude compared to a straightforward design of federated NAS.
Regularizing Semi-supervised Graph Convolutional Networks with a Manifold Smoothness Loss
Authors: Qilin Li, Wanquan Liu, Ling Li
Date: 2020-02-11 00:00:00

Existing graph convolutional networks focus on the neighborhood aggregation scheme. When applied to semi-supervised learning, they often suffer from the overfitting problem as the networks are trained with the cross-entropy loss on a small potion of labeled data. In this paper, we propose an unsupervised manifold smoothness loss defined with respect to the graph structure, which can be added to the loss function as a regularization. We draw connections between the proposed loss with an iterative diffusion process, and show that minimizing the loss is equivalent to aggregate neighbor predictions with infinite layers. We conduct experiments on multi-layer perceptron and existing graph networks, and demonstrate that adding the proposed loss can improve the performance consistently.
Fully convolutional networks for structural health monitoring through multivariate time series classification
Authors: Luca Rosafalco, Andrea Manzoni, Stefano Mariani, Alberto Corigliano
Date: 2020-02-12 00:00:00
Tasks: General Classification, Structural Health Monitoring, Time Series, Time Series Analysis, Time Series Classification

We propose a novel approach to Structural Health Monitoring (SHM), aiming at the automatic identification of damage-sensitive features from data acquired through pervasive sensor systems. Damage detection and localization are formulated as classification problems, and tackled through Fully Convolutional Networks (FCNs). A supervised training of the proposed network architecture is performed on data extracted from numerical simulations of a physics-based model (playing the role of digital twin of the structure to be monitored) accounting for different damage scenarios. By relying on this simplified model of the structure, several load conditions are considered during the training phase of the FCN, whose architecture has been designed to deal with time series of different length. The training of the neural network is done before the monitoring system starts operating, thus enabling a real time damage classification. The numerical performances of the proposed strategy are assessed on a numerical benchmark case consisting of an eight-story shear building subjected to two load types, one of which modeling random vibrations due to low-energy seismicity. Measurement noise has been added to the responses of the structure to mimic the outputs of a real monitoring system. Extremely good classification capacities are shown: among the nine possible alternatives (represented by the healthy state and by a damage at any floor), damage is correctly classified in up to 95% of cases, thus showing the strong potential of the proposed approach in view of the application to real-life cases.
Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing
Authors: Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Baek, Byung-soo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae, Jaewe Heo
Date: 2020-02-14 00:00:00
Tasks: Collaborative Filtering, Decoder, Knowledge Tracing

Knowledge tracing, the act of modeling a student's knowledge through learning activities, is an extensively studied problem in the field of computer-aided education. Although models with attention mechanism have outperformed traditional approaches such as Bayesian knowledge tracing and collaborative filtering, they share two limitations. Firstly, the models rely on shallow attention layers and fail to capture complex relations among exercises and responses over time. Secondly, different combinations of queries, keys and values for the self-attention layer for knowledge tracing were not extensively explored. Usual practice of using exercises and interactions (exercise-response pairs) as queries and keys/values respectively lacks empirical support. In this paper, we propose a novel Transformer based model for knowledge tracing, SAINT: Separated Self-AttentIve Neural Knowledge Tracing. SAINT has an encoder-decoder structure where exercise and response embedding sequence separately enter the encoder and the decoder respectively, which allows to stack attention layers multiple times. To the best of our knowledge, this is the first work to suggest an encoder-decoder model for knowledge tracing that applies deep self-attentive layers to exercises and responses separately. The empirical evaluations on a large-scale knowledge tracing dataset show that SAINT achieves the state-of-the-art performance in knowledge tracing with the improvement of AUC by 1.8% compared to the current state-of-the-art models.
Back-and-Forth prediction for deep tensor compression
Authors: Hyomin Choi, Robert A. Cohen, Ivan V. Bajic
Date: 2020-02-14 00:00:00
Tasks: Prediction

Recent AI applications such as Collaborative Intelligence with neural networks involve transferring deep feature tensors between various computing devices. This necessitates tensor compression in order to optimize the usage of bandwidth-constrained channels between devices. In this paper we present a prediction scheme called Back-and-Forth (BaF) prediction, developed for deep feature tensors, which allows us to dramatically reduce tensor size and improve its compressibility. Our experiments with a state-of-the-art object detector demonstrate that the proposed method allows us to significantly reduce the number of bits needed for compressing feature tensors extracted from deep within the model, with negligible degradation of the detection performance and without requiring any retraining of the network weights. We achieve a 62% and 75% reduction in tensor size while keeping the loss in accuracy of the network to less than 1% and 2%, respectively.
Object-based Metamorphic Testing through Image Structuring
Authors: Adrian Wildandyawan, Yasuharu Nishi
Date: 2020-02-12 00:00:00
Tasks: Object

Testing software is often costly due to the need of mass-producing test cases and providing a test oracle for it. This is often referred to as the oracle problem. One method that has been proposed in order to alleviate the oracle problem is metamorphic testing. Metamorphic testing produces new test cases by altering an existing test case, and uses the metamorphic relation between the inputs and the outputs of the System Under Test (SUT) to predict the expected outputs of the produced test cases. Metamorphic testing has often been used for image processing software, where changes are applied to the image's attributes to create new test cases with annotations that are the same as the original image. We refer to this existing method as the image-based metamorphic testing. In this research, we propose an object-based metamorphic testing and a composite metamorphic testing which combines different metamorphic testing approaches to relatively increase test coverage.
Bit Allocation for Multi-Task Collaborative Intelligence
Authors: Saeed Ranjbar Alvar, Ivan V. Bajić
Date: 2020-02-14 00:00:00

Recent studies have shown that collaborative intelligence (CI) is a promising framework for deployment of Artificial Intelligence (AI)-based services on mobile devices. In CI, a deep neural network is split between the mobile device and the cloud. Deep features obtained at the mobile are compressed and transferred to the cloud to complete the inference. So far, the methods in the literature focused on transferring a single deep feature tensor from the mobile to the cloud. Such methods are not applicable to some recent, high-performance networks with multiple branches and skip connections. In this paper, we propose the first bit allocation method for multi-stream, multi-task CI. We first establish a model for the joint distortion of the multiple tasks as a function of the bit rates assigned to different deep feature tensors. Then, using the proposed model, we solve the rate-distortion optimization problem under a total rate constraint to obtain the best rate allocation among the tensors to be transferred. Experimental results illustrate the efficacy of the proposed scheme compared to several alternative bit allocation methods.
Retrain or not retrain? -- efficient pruning methods of deep CNN networks
Authors: Marcin Pietron, Maciej Wielgosz
Date: 2020-02-12 00:00:00
Tasks: image-classification, Image Classification, object-detection, Object Detection, Semantic Segmentation

Convolutional neural networks (CNN) play a major role in image processing tasks like image classification, object detection, semantic segmentation. Very often CNN networks have from several to hundred stacked layers with several megabytes of weights. One of the possible methods to reduce complexity and memory footprint is pruning. Pruning is a process of removing weights which connect neurons from two adjacent layers in the network. The process of finding near optimal solution with specified drop in accuracy can be more sophisticated when DL model has higher number of convolutional layers. In the paper few approaches based on retraining and no retraining are described and compared together.
The Big Three: A Methodology to Increase Data Science ROI by Answering the Questions Companies Care About
Authors: Daniel K. Griffin
Date: 2020-02-12 00:00:00
Tasks: Clustering

Companies may be achieving only a third of the value they could be getting from data science in industry applications. In this paper, we propose a methodology for categorizing and answering 'The Big Three' questions (what is going on, what is causing it, and what actions can I take that will optimize what I care about) using data science. The applications of data science seem to be nearly endless in today's modern landscape, with each company jockeying for position in the new data and insights economy. Yet, data scientists seem to be solely focused on using classification, regression, and clustering methods to answer the question 'what is going on'. Answering questions about why things are happening or how to take optimal actions to improve metrics are relegated to niche fields of research and generally neglected in industry data science analysis. We survey technical methods to answer these other important questions, describe areas in which some of these methods are being applied, and provide a practical example of how to apply our methodology and selected methods to a real business use case.
Graph Deconvolutional Generation
Authors: Daniel Flam-Shepherd, Tony Wu, Alan Aspuru-Guzik
Date: 2020-02-14 00:00:00
Tasks: Decoder, Graph Generation, Graph Matching

Graph generation is an extremely important task, as graphs are found throughout different areas of science and engineering. In this work, we focus on the modern equivalent of the Erdos-Renyi random graph model: the graph variational autoencoder (GVAE). This model assumes edges and nodes are independent in order to generate entire graphs at a time using a multi-layer perceptron decoder. As a result of these assumptions, GVAE has difficulty matching the training distribution and relies on an expensive graph matching procedure. We improve this class of models by building a message passing neural network into GVAE's encoder and decoder. We demonstrate our model on the specific task of generating small organic molecules
Piecewise Convex Function Estimation and Model Selection
Authors: Kurt S. Riedel
Date: 2018-03-11 00:00:00
Tasks: model, Model Selection

Given noisy data, function estimation is considered when the unknown function is known apriori to consist of a small number of regions where the function is either convex or concave. When the regions are known apriori, the estimate is reduced to a finite dimensional convex optimization in the dual space. When the number of regions is unknown, the model selection problem is to determine the number of convexity change points. We use a pilot estimator based on the expected number of false inflection points.
An implicit function learning approach for parametric modal regression
Authors: Yangchen Pan, Ehsan Imani, Martha White, Amir-Massoud Farahmand
Date: 2020-02-14 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: regression

For multi-valued functions---such as when the conditional distribution on targets given the inputs is multi-modal---standard regression approaches are not always desirable because they provide the conditional mean. Modal regression algorithms address this issue by instead finding the conditional mode(s). Most, however, are nonparametric approaches and so can be difficult to scale. Further, parametric approximators, like neural networks, facilitate learning complex relationships between inputs and targets. In this work, we propose a parametric modal regression algorithm. We use the implicit function theorem to develop an objective, for learning a joint function over inputs and targets. We empirically demonstrate on several synthetic problems that our method (i) can learn multi-valued functions and produce the conditional modes, (ii) scales well to high-dimensional inputs, and (iii) can even be more effective for certain uni-modal problems, particularly for high-frequency functions. We demonstrate that our method is competitive in a real-world modal regression problem and two regular regression datasets.
Ensemble Slice Sampling: Parallel, black-box and gradient-free inference for correlated & multimodal distributions
Authors: Minas Karamanis, Florian Beutler
Date: 2020-02-14 00:00:00

Slice Sampling has emerged as a powerful Markov Chain Monte Carlo algorithm that adapts to the characteristics of the target distribution with minimal hand-tuning. However, Slice Sampling's performance is highly sensitive to the user-specified initial length scale hyperparameter and the method generally struggles with poorly scaled or strongly correlated distributions. This paper introduces Ensemble Slice Sampling (ESS), a new class of algorithms that bypasses such difficulties by adaptively tuning the initial length scale and utilising an ensemble of parallel walkers in order to efficiently handle strong correlations between parameters. These affine-invariant algorithms are trivial to construct, require no hand-tuning, and can easily be implemented in parallel computing environments. Empirical tests show that Ensemble Slice Sampling can improve efficiency by more than an order of magnitude compared to conventional MCMC methods on a broad range of highly correlated target distributions. In cases of strongly multimodal target distributions, Ensemble Slice Sampling can sample efficiently even in high dimensions. We argue that the parallel, black-box and gradient-free nature of the method renders it ideal for use in scientific fields such as physics, astrophysics and cosmology which are dominated by a wide variety of computationally expensive and non-differentiable models.
Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples
Authors: Samarth Sinha, Zhengli Zhao, Anirudh Goyal, Colin Raffel, Augustus Odena
Date: 2020-02-14 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Generative Adversarial Network

We introduce a simple (one line of code) modification to the Generative Adversarial Network (GAN) training algorithm that materially improves results with no increase in computational cost: When updating the generator parameters, we simply zero out the gradient contributions from the elements of the batch that the critic scores as `least realistic'. Through experiments on many different GAN variants, we show that this `top-k update' procedure is a generally applicable improvement. In order to understand the nature of the improvement, we conduct extensive analysis on a simple mixture-of-Gaussians dataset and discover several interesting phenomena. Among these is that, when gradient updates are computed using the worst-scoring batch elements, samples can actually be pushed further away from their nearest mode. We also apply our method to recent GAN variants and improve state-of-the-art FID for conditional generation from 9.21 to 8.57 on CIFAR-10.
Boosted Locality Sensitive Hashing: Discriminative Binary Codes for Source Separation
Authors: Sunwoo Kim, Haici Yang, Minje Kim
Date: 2020-02-14 00:00:00
Tasks: Binary Classification, Denoising, Speech Denoising, Speech Enhancement

Speech enhancement tasks have seen significant improvements with the advance of deep learning technology, but with the cost of increased computational complexity. In this study, we propose an adaptive boosting approach to learning locality sensitive hash codes, which represent audio spectra efficiently. We use the learned hash codes for single-channel speech denoising tasks as an alternative to a complex machine learning model, particularly to address the resource-constrained environments. Our adaptive boosting algorithm learns simple logistic regressors as the weak learners. Once trained, their binary classification results transform each spectrum of test noisy speech into a bit string. Simple bitwise operations calculate Hamming distance to find the K-nearest matching frames in the dictionary of training noisy speech spectra, whose associated ideal binary masks are averaged to estimate the denoising mask for that test mixture. Our proposed learning algorithm differs from AdaBoost in the sense that the projections are trained to minimize the distances between the self-similarity matrix of the hash codes and that of the original spectra, rather than the misclassification rate. We evaluate our discriminative hash codes on the TIMIT corpus with various noise types, and show comparative performance to deep learning methods in terms of denoising performance and complexity.
TPLVM: Portfolio Construction by Student's $t$-process Latent Variable Model
Authors: Yusuke Uchiyama, Kei Nakagawa
Date: 2020-01-29 00:00:00

Optimal asset allocation is a key topic in modern finance theory. To realize the optimal asset allocation on investor's risk aversion, various portfolio construction methods have been proposed. Recently, the applications of machine learning are rapidly growing in the area of finance. In this article, we propose the Student's $t$-process latent variable model (TPLVM) to describe non-Gaussian fluctuations of financial timeseries by lower dimensional latent variables. Subsequently, we apply the TPLVM to minimum-variance portfolio as an alternative of existing nonlinear factor models. To test the performance of the proposed portfolio, we construct minimum-variance portfolios of global stock market indices based on the TPLVM or Gaussian process latent variable model. By comparing these portfolios, we confirm the proposed portfolio outperforms that of the existing Gaussian process latent variable model.
TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
Authors: Wenhao Lu, Jian Jiao, Ruofei Zhang
Date: 2020-02-14 00:00:00
Tasks: Retrieval

Pre-trained language models like BERT have achieved great success in a wide variety of NLP tasks, while the superior performance comes with high demand in computational resources, which hinders the application in low-latency IR systems. We present TwinBERT model for effective and efficient retrieval, which has twin-structured BERT-like encoders to represent query and document respectively and a crossing layer to combine the embeddings and produce a similarity score. Different from BERT, where the two input sentences are concatenated and encoded together, TwinBERT decouples them during encoding and produces the embeddings for query and document independently, which allows document embeddings to be pre-computed offline and cached in memory. Thereupon, the computation left for run-time is from the query encoding and query-document crossing only. This single change can save large amount of computation time and resources, and therefore significantly improve serving efficiency. Moreover, a few well-designed network layers and training strategies are proposed to further reduce computational cost while at the same time keep the performance as remarkable as BERT model. Lastly, we develop two versions of TwinBERT for retrieval and relevance tasks correspondingly, and both of them achieve close or on-par performance to BERT-Base model. The model was trained following the teacher-student framework and evaluated with data from one of the major search engines. Experimental results showed that the inference time was significantly reduced and was firstly controlled around 20ms on CPUs while at the same time the performance gain from fine-tuned BERT-Base model was mostly retained. Integration of the models into production systems also demonstrated remarkable improvements on relevance metrics with negligible influence on latency.
Accurate Stress Assessment based on functional Near Infrared Spectroscopy using Deep Learning Approach
Authors: Mahya Mirbagheri, Ata Jodeiri, Naser Hakimi, Vahid Zakeri, Seyed Kamaledin Setarehdan
Date: 2020-02-14 00:00:00
Tasks: Deep Learning, General Classification

Stress is known as one of the major factors threatening human health. A large number of studies have been performed in order to either assess or relieve stress by analyzing the brain and heart-related signals. In this study, signals produced by functional Near-Infrared Spectroscopy (fNIRS) of the brain recorded from 10 healthy volunteers are employed to assess the stress induced by the Montreal Imaging Stress Task by means of a deep learning system. The proposed deep learning system consists of two main parts: First, the one-dimensional convolutional neural network is employed to build informative feature maps. Then, a stack of deep fully connected layers is used to predict the stress existence probability. Experiment results showed that the trained fNIRS model performs stress classification by achieving 88.52 -+ 0.77% accuracy. Employment of the proposed deep learning system trained on the fNIRS measurements leads to higher stress classification accuracy than the existing methods proposed in fNIRS studies in which the same experimental procedure has been employed. The proposed method suggests better stability with lower variation in prediction. Furthermore, its low computational cost opens up the possibility to be applied in real-time stress assessment.