pwc-archive/papers-with-abstracts huggingface dataset:
Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning
Authors: Han-Jia Ye, Hong-You Chen, De-Chuan Zhan, Wei-Lun Chao
Date: 2020-01-06 00:00:00
Tasks: Deep Learning

Classifiers trained with class-imbalanced data are known to perform poorly on test data of the "minor" classes, of which we have insufficient training data. In this paper, we investigate learning a ConvNet classifier under such a scenario. We found that a ConvNet significantly over-fits the minor classes, which is quite opposite to traditional machine learning algorithms that often under-fit minor classes. We conducted a series of analysis and discovered the feature deviation phenomenon -- the learned ConvNet generates deviated features between the training and test data of minor classes -- which explains how over-fitting happens. To compensate for the effect of feature deviation which pushes test data toward low decision value regions, we propose to incorporate class-dependent temperatures (CDT) in training a ConvNet. CDT simulates feature deviation in the training phase, forcing the ConvNet to enlarge the decision values for minor-class data so that it can overcome real feature deviation in the test phase. We validate our approach on benchmark datasets and achieve promising performance. We hope that our insights can inspire new ways of thinking in resolving class-imbalanced deep learning.
A Boolean Task Algebra for Reinforcement Learning
Authors: Geraud Nangue Tasse, Steven James, Benjamin Rosman
Date: 2020-01-06 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Lifelong learning, Negation, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

The ability to compose learned skills to solve new tasks is an important property of lifelong-learning agents. In this work, we formalise the logical composition of tasks as a Boolean algebra. This allows us to formulate new tasks in terms of the negation, disjunction and conjunction of a set of base tasks. We then show that by learning goal-oriented value functions and restricting the transition dynamics of the tasks, an agent can solve these new tasks with no further learning. We prove that by composing these value functions in specific ways, we immediately recover the optimal policies for all tasks expressible under the Boolean algebra. We verify our approach in two domains---including a high-dimensional video game environment requiring function approximation---where an agent first learns a set of base skills, and then composes them to solve a super-exponential number of new tasks.
Mel-spectrogram augmentation for sequence to sequence voice conversion
Authors: Yeongtae Hwang, Hyemin Cho, Hongsun Yang, Dong-Ok Won, Insoo Oh, Seong-Whan Lee
Date: 2020-01-06 00:00:00
Tasks: Voice Conversion

For training the sequence-to-sequence voice conversion model, we need to handle an issue of insufficient data about the number of speech pairs which consist of the same utterance. This study experimentally investigated the effects of Mel-spectrogram augmentation on training the sequence-to-sequence voice conversion (VC) model from scratch. For Mel-spectrogram augmentation, we adopted the policies proposed in SpecAugment. In addition, we proposed new policies (i.e., frequency warping, loudness and time length control) for more data variations. Moreover, to find the appropriate hyperparameters of augmentation policies without training the VC model, we proposed hyperparameter search strategy and the new metric for reducing experimental cost, namely deformation per deteriorating ratio. We compared the effect of these Mel-spectrogram augmentation methods based on various sizes of training set and augmentation policies. In the experimental results, the time axis warping based policies (i.e., time length control and time warping.) showed better performance than other policies. These results indicate that the use of the Mel-spectrogram augmentation is more beneficial for training the VC model.
Retrosynthesis Prediction with Conditional Graph Logic Network
Authors: Hanjun Dai, Chengtao Li, Connor W. Coley, Bo Dai, Le Song
Date: 2020-01-06 00:00:00
Proceeding: NeurIPS 2019 12
Tasks: Prediction, Retrosynthesis, Single-step retrosynthesis

Retrosynthesis is one of the fundamental problems in organic chemistry. The task is to identify reactants that can be used to synthesize a specified product molecule. Recently, computer-aided retrosynthesis is finding renewed interest from both chemistry and computer science communities. Most existing approaches rely on template-based models that define subgraph matching rules, but whether or not a chemical reaction can proceed is not defined by hard decision rules. In this work, we propose a new approach to this task using the Conditional Graph Logic Network, a conditional graphical model built upon graph neural networks that learns when rules from reaction templates should be applied, implicitly considering whether the resulting reaction would be both chemically feasible and strategic. We also propose an efficient hierarchical sampling to alleviate the computation cost. While achieving a significant improvement of $8.1\%$ over current state-of-the-art methods on the benchmark dataset, our model also offers interpretations for the prediction.
Deeper Insights into Weight Sharing in Neural Architecture Search
Authors: Yuge Zhang, Zejun Lin, Junyang Jiang, Quanlu Zhang, Yujing Wang, Hui Xue, Chen Zhang, Yaming Yang
Date: 2020-01-06 00:00:00
Tasks: Neural Architecture Search

With the success of deep neural networks, Neural Architecture Search (NAS) as a way of automatic model design has attracted wide attention. As training every child model from scratch is very time-consuming, recent works leverage weight-sharing to speed up the model evaluation procedure. These approaches greatly reduce computation by maintaining a single copy of weights on the super-net and share the weights among every child model. However, weight-sharing has no theoretical guarantee and its impact has not been well studied before. In this paper, we conduct comprehensive experiments to reveal the impact of weight-sharing: (1) The best-performing models from different runs or even from consecutive epochs within the same run have significant variance; (2) Even with high variance, we can extract valuable information from training the super-net with shared weights; (3) The interference between child models is a main factor that induces high variance; (4) Properly reducing the degree of weight sharing could effectively reduce variance and improve performance.
Deep Learning-Based Solvability of Underdetermined Inverse Problems in Medical Imaging
Authors: Chang Min Hyun, Seong Hyeon Baek, Mingyu Lee, Sung Min Lee, Jin Keun Seo
Date: 2020-01-06 00:00:00
Tasks: Deep Learning

Recently, with the significant developments in deep learning techniques, solving underdetermined inverse problems has become one of the major concerns in the medical imaging domain. Typical examples include undersampled magnetic resonance imaging, interior tomography, and sparse-view computed tomography, where deep learning techniques have achieved excellent performances. Although deep learning methods appear to overcome the limitations of existing mathematical methods when handling various underdetermined problems, there is a lack of rigorous mathematical foundations that would allow us to elucidate the reasons for the remarkable performance of deep learning methods. This study focuses on learning the causal relationship regarding the structure of the training data suitable for deep learning, to solve highly underdetermined inverse problems. We observe that a majority of the problems of solving underdetermined linear systems in medical imaging are highly non-linear. Furthermore, we analyze if a desired reconstruction map can be learnable from the training data and underdetermined system.
Express Wavenet -- a low parameter optical neural network with random shift wavelet pattern
Authors: Yingshi Chen
Date: 2020-01-06 00:00:00

Express Wavenet is an improved optical diffractive neural network. At each layer, it uses wavelet-like pattern to modulate the phase of optical waves. For input image with n2 pixels, express wavenet reduce parameter number from O(n2) to O(n). Only need one percent of the parameters, and the accuracy is still very high. In the MNIST dataset, it only needs 1229 parameters to get accuracy of 92%, while the standard optical network needs 125440 parameters. The random shift wavelets show the characteristics of optical network more vividly. Especially the vanishing gradient phenomenon in the training process. We present a modified expressway structure for this problem. Experiments verified the effect of random shift wavelet and expressway structure. Our work shows optical diffractive network would use much fewer parameters than other neural networks. The source codes are available at https://github.com/closest-git/ONNet.
Think Locally, Act Globally: Federated Learning with Local and Global Representations
Authors: Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B. Allen, Randy P. Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
Date: 2020-01-06 00:00:00
Tasks: Federated Learning, Representation Learning, Visual Navigation, Visual Question Answering (VQA)

Federated learning is a method of training models on private data distributed over multiple devices. To keep device data private, the global model is trained by only communicating parameters and updates which poses scalability challenges for large models. To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices. As a result, the global model can be smaller since it only operates on local representations, reducing the number of communicated parameters. Theoretically, we provide a generalization analysis which shows that a combination of local and global models reduces both variance in the data as well as variance across device distributions. Empirically, we demonstrate that local models enable communication-efficient training while retaining performance. We also evaluate on the task of personalized mood prediction from real-world mobile data where privacy is key. Finally, local models handle heterogeneous data from new devices, and learn fair representations that obfuscate protected attributes such as race, age, and gender.
Estimation of the spatial weighting matrix for regular lattice data -- An adaptive lasso approach with cross-sectional resampling
Authors: Miryam S. Merk, Philipp Otto
Date: 2020-01-06 00:00:00

Spatial econometric research typically relies on the assumption that the spatial dependence structure is known in advance and is represented by a deterministic spatial weights matrix. Contrary to classical approaches, we investigate the estimation of sparse spatial dependence structures for regular lattice data. In particular, an adaptive least absolute shrinkage and selection operator (lasso) is used to select and estimate the individual connections of the spatial weights matrix. To recover the spatial dependence structure, we propose cross-sectional resampling, assuming that the random process is exchangeable. The estimation procedure is based on a two-step approach to circumvent simultaneity issues that typically arise from endogenous spatial autoregressive dependencies. The two-step adaptive lasso approach with cross-sectional resampling is verified using Monte Carlo simulations. Eventually, we apply the procedure to model nitrogen dioxide ($\mathrm{NO_2}$) concentrations and show that estimating the spatial dependence structure contrary to using prespecified weights matrices improves the prediction accuracy considerably.
Shear Stress Distribution Prediction in Symmetric Compound Channels Using Data Mining and Machine Learning Models
Authors: Zohreh Sheikh Khozani, Khabat Khosravi, Mohammadamin Torabi, Amir Mosavi, Bahram Rezaei, Timon Rabczuk
Date: 2019-12-20 00:00:00
Tasks: BIG-bench Machine Learning

Shear stress distribution prediction in open channels is of utmost importance in hydraulic structural engineering as it directly affects the design of stable channels. In this study, at first, a series of experimental tests were conducted to assess the shear stress distribution in prismatic compound channels. The shear stress values around the whole wetted perimeter were measured in the compound channel with different floodplain widths also in different flow depths in subcritical and supercritical conditions. A set of, data mining and machine learning models including Random Forest (RF), M5P, Random Committee (RC), KStar and Additive Regression Model (AR) implemented on attained data to predict the shear stress distribution in the compound channel. Results indicated among these five models, RF method indicated the most precise results with the highest R2 value of 0.9. Finally, the most powerful data mining method which studied in this research (RF) compared with two well-known analytical models of Shiono and Knight Method (SKM) and Shannon method to acquire the proposed model functioning in predicting the shear stress distribution. The results showed that the RF model has the best prediction performance compared to SKM and Shannon models.
Universal Hysteresis Identification Using Extended Preisach Neural Network
Authors: Mojtaba Farrokh, Mehrdad Shafiei Dizaji, Farzad Shafiei Dizaji, Nazanin Moradinasab
Date: 2019-12-22 00:00:00

Hysteresis phenomena have been observed in different branches of physics and engineering sciences. Therefore, several models have been proposed for hysteresis simulation in different fields; however, almost neither of them can be utilized universally. In this paper by inspiring of Preisach Neural Network which was inspired by the Preisach model that basically stemmed from Madelungs rules and using the learning capability of the neural networks, an adaptive universal model for hysteresis is introduced and called Extended Preisach Neural Network Model. It is comprised of input, output and, two hidden layers. The input and output layers contain linear neurons while the first hidden layer incorporates neurons called Deteriorating Stop neurons, which their activation function follows Deteriorating Stop operator. Deteriorating Stop operators can generate non-congruent hysteresis loops. The second hidden layer includes Sigmoidal neurons. Adding the second hidden layer, helps the neural network learn non-Masing and asymmetric hysteresis loops very smoothly. At the input layer, besides input data the rate at which input data changes, is included as well in order to give the model the capability of learning rate-dependent hysteresis loops. Hence, the proposed approach has the capability of the simulation of both rate-independent and rate-dependent hysteresis with either congruent or non-congruent loops as well as symmetric and asymmetric loops. A new hybridized algorithm has been adopted for training the model which is based on a combination of the Genetic Algorithm and the optimization method of sub-gradient with space dilatation. The generality of the proposed model has been evaluated by applying it to various hysteresis from different areas of engineering with different characteristics. The results show that the model is successful in the identification of the considered hystereses.
Study of Robust Two-Stage Reduced-Dimension Sparsity-Aware STAP with Coprime Arrays
Authors: X. Wang, Z. Yang, J. Huang, R. C. de Lamare
Date: 2019-12-23 00:00:00

Space-time adaptive processing (STAP) algorithms with coprime arrays can provide good clutter suppression potential with low cost in airborne radar systems as compared with their uniform linear arrays counterparts. However, the performance of these algorithms is limited by the training samples support in practical applications. To address this issue, a robust two-stage reduced-dimension (RD) sparsity-aware STAP algorithm is proposed in this work. In the first stage, an RD virtual snapshot is constructed using all spatial channels but only $m$ adjacent Doppler channels around the target Doppler frequency to reduce the slow-time dimension of the signal. In the second stage, an RD sparse measurement modeling is formulated based on the constructed RD virtual snapshot, where the sparsity of clutter and the prior knowledge of the clutter ridge are exploited to formulate an RD overcomplete dictionary. Moreover, an orthogonal matching pursuit (OMP)-like method is proposed to recover the clutter subspace. In order to set the stopping parameter of the OMP-like method, a robust clutter rank estimation approach is developed. Compared with recently developed sparsity-aware STAP algorithms, the size of the proposed sparse representation dictionary is much smaller, resulting in low complexity. Simulation results show that the proposed algorithm is robust to prior knowledge errors and can provide good clutter suppression performance in low sample support.
Simulation of Turbulent Flow around a Generic High-Speed Train using Hybrid Models of RANS Numerical Method with Machine Learning
Authors: Alireza Hajipour, Arash Mirabdolah Lavasani, Mohammad Eftekhari Yazdi, Amir Mosavi, Shahaboddin Shamshirband, Kwok-wing Chau
Date: 2019-12-25 00:00:00
Tasks: GPR

In the present paper, an aerodynamic investigation of a high-speed train is performed. In the first section of this article, a generic high-speed train against a turbulent flow is simulated, numerically. The Reynolds-Averaged Navier-Stokes (RANS) equations combined with the turbulence model are applied to solve incompressible turbulent flow around a high-speed train. Flow structure, velocity and pressure contours and streamlines at some typical wind directions are the most important results of this simulation. The maximum and minimum values are specified and discussed. Also, the pressure coefficient for some critical points on the train surface is evaluated. In the following, the wind direction influence the aerodynamic key parameters as drag, lift, and side forces at the mentioned wind directions are analyzed and compared. Moreover, the effects of velocity changes (50, 60, 70, 80 and 90 m/s) are estimated and compared on the above flow and aerodynamic parameters. In the second section of the paper, various data-driven methods including Gene Expression Programming (GEP), Gaussian Process Regression (GPR), and random forest (RF), are applied for predicting output parameters. So, drag, lift, and side forces and also minimum and a maximum of pressure coefficients for mentioned wind directions and velocity are predicted and compared using statistical parameters. Obtained results indicated that RF in all coefficients of wind direction and most coefficients of free stream velocity provided the most accurate predictions. As a conclusion, RF may be recommended for the prediction of aerodynamic coefficients.
Dissecting Catastrophic Forgetting in Continual Learning by Deep Visualization
Authors: Giang Nguyen, Shuan Chen, Thao Do, Tae Joon Jun, Ho-Jin Choi, Daeyoung Kim
Date: 2020-01-06 00:00:00
Tasks: Continual Learning

Interpreting the behaviors of Deep Neural Networks (usually considered as a black box) is critical especially when they are now being widely adopted over diverse aspects of human life. Taking the advancements from Explainable Artificial Intelligent, this paper proposes a novel technique called Auto DeepVis to dissect catastrophic forgetting in continual learning. A new method to deal with catastrophic forgetting named critical freezing is also introduced upon investigating the dilemma by Auto DeepVis. Experiments on a captioning model meticulously present how catastrophic forgetting happens, particularly showing which components are forgetting or changing. The effectiveness of our technique is then assessed; and more precisely, critical freezing claims the best performance on both previous and coming tasks over baselines, proving the capability of the investigation. Our techniques could not only be supplementary to existing solutions for completely eradicating catastrophic forgetting for life-long learning but also explainable.
A Note on Portfolio Optimization with Quadratic Transaction Costs
Authors: Pierre Chen, Edmond Lezmi, Thierry Roncalli, Jiali Xu
Date: 2020-01-06 00:00:00
Tasks: Portfolio Optimization

In this short note, we consider mean-variance optimized portfolios with transaction costs. We show that introducing quadratic transaction costs makes the optimization problem more difficult than using linear transaction costs. The reason lies in the specification of the budget constraint, which is no longer linear. We provide numerical algorithms for solving this issue and illustrate how transaction costs may considerably impact the expected returns of optimized portfolios.
ARA : Aggregated RAPPOR and Analysis for Centralized Differential Privacy
Authors: Sudipta Paul, Subhankar Mishra
Date: 2020-01-06 00:00:00
Tasks: Position

Differential privacy(DP) has now become a standard in case of sensitive statistical data analysis. The two main approaches in DP is local and central. Both the approaches have a clear gap in terms of data storing,amount of data to be analyzed, analysis, speed etc. Local wins on the speed. We have tested the state of the art standard RAPPOR which is a local approach and supported this gap. Our work completely focuses on that part too. Here, we propose a model which initially collects RAPPOR reports from multiple clients which are then pushed to a Tf-Idf estimation model. The Tf-Idf estimation model then estimates the reports on the basis of the occurrence of "on bit" in a particular position and its contribution to that position. Thus it generates a centralized differential privacy analysis from multiple clients. Our model successfully and efficiently analyzed the major truth value every time.
Optimal Options for Multi-Task Reinforcement Learning Under Time Constraints
Authors: Manuel Del Verme, Bruno Castro da Silva, Gianluca Baldassarre
Date: 2020-01-06 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Reinforcement learning can greatly benefit from the use of options as a way of encoding recurring behaviours and to foster exploration. An important open problem is how can an agent autonomously learn useful options when solving particular distributions of related tasks. We investigate some of the conditions that influence optimality of options, in settings where agents have a limited time budget for learning each task and the task distribution might involve problems with different levels of similarity. We directly search for optimal option sets and show that the discovered options significantly differ depending on factors such as the available learning time budget and that the found options outperform popular option-generation heuristics.
Are skip connections necessary for biologically plausible learning rules?
Authors: Daniel Jiwoong Im, Rutuja Patil, Kristin Branson
Date: 2019-12-04 00:00:00
Proceeding: NeurIPS Workshop Neuro_AI 2019 12

Backpropagation is the workhorse of deep learning, however, several other biologically-motivated learning rules have been introduced, such as random feedback alignment and difference target propagation. None of these methods have produced a competitive performance against backpropagation. In this paper, we show that biologically-motivated learning rules with skip connections between intermediate layers can perform as well as backpropagation on the MNIST dataset and are robust to various sets of hyper-parameters.