pwc-archive/papers-with-abstracts huggingface dataset:
Transition Based Dependency Parser for Amharic Language Using Deep Learning
Authors: Mizanu Zelalem, Million Meshesha (PhD)
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: ARC, Deep Learning

Researches shows that attempts done to apply existing dependency parser on morphological rich languages including Amharic shows a poor performance. In this study, a dependency parser for Amharic language is implemented using arc-eager transition system and LSTM network. The study introduced another way of building labeled dependency structure by using a separate network model to predict dependency relation. This helps the number of classes to decrease from 2n+2 into n, where n is the number of relationship types in the language and increases the number of examples for each class in the data set. Evaluation of the parser model results 91.54 and 81.4 unlabeled and labeled attachment score respectively. The major challenge in this study was the decrease of the accuracy of labeled attachment score. This is mainly due to the size and quality of the tree-bank available for Amharic language.  Improving the tree-bank by increasing the size and by adding morphological information can make the performance of parser better.
HOW IMPORTANT ARE NETWORK WEIGHTS? TO WHAT EXTENT DO THEY NEED AN UPDATE?
Authors: Fawaz Sammani, Mahmoud Elsayed, Abdelsalam Hamdi
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Image Captioning

In the context of optimization, a gradient of a neural network indicates the amount a specific weight should change with respect to the loss. Therefore, small gradients indicate a good value of the weight that requires no change and can be kept frozen during training. This paper provides an experimental study on the importance of a neural network weights, and to which extent do they need to be updated. We wish to show that starting from the third epoch, freezing weights which have no informative gradient and are less likely to be changed during training, results in a very slight drop in the overall accuracy (and in sometimes better). We experiment on the MNIST, CIFAR10 and Flickr8k datasets using several architectures (VGG19,
ResNet-110 and DenseNet-121). On CIFAR10, we show that freezing 80% of the VGG19 network parameters from the third epoch onwards results in 0.24% drop in accuracy, while freezing 50% of Resnet-110 parameters results in 0.9% drop in accuracy and finally freezing 70% of Densnet-121 parameters results in 0.57% drop in accuracy. Furthermore, to experiemnt with real-life applications, we train an image captioning model with attention mechanism on the Flickr8k dataset using LSTM networks, freezing 60% of the parameters from the third epoch onwards, resulting in a better BLEU-4 score than the fully trained model. Our source code can be found in the appendix.
Efficient Systolic Array Based on Decomposable MAC for Quantized Deep Neural Networks
Authors: Ning-Chi Huang, Huan-Jan Chou, Kai-Chiang Wu
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Quantization

Deep Neural Networks (DNNs) have achieved high accuracy in various machine learning applications in recent years. As the recognition accuracy of deep learning applications increases, reducing the complexity of these neural networks and performing the DNN computation on embedded systems or mobile devices become an emerging and crucial challenge. Quantization has been presented to reduce the utilization of computational resources by compressing the input data and weights from floating-point numbers to integers with shorter bit-width. For practical power reduction, it is necessary to operate these DNNs with quantized parameters on appropriate hardware. Therefore, systolic arrays are adopted to be the major computation units for matrix multiplication in DNN accelerators. To obtain a better tradeoff between the precision/accuracy and power consumption, using parameters with various bit-widths among different layers within a DNN is an advanced quantization method. In this paper, we propose a novel decomposition strategy to construct a low-power decomposable multiplier-accumulator (MAC) for the energy efficiency of quantized DNNs. In the experiments, when 65% multiplication operations of VGG-16 are operated in shorter bit-width with at most 1% accuracy loss on the CIFAR-10 dataset, our decomposable MAC has 50% energy reduction compared with a non-decomposable MAC.
Analytical Moment Regularizer for Training Robust Networks
Authors: Modar Alfadly, Adel Bibi, Muhammed Kocabas, Bernard Ghanem
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Data Augmentation

Despite the impressive performance of deep neural networks (DNNs) on numerous learning  tasks, they still exhibit uncouth behaviours. One puzzling behaviour is the subtle sensitive reaction of DNNs to various noise attacks. Such a nuisance has strengthened the line of research around developing and training noise-robust networks. In this work, we propose a new training regularizer that aims to minimize the probabilistic expected training loss of a DNN subject to a generic Gaussian input. We provide an efficient and simple approach to approximate such a regularizer for arbitrarily deep networks. This is done by leveraging the analytic expression of the output mean of a shallow neural network, avoiding the need for memory and computation expensive data augmentation. We conduct extensive experiments on LeNet and AlexNet on various datasets including MNIST, CIFAR10, and CIFAR100 to demonstrate the effectiveness of our proposed regularizer. In particular, we show that networks that are trained with the proposed regularizer benefit from a boost in robustness against Gaussian noise to an equivalent amount of performing 3-21 folds of noisy data augmentation. Moreover, we empirically show on several architectures and datasets that improving robustness against Gaussian noise, by using the new regularizer, can improve the overall robustness against 6 other types of attacks by two orders of magnitude.
Adversarially Robust Neural Networks via Optimal Control: Bridging Robustness with Lyapunov Stability
Authors: Zhiyang Chen, Hang Su
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Adversarial Robustness

Deep neural networks are known to be vulnerable to adversarial perturbations. In this paper, we bridge adversarial robustness of neural nets with Lyapunov stability of dynamical systems. From this viewpoint, training neural nets is equivalent to finding an optimal control of the discrete dynamical system, which allows one to utilize methods of successive approximations, an optimal control algorithm based on Pontryagin's maximum principle, to train neural nets. This decoupled training method allows us to add constraints to the optimization, which makes the deep model more robust. The constrained optimization problem can be formulated as a semi-definite programming problem and hence can be solved efficiently. Experiments show that our method effectively improves deep model's adversarial robustness.
Learning Generative Image Object Manipulations from Language Instructions
Authors: Martin Längkvist, Andreas Persson, Amy Loutfi
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Object

The use of adequate feature representations is essential for achieving high performance in high-level human cognitive tasks in computational modeling. Recent developments in deep convolutional and recurrent neural networks architectures enable learning powerful feature representations from both images and natural language text. Besides, other types of networks such as Relational Networks (RN) can learn relations between objects and Generative Adversarial Networks (GAN) have shown to generate realistic images. In this paper, we combine these four techniques to acquire a shared feature representation of the relation between objects in an input image and an object manipulation action description in the form of human language encodings to generate an image that shows the resulting end-effect the action would have on a computer-generated scene. The system is trained and evaluated on a simulated dataset and experimentally used on real-world photos.
Smooth Kernels Improve Adversarial Robustness and Perceptually-Aligned Gradients
Authors: Haohan Wang, Xindi Wu, Songwei Ge, Zachary C. Lipton, Eric P. Xing
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Adversarial Robustness

Recent research has shown that CNNs are often overly sensitive to high-frequency textural patterns. Inspired by the intuition that humans are more sensitive to the lower-frequency (larger-scale) patterns we design a regularization scheme that penalizes large differences between adjacent components within each convolutional kernel. We apply our regularization onto several popular training methods, demonstrating that the models with the proposed smooth kernels enjoy improved adversarial robustness. Further, building on recent work establishing connections between adversarial robustness and interpretability, we show that our method appears to give more perceptually-aligned gradients.
Collaborative Generated Hashing for Market Analysis and Fast Cold-start Recommendation
Authors: Yan Zhang, Ivor W. Tsang, Lixin Duan, Guowu Yang
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Marketing, Recommendation Systems

Cold-start and efficiency issues of the Top-k recommendation are critical to large-scale recommender systems. Previous hybrid recommendation methods are effective to deal with the cold-start issues by extracting real latent factors of cold-start items(users) from side information, but they still suffer low efficiency in online recommendation caused by the expensive similarity search in real latent space. This paper presents a collaborative generated hashing (CGH) to improve the efficiency by denoting users and items as binary codes, which applies to various settings: cold-start users, cold-start items and warm-start ones. Specifically, CGH is designed to learn hash functions of users and items through the Minimum Description Length (MDL) principle; thus, it can deal with various recommendation settings. In addition, CGH initiates a new marketing strategy through mining potential users by a generative step. To reconstruct effective users, the MDL principle is used to learn compact and informative binary codes from the content data. Extensive experiments on two public datasets show the advantages for recommendations in various settings over competing baselines and analyze the feasibility of the application in marketing.
A shallow feature extraction network with a large receptive field for stereo matching tasks
Authors: Jianguo Liu, Yunjian Feng, Guo Ji, Fuwu Yan
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Stereo Matching

Stereo matching is one of the important basic tasks in the computer vision field. In recent years, stereo matching algorithms based on deep learning have achieved excellent performance and become the mainstream research direction. Existing algorithms generally use deep convolutional neural networks (DCNNs) to extract more abstract semantic information, but we believe that the detailed information of the spatial structure is more important for stereo matching tasks. Based on this point of view, this paper proposes a shallow feature extraction network with a large receptive field. The network consists of three parts: a primary feature extraction module, an atrous spatial pyramid pooling (ASPP) module and a feature fusion module. The primary feature extraction network contains only three convolution layers. This network utilizes the basic feature extraction ability of the shallow network to extract and retain the detailed information of the spatial structure. In this paper, the dilated convolution and atrous spatial pyramid pooling (ASPP) module is introduced to increase the size of receptive field. In addition, a feature fusion module is designed, which integrates the feature maps with multiscale receptive fields and mutually complements the feature information of different scales. We replaced the feature extraction part of the existing stereo matching algorithms with our shallow feature extraction network, and achieved state-of-the-art performance on the KITTI 2015 dataset. Compared with the reference network, the number of parameters is reduced by 42%, and the matching accuracy is improved by 1.9%.
UW-NET: AN INCEPTION-ATTENTION NETWORK FOR UNDERWATER IMAGE CLASSIFICATION
Authors: Miao Yang and Ke Hu, Chongyi Li, Zhiqiang Wei
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Classification, General Classification, image-classification, Image Classification, Object Recognition

The classification of images taken in special imaging environments except air is the first challenge in extending the applications of deep learning. We report on an UW-Net (Underwater Network), a new convolutional neural network (CNN) based network for underwater image classification. In this model, we simulate the visual correlation of background attention with image understanding for special environments, such as fog and underwater by constructing an inception-attention (I-A) module. The experimental results demonstrate that the proposed UW-Net achieves an accuracy of 99.3% on underwater image classification, which is significantly better than other image classification networks, such as AlexNet, InceptionV3, ResNet and Se-ResNet. Moreover, we demonstrate the proposed IA module can be used to boost the performance of the existing object recognition networks. By substituting the inception module with the I-A module, the Inception-ResnetV2 network achieves a 10.7% top1 error rate and a 0% top5 error rate on the subset of ILSVRC-2012, which further illustrates the function of the background attention in the image classifications.
Attacking Lifelong Learning Models with Gradient Reversion
Authors: Yunhui Guo, Mingrui Liu, Yandong Li, Liqiang Wang, Tianbao Yang, Tajana Rosing
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Continual Learning, Lifelong learning

Lifelong  learning  aims  at  avoiding  the  catastrophic  forgetting  problem  of  traditional supervised learning  models.   Episodic memory based  lifelong learning methods such as A-GEM (Chaudhry et al., 2018b) are shown to achieve the state-of-the-art results across the benchmarks. In A-GEM, a small episodic memory is utilized to store a random subset of the examples from previous tasks.  While the model is trained on a new task, a reference gradient is computed on the episodic memory to guide the direction of the current update.  While A-GEM has strong continual learning ability,  it is not clear that if it can retain the performance in the presence of adversarial attacks.  In this paper, we examine the robustness ofA-GEM against adversarial attacks to the examples in the episodic memory.  We evaluate the effectiveness of traditional attack methods such as FGSM and PGD.The results show that A-GEM still possesses strong continual learning ability in the  presence  of  adversarial  examples  in  the  memory  and  simple  defense  techniques such as label smoothing can further alleviate the adversarial effects.  We presume that traditional attack methods are specially designed for standard supervised learning models rather than lifelong learning models. we therefore propose a principled way for attacking A-GEM called gradient reversion(GREV) which is shown to be more effective.  Our results indicate that future lifelong learning research should bear adversarial attacks in mind to develop more robust lifelong learning algorithms.
Enforcing Physical Constraints in Neural Neural Networks through Differentiable PDE Layer
Authors: Chiyu "Max" Jiang, Karthik Kashinath, Prabhat, Philip Marcus
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Generative Adversarial Network, Super-Resolution

Recent studies at the intersection of physics and deep learning have illustrated successes in the application of deep neural networks to partially or fully replace costly physics simulations. Enforcing physical constraints to solutions generated
by neural networks remains a challenge, yet it is essential to the accuracy and trustworthiness of such model predictions. Many systems in the physical sciences are governed by Partial Differential Equations (PDEs). Enforcing these as hard
constraints, we show, are inefficient in conventional frameworks due to the high dimensionality of the generated fields. To this end, we propose the use of a novel differentiable spectral projection layer for neural networks that efficiently enforces
spatial PDE constraints using spectral methods, yet is fully differentiable, allowing for its use as a layer in neural networks that supports end-to-end training. We show that its computational cost is cheaper than a regular convolution layer. We apply it to
an important class of physical systems – incompressible turbulent flows, where the divergence-free PDE constraint is required. We train a 3D Conditional Generative Adversarial Network (CGAN) for turbulent flow super-resolution efficiently, whilst
guaranteeing the spatial PDE constraint of zero divergence. Furthermore, our empirical results show that the model produces realistic flow fields with more accurate flow statistics when trained with hard constraints imposed via the proposed
novel differentiable spectral projection layer, as compared to soft constrained and unconstrained counterparts.
Hybrid Model-Based and Data-Driven Wind Velocity Estimator for the Navigation System of a Robotic Airship
Authors: Apolo Silva Marton, André Ricardo Fioravanti, José Raul Azinheira, Ely Carneiro de Paiva
Date: 2019-07-14 00:00:00

In the context of autonomous airships, several works in control and guidance use wind velocity to design a control law. However, in general, this information is not directly measured in robotic airships. This paper presents three alternative versions for estimation of wind velocity. Firstly, an Extended Kalman Filter is designed as a model-based approach. Then a Neural Network is designed as a data-driven approach. Finally, a hybrid estimator is proposed by performing a fusion between the previous designed estimators: model-based and data-driven. All approaches consider only GPS, IMU and Pitot tube as available sensors. Simulations in a realistic nonlinear model of the airship suggest that the cooperation between these two techniques increases the estimation performance.
Multiobjective Optimization and Comparisonof Nonsingleton Type-1 and Singleton IntervalType-2 Fuzzy Logic Systems
Authors: Ana Bel ́en Cara, Christian Wagner
Date: 2013-12-21 00:00:00
Proceeding: journal 2013 12
Tasks: Multiobjective Optimization, Vocal Bursts Type Prediction

Singleton interval type-2 fuzzy logic systems (FLSs)have been widely applied in several real-world applications, whereit was shown that the singleton interval type-2 FLSs outperformtheir singleton type-1 counterparts in applications with high un-certainty levels. However, one of the main criticisms of singletoninterval  type-2  FLSs  is  the  fact  that  they  outperform  singletontype-1 FLSs solely based on their use of extra degrees of freedom(extra parameters) and that type-1 FLSs with a sufficiently largenumber of parameters may provide the same performance as in-terval type-2 FLSs. In addition, most works on type-2 FLSs onlycompare their results with singleton type-1 FLSs but fail to con-sider nonsingleton type-1 systems. In this paper, we aim to directlyaddress and investigate this criticism. In order to do so, we willperform a comparative study between optimized singleton type-1,nonsingleton type-1, and singleton interval type-2 FLSs under thepresence of noise. We will also present a multiobjective evolution-ary  algorithm  (MOEA) for  the  optimization of  singleton type-1,nonsingleton type-1, and singleton interval type-2 fuzzy systems forfunction approximation problems. The MOEA will aim to satisfytwo objectives to maximize the accuracy of the FLS and minimizethe number of rules in the FLS, thus improving its interpretability.Furthermore, we will present a methodology to obtain “optimal”consequents  for  the  FLSs.  Hence,  this  paper  has  two  main  con-tributions: First, it provides a common methodology to learn thethree types of FLSs (i.e., singleton type-1, nonsingleton type-1, andsingleton  interval  type-2  FLSs)  from  data  samples.  The  secondcontribution is the creation of a common framework for the com-parison of type-1 and type-2 FLSs that allows us to address theaforementioned criticism. We provide details of a series of experi-ments and include statistical analysis showing that the type-2 FLSis able to handle higher levels of noise than its nonsingleton andsingleton type-1 counterparts.
Fully Bayesian Logistic Regression with Hyper-Lasso Priors for High-dimensional Feature Selection
Authors: Longhai Li, Weixin Yao
Date: 2014-05-13 00:00:00
Tasks: feature selection, regression

High-dimensional feature selection arises in many areas of modern science. For example, in genomic research we want to find the genes that can be used to separate tissues of different classes (e.g. cancer and normal) from tens of thousands of genes that are active (expressed) in certain tissue cells. To this end, we wish to fit regression and classification models with a large number of features (also called variables, predictors). In the past decade, penalized likelihood methods for fitting regression models based on hyper-LASSO penalization have received increasing attention in the literature. However, fully Bayesian methods that use Markov chain Monte Carlo (MCMC) are still in lack of development in the literature. In this paper we introduce an MCMC (fully Bayesian) method for learning severely multi-modal posteriors of logistic regression models based on hyper-LASSO priors (non-convex penalties). Our MCMC algorithm uses Hamiltonian Monte Carlo in a restricted Gibbs sampling framework; we call our method Bayesian logistic regression with hyper-LASSO (BLRHL) priors. We have used simulation studies and real data analysis to demonstrate the superior performance of hyper-LASSO priors, and to investigate the issues of choosing heaviness and scale of hyper-LASSO priors.
Bayesian inference of species trees using diffusion models
Authors: Marnus Stoltz, Boris Bauemer, Remco Bouckaert, Colin Fox, Gordon Hiscott, David Bryant
Date: 2019-09-16 00:00:00
Tasks: Bayesian Inference

We describe a new and computationally efficient Bayesian methodology for inferring species trees and demographics from unlinked binary markers. Likelihood calculations are carried out using diffusion models of allele frequency dynamics combined with a new algorithm for numerically computing likelihoods of quantitative traits. The diffusion approach allows for analysis of datasets containing hundreds or thousands of individuals. The method, which we call \snapper, has been implemented as part of the Beast2 package. We introduce the models, the efficient algorithms, and report performance of \snapper on simulated data sets and on SNP data from rattlesnakes and freshwater turtles.