pwc-archive/papers-with-abstracts huggingface dataset:
Non-linear Neurons with Human-like Apical Dendrite Activations
Authors: Mariana-Iuliana Georgescu, Radu Tudor Ionescu, Nicolae-Catalin Ristea, Nicu Sebe
Date: 2020-02-02 00:00:00
Tasks: Speech Emotion Recognition

In order to classify linearly non-separable data, neurons are typically organized into multi-layer neural networks that are equipped with at least one hidden layer. Inspired by some recent discoveries in neuroscience, we propose a new model of artificial neuron along with a novel activation function enabling the learning of nonlinear decision boundaries using a single neuron. We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. Furthermore, we conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing, i.e. MOROCO, UTKFace, CREMA-D, Fashion-MNIST, Tiny ImageNet and ImageNet, showing that the ADA and the leaky ADA functions provide superior results to Rectified Linear Units (ReLU), leaky ReLU, RBF and Swish, for various neural network architectures, e.g. one-hidden-layer or two-hidden-layer multi-layer perceptrons (MLPs) and convolutional neural networks (CNNs) such as LeNet, VGG, ResNet and Character-level CNN. We obtain further performance improvements when we change the standard model of the neuron with our pyramidal neuron with apical dendrite activations (PyNADA). Our code is available at: https://github.com/raduionescu/pynada.
R 1 -PCA: Rotational Invariant L 1 -norm Principal Component Analysis for Robust Subspace Factorization
Authors: Chris Ding ，Ding Zhou ，Xiaofeng He ，Hongyuan Zha
Date: 2006-06-01 00:00:00
Proceeding: ICML '06: Proceedings of the 23rd international conference on Machine learningJune 2006 Pages 281–288 2006 6
Tasks: Clustering

Principal component analysis (PCA) mini-
mizes the sum of squared errors (L 2 -norm)
and is sensitive to the presence of outliers.
We propose a rotational invariant L 1 -norm
PCA (R 1 -PCA). R 1 -PCA is similar to PCA
in that (1) it has a unique global solution,
(2) the solution are principal eigenvectors of
a robust covariance matrix (re-weighted to
soften the effects of outliers), (3) the solu-
tion is rotational invariant. These proper-
ties are not shared by the L 1 -norm PCA. A
new subspace iteration algorithm is given to
compute R 1 -PCA efficiently. Experiments on
several real-life datasets show R 1 -PCA can
effectively handle outliers. We extend R 1 -
norm to K-means clustering and show that
L 1 -norm K-means leads to poor results while
R 1 -K-means outperforms standard K-means.
Natural Evolution Strategies
Authors: Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jürgen Schmidhuber
Date: 2011-06-22 00:00:00
Tasks: Evolutionary Algorithms, global-optimization

This paper presents Natural Evolution Strategies (NES), a recent family of algorithms that constitute a more principled approach to black-box optimization than established evolutionary algorithms. NES maintains a parameterized distribution on the set of solution candidates, and the natural gradient is used to update the distribution's parameters in the direction of higher expected fitness. We introduce a collection of techniques that address issues of convergence, robustness, sample complexity, computational complexity and sensitivity to hyperparameters. This paper explores a number of implementations of the NES family, ranging from general-purpose multi-variate normal distributions to heavy-tailed and separable distributions tailored towards global optimization and search in high dimensional spaces, respectively. Experimental results show best published performance on various standard benchmarks, as well as competitive performance on others.
Empirical evaluation of scoring functions for Bayesian network model selection
Authors: Zhifa Liu, Brandon Malone, Changhe Yuan
Date: 2012-09-11 00:00:00
Proceeding: Ninth Annual MCBIOS Conference. Dealing with the Omics Data Deluge 2012 9
Tasks: Experimental Design, Model Selection

In this work, we empirically evaluate the capability of various scoring functions of Bayesian networks for recovering true underlying structures. Similar investigations have been carried out before, but they typically relied on approximate learning algorithms to learn the network structures. The suboptimal structures found by the approximation methods have unknown quality and may affect the reliability of their conclusions. Our study uses an optimal algorithm to learn Bayesian network structures from datasets generated from a set of gold standard Bayesian networks. Because all optimal algorithms always learn equivalent networks, this ensures that only the choice of scoring function affects the learned networks. Another shortcoming of the previous studies stems from their use of random synthetic networks as test cases. There is no guarantee that these networks reflect real-world data. We use real-world data to generate our gold-standard structures, so our experimental design more closely approximates real-world situations. A major finding of our study suggests that, in contrast to results reported by several prior works, the Minimum Description Length (MDL) (or equivalently, Bayesian information criterion (BIC)) consistently outperforms other scoring functions such as Akaike's information criterion (AIC), Bayesian Dirichlet equivalence score (BDeu), and factorized normalized maximum likelihood (fNML) in recovering the underlying Bayesian network structures. We believe this finding is a result of using both datasets generated from real-world applications rather than from random processes used in previous studies and learning algorithms to select high-scoring structures rather than selecting random models. Other findings of our study support existing work, e.g., large sample sizes result in learning structures closer to the true underlying structure; the BDeu score is sensitive to the parameter settings; and the fNML performs pretty well on small datasets. We also tested a greedy hill climbing algorithm and observed similar results as the optimal algorithm.
Argus: Efficient Activity Detection System for Extended Video Analysis
Authors: Wenhe Liu, Guoliang Kang, Po-Yao Huang, Xiaojun Chang, Yijun Qian, Junwei Liang, Liangke Gui, Jing Wen, Peng Chen
Date: 2020-03-02 00:00:00
Proceeding: Proceedings of the IEEE Winter Conference on Applications of Computer Vision Workshops 2020 3
Tasks: Action Detection, Activity Detection, Event Detection, Multi-Object Tracking, object-detection, Object Detection, Object Tracking, Video Object Detection, Video Object Tracking

We propose an Efficient Activity Detection System, Argus, for Extended Video Analysis in the surveillance scenario. For the spatial-temporal event detection in the surveillance video, we first generate video proposals by applying object detection and tracking algorithm which shared the detection features. After that, we extract several different features and apply sequential activity classification with them. Finally, we eliminate inaccurate events and fuse all the predictions from different features. The proposed system wins Trecvid Activities in Extended Video (ActEV) challenge 2019. It achieves the first place with 60.5 mean weighted Pmiss, out-performing the second place system by 14.5 and the baseline R-C3D by 29.0. In TRECVID 2019 Challenge, the proposed system wins the first place with pAUDC@ 0.2 tfa 0.48407
Knowledge Graph Embedding via Graph Attenuated Attention Networks
Authors: Rui Wang, Bicheng Li, Shengwei Hu, Wenqian Du, Min Zhang
Date: 2019-12-31 00:00:00
Proceeding: IEEE Access 2019 12
Tasks: Graph Embedding, Knowledge Base Completion, Knowledge Graph Completion, Knowledge Graph Embedding, Knowledge Graphs, Link Prediction, Relation, Relational Reasoning, World Knowledge

Knowledge graphs contain a wealth of real-world knowledge that can provide strong support for artificial intelligence applications. Much progress has been made in knowledge graph completion, state-of-the-art models are based on graph convolutional neural networks. These models automatically extract features, in combination with the features of the graph model, to generate feature embeddings with a strong expressive ability. However, these methods assign the same weights on the relation path in the knowledge graph and ignore the rich information presented in neighbor nodes, which result in incomplete mining of triple features. To this end, we propose Graph Attenuated Attention networks(GAATs), a novel representation method, which integrates an attenuated attention mechanism to assign different weight in different relation path and acquire the information from the neighborhoods. As a result, entities and relations can be learned in any neighbors. Our empirical research provides insight into the effectiveness of the attenuated attention-based models, and we show significant improvement compared to the state-of-the-art methods on two benchmark datasets WN18RR and FB15k-237.
Synaptic Metaplasticity in Binarized Neural Networks
Authors: Axel Laborieux, Maxence Ernoult, Tifenn Hirtzlin, Damien Querlioz
Date: 2020-03-07 00:00:00

While deep neural networks have surpassed human performance in multiple situations, they are prone to catastrophic forgetting: upon training a new task, they rapidly forget previously learned ones. Neuroscience studies, based on idealized tasks, suggest that in the brain, synapses overcome this issue by adjusting their plasticity depending on their past history. However, such "metaplastic" behaviours do not transfer directly to mitigate catastrophic forgetting in deep neural networks. In this work, we interpret the hidden weights used by binarized neural networks, a low-precision version of deep neural networks, as metaplastic variables, and modify their training technique to alleviate forgetting. Building on this idea, we propose and demonstrate experimentally, in situations of multitask and stream learning, a training technique that reduces catastrophic forgetting without needing previously presented data, nor formal boundaries between datasets and with performance approaching more mainstream techniques with task boundaries. We support our approach with a theoretical analysis on a tractable task. This work bridges computational neuroscience and deep learning, and presents significant assets for future embedded and neuromorphic systems, especially when using novel nanodevices featuring physics analogous to metaplasticity.
Towards Solving Large-scale Expensive Optimization Problems Efficiently Using Coordinate Descent Algorithm
Authors: Shahryar Rahnamayan, Seyed Jalaleddin Mousavirad
Date: 2020-03-07 00:00:00
Tasks: global-optimization

Many real-world problems are categorized as large-scale problems, and metaheuristic algorithms as an alternative method to solve large-scale problem; they need the evaluation of many candidate solutions to tackle them prior to their convergence, which is not affordable for practical applications since the most of them are computationally expensive. In other words, these problems are not only large-scale but also computationally expensive, that makes them very difficult to solve. There is no efficient surrogate model to support large-scale expensive global optimization (LSEGO) problems. As a result, the algorithms should address LSEGO problems using a limited computational budget to be applicable in real-world applications. Coordinate Descent (CD) algorithm is an optimization strategy based on the decomposition of a n-dimensional problem into n one-dimensional problem. To the best our knowledge, there is no significant study to assess benchmark functions with various dimensions and landscape properties to investigate CD algorithm. In this paper, we propose a modified Coordinate Descent algorithm (MCD) to tackle LSEGO problems with a limited computational budget. Our proposed algorithm benefits from two leading steps, namely, finding the region of interest and then shrinkage of the search space by folding it into the half with exponential speed. One of the main advantages of the proposed algorithm is being free of any control parameters, which makes it far from the intricacies of the tuning process. The proposed algorithm is compared with cooperative co-evolution with delta grouping on 20 benchmark functions with dimension 1000. Also, we conducted some experiments on CEC-2017, D=10, 30, 50, and 100, to investigate the behavior of MCD algorithm in lower dimensions. The results show that MCD is beneficial not only in large-scale problems, but also in low-scale optimization problems.
Nature-Inspired Optimization Algorithms: Challenges and Open Problems
Authors: Xin-She Yang
Date: 2020-03-08 00:00:00
Tasks: Benchmarking

Many problems in science and engineering can be formulated as optimization problems, subject to complex nonlinear constraints. The solutions of highly nonlinear problems usually require sophisticated optimization algorithms, and traditional algorithms may struggle to deal with such problems. A current trend is to use nature-inspired algorithms due to their flexibility and effectiveness. However, there are some key issues concerning nature-inspired computation and swarm intelligence. This paper provides an in-depth review of some recent nature-inspired algorithms with the emphasis on their search mechanisms and mathematical foundations. Some challenging issues are identified and five open problems are highlighted, concerning the analysis of algorithmic convergence and stability, parameter tuning, mathematical framework, role of benchmarking and scalability. These problems are discussed with the directions for future research.
Influence of Initialization on the Performance of Metaheuristic Optimizers
Authors: Qian Li, San-Yang Liu, Xin-She Yang
Date: 2020-03-08 00:00:00
Tasks: Metaheuristic Optimization

All metaheuristic optimization algorithms require some initialization, and the initialization for such optimizers is usually carried out randomly. However, initialization can have some significant influence on the performance of such algorithms. This paper presents a systematic comparison of 22 different initialization methods on the convergence and accuracy of five optimizers: differential evolution (DE), particle swarm optimization (PSO), cuckoo search (CS), artificial bee colony (ABC) algorithm and genetic algorithm (GA). We have used 19 different test functions with different properties and modalities to compare the possible effects of initialization, population sizes and the numbers of iterations. Rigorous statistical ranking tests indicate that 43.37\% of the functions using the DE algorithm show significant differences for different initialization methods, while 73.68\% of the functions using both PSO and CS algorithms are significantly affected by different initialization methods. The simulations show that DE is less sensitive to initialization, while both PSO and CS are more sensitive to initialization. In addition, under the condition of the same maximum number of function evaluations (FEs), the population size can also have a strong effect. Particle swarm optimization usually requires a larger population, while the cuckoo search needs only a small population size. Differential evolution depends more heavily on the number of iterations, a relatively small population with more iterations can lead to better results. Furthermore, ABC is more sensitive to initialization, while such initialization has little effect on GA. Some probability distributions such as the beta distribution, exponential distribution and Rayleigh distribution can usually lead to better performance. The implications of this study and further research topics are also discussed in detail.
Real-World Airline Crew Pairing Optimization: Customized Genetic Algorithm versus Column Generation Method
Authors: Divyam Aggarwal, Dhish Kumar Saxena, Thomas Back, Michael Emmerich
Date: 2020-03-08 00:00:00
Tasks: Combinatorial Optimization

Airline crew pairing optimization problem (CPOP) aims to find a set of flight sequences (crew pairings) that cover all flights in an airline's highly constrained flight schedule at minimum cost. Since crew cost is second only to the fuel cost, CPOP solutioning is critically important for an airline. However, CPOP is NP-hard, and tackling it is quite challenging. The literature suggests, that when the CPOP's scale and complexity is reasonably limited, and an enumeration of all crew pairings is possible, then Metaheuristics are used, predominantly Genetic Algorithms (GAs). Else, Column Generation (CG) based Mixed Integer Programming techniques are used. Notably, as per the literature, a maximum of 45,000 crew pairings have been tackled by GAs. In a significant departure, this paper considers over 800 flights of a US-based large airline (with a monthly network of over 33,000 flights), and tests the efficacy of GAs by enumerating all 400,000+ crew pairings, apriori. Towards it, this paper proposes a domain-knowledge-driven customized-GA. The utility of incorporating domain-knowledge in GA operations, particularly initialization and crossover, is highlighted through suitable experiments. Finally, the proposed GA's performance is compared with a CG-based approach (developed in-house by the authors). Though the latter is found to perform better in terms of solution's cost-quality and run time, it is hoped that this paper will help in better understanding the strengths and limitations of domain-knowledge-driven customizations in GAs, for solving combinatorial optimization problems, including CPOPs.
Interactive Constrained MAP-Elites: Analysis and Evaluation of the Expressiveness of the Feature Dimensions
Authors: Alberto Alvarez, Steve Dahlskog, Jose Font, Julian Togelius
Date: 2020-03-06 00:00:00
Tasks: Diversity

We propose the Interactive Constrained MAP-Elites, a quality-diversity solution for game content generation, implemented as a new feature of the Evolutionary Dungeon Designer: a mixed-initiative co-creativity tool for designing dungeons. The feature uses the MAP-Elites algorithm, an illumination algorithm that segregates the population among several cells depending on their scores with respect to different behavioral dimensions. Users can flexibly and dynamically alternate between these dimensions anytime, thus guiding the evolutionary process in an intuitive way, and then incorporate suggestions produced by the algorithm in their room designs. At the same time, any modifications performed by the human user will feed back into MAP-Elites, closing a circular workflow of constant mutual inspiration. This paper presents the algorithm followed by an in-depth analysis of its behaviour, with the aims of evaluating the expressive range of all possible dimension combinations in several scenarios, as well as discussing their influence in the fitness landscape and in the overall performance of the mixed-initiative procedural content generation.
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
Authors: Esteban Real, Chen Liang, David R. So, Quoc V. Le
Date: 2020-03-06 00:00:00
Tasks: AutoML, BIG-bench Machine Learning

Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.
Learned Spectral Computed Tomography
Authors: Dimitris Kamilis, Mario Blatter, Nick Polydorides
Date: 2020-03-09 00:00:00
Tasks: Computed Tomography (CT)

Spectral Photon-Counting Computed Tomography (SPCCT) is a promising technology that has shown a number of advantages over conventional X-ray Computed Tomography (CT) in the form of material separation, artefact removal and enhanced image quality. However, due to the increased complexity and non-linearity of the SPCCT governing equations, model-based reconstruction algorithms typically require handcrafted regularisation terms and meticulous tuning of hyperparameters making them impractical to calibrate in variable conditions. Additionally, they typically incur high computational costs and in cases of limited-angle data, their imaging capability deteriorates significantly. Recently, Deep Learning has proven to provide state-of-the-art reconstruction performance in medical imaging applications while circumventing most of these challenges. Inspired by these advances, we propose a Deep Learning imaging method for SPCCT that exploits the expressive power of Neural Networks while also incorporating model knowledge. The method takes the form of a two-step learned primal-dual algorithm that is trained using case-specific data. The proposed approach is characterised by fast reconstruction capability and high imaging performance, even in limited-data cases, while avoiding the hand-tuning that is required by other optimisation approaches. We demonstrate the performance of the method in terms of reconstructed images and quality metrics via numerical examples inspired by the application of cardiovascular imaging.
Deep Inverse Feature Learning: A Representation Learning of Error
Authors: Behzad Ghazanfari, Fatemeh Afghah
Date: 2020-03-09 00:00:00
Tasks: Classification, Clustering, Deep Clustering, General Classification, Representation Learning

This paper introduces a novel perspective about error in machine learning and proposes inverse feature learning (IFL) as a representation learning approach that learns a set of high-level features based on the representation of error for classification or clustering purposes. The proposed perspective about error representation is fundamentally different from current learning methods, where in classification approaches they interpret the error as a function of the differences between the true labels and the predicted ones or in clustering approaches, in which the clustering objective functions such as compactness are used. Inverse feature learning method operates based on a deep clustering approach to obtain a qualitative form of the representation of error as features. The performance of the proposed IFL method is evaluated by applying the learned features along with the original features, or just using the learned features in different classification and clustering techniques for several data sets. The experimental results show that the proposed method leads to promising results in classification and especially in clustering. In classification, the proposed features along with the primary features improve the results of most of the classification methods on several popular data sets. In clustering, the performance of different clustering methods is considerably improved on different data sets. There are interesting results that show some few features of the representation of error capture highly informative aspects of primary features. We hope this paper helps to utilize the error representation learning in different feature learning domains.