pwc-archive/papers-with-abstracts huggingface dataset:
Application of the Neural Network Dependability Kit in Real-World Environments
Authors: Amit Sahu, Noelia Vállez, Rosana Rodríguez-Bobada, Mohamad Alhaddad, Omar Moured, Georg Neugschwandtner
Date: 2020-12-14 00:00:00
Tasks: Classification, General Classification, image-classification, Image Classification, Medical Image Classification

In this paper, we provide a guideline for using the Neural Network Dependability Kit (NNDK) during the development process of NN models, and show how the algorithm is applied in two image classification use cases. The case studies demonstrate the usage of the dependability kit to obtain insights about the NN model and how they informed the development process of the neural network model. After interpreting neural networks via the different metrics available in the NNDK, the developers were able to increase the NNs' accuracy, trust the developed networks, and make them more robust. In addition, we obtained a novel application-oriented technique to provide supporting evidence for an NN's classification result to the user. In the medical image classification use case, it was used to retrieve case images from the training dataset that were similar to the current patient's image and could therefore act as a support for the NN model's decision and aid doctors in interpreting the results.
Sparsifying networks by traversing Geodesics
Authors: Guruprasad Raghavan, Matt Thomson
Date: 2020-12-12 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12

The geometry of weight spaces and functional manifolds of neural networks play an important role towards 'understanding' the intricacies of ML. In this paper, we attempt to solve certain open questions in ML, by viewing them through the lens of geometry, ultimately relating it to the discovery of points or paths of equivalent function in these spaces. We propose a mathematical framework to evaluate geodesics in the functional space, to find high-performance paths from a dense network to its sparser counterpart. Our results are obtained on VGG-11 trained on CIFAR-10 and MLP's trained on MNIST. Broadly, we demonstrate that the framework is general, and can be applied to a wide variety of problems, ranging from sparsification to alleviating catastrophic forgetting.
Kernelized Classification in Deep Networks
Authors: Sadeep Jayasumana, Srikumar Ramalingam, Sanjiv Kumar
Date: 2020-12-08 00:00:00
Tasks: Classification, General Classification

We propose a kernelized classification layer for deep networks. Although conventional deep networks introduce an abundance of nonlinearity for representation (feature) learning, they almost universally use a linear classifier on the learned feature vectors. We advocate a nonlinear classification layer by using the kernel trick on the softmax cross-entropy loss function during training and the scorer function during testing. However, the choice of the kernel remains a challenge. To tackle this, we theoretically show the possibility of optimizing over all possible positive definite kernels applicable to our problem setting. This theory is then used to device a new kernelized classification layer that learns the optimal kernel function for a given problem automatically within the deep network itself. We show the usefulness of the proposed nonlinear classification layer on several datasets and tasks.
Draw your Neural Networks
Authors: Jatin Sharma, Shobha Lata
Date: 2020-12-12 00:00:00

Deep Neural Networks are the basic building blocks of modern Artificial Intelligence. They are increasingly replacing or augmenting existing software systems due to their ability to learn directly from the data and superior accuracy on variety of tasks. Existing Software Development Life Cycle (SDLC) methodologies fall short on representing the unique capabilities and requirements of AI Development and must be replaced with Artificial Intelligence Development Life Cycle (AIDLC) methodologies. In this paper, we discuss an alternative and more natural approach to develop neural networks that involves intuitive GUI elements such as blocks and lines to draw them instead of complex computer programming. We present Sketch framework, that uses this GUI-based approach to design and modify the neural networks and provides interoperability with traditional frameworks. The system provides popular layers and operations out-of-the-box and could import any supported pre-trained model making it a faster method to design and train complex neural networks and ultimately democratizing the AI by removing the learning curve.
Validate and Enable Machine Learning in Industrial AI
Authors: Hongbo Zou, Guangjing Chen, Pengtao Xie, Sean Chen, Yongtian He, Hochih Huang, Zheng Nie, Hongbao Zhang, Tristan Bala, Kazi Tulip, Yuqi Wang, Shenlin Qin, Eric P. Xing
Date: 2020-10-30 00:00:00
Tasks: BIG-bench Machine Learning

Industrial Artificial Intelligence (Industrial AI) is an emerging concept which refers to the application of artificial intelligence to industry. Industrial AI promises more efficient future industrial control systems. However, manufacturers and solution partners need to understand how to implement and integrate an AI model into the existing industrial control system. A well-trained machine learning (ML) model provides many benefits and opportunities for industrial control optimization; however, an inferior Industrial AI design and integration limits the capability of ML models. To better understand how to develop and integrate trained ML models into the traditional industrial control system, test the deployed AI control system, and ultimately outperform traditional systems, manufacturers and their AI solution partners need to address a number of challenges. Six top challenges, which were real problems we ran into when deploying Industrial AI, are explored in the paper. The Petuum Optimum system is used as an example to showcase the challenges in making and testing AI models, and more importantly, how to address such challenges in an Industrial AI system.
Model-based Reinforcement Learning for Continuous Control with Posterior Sampling
Authors: Ying Fan, Yifei Ming
Date: 2020-11-20 00:00:00
Tasks: continuous-control, Continuous Control, Efficient Exploration, Gaussian Processes, Model-based Reinforcement Learning, Model Predictive Control, regression, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Balancing exploration and exploitation is crucial in reinforcement learning (RL). In this paper, we study model-based posterior sampling for reinforcement learning (PSRL) in continuous state-action spaces theoretically and empirically. First, we show the first regret bound of PSRL in continuous spaces which is polynomial in the episode length to the best of our knowledge. With the assumption that reward and transition functions can be modeled by Bayesian linear regression, we develop a regret bound of $\tilde{O}(H^{3/2}d\sqrt{T})$, where $H$ is the episode length, $d$ is the dimension of the state-action space, and $T$ indicates the total time steps. This result matches the best-known regret bound of non-PSRL methods in linear MDPs. Our bound can be extended to nonlinear cases as well with feature embedding: using linear kernels on the feature representation $\phi$, the regret bound becomes $\tilde{O}(H^{3/2}d_{\phi}\sqrt{T})$, where $d_\phi$ is the dimension of the representation space. Moreover, we present MPC-PSRL, a model-based posterior sampling algorithm with model predictive control for action selection. To capture the uncertainty in models, we use Bayesian linear regression on the penultimate layer (the feature representation layer $\phi$) of neural networks. Empirical results show that our algorithm achieves the state-of-the-art sample efficiency in benchmark continuous control tasks compared to prior model-based algorithms, and matches the asymptotic performance of model-free algorithms.
Who is more ready to get back in shape?
Authors: Rajius Idzalika
Date: 2020-11-30 00:00:00

This empirical study estimates resilience (adaptive capacity) around the periods of the 2013 heavy flood in Cambodia. We use nearly 1.2 million microfinance institution (MFI) customer data and implement the unsupervised learning method. Our results highlight the opportunity to develop resilience by having a better understanding of which areas are likely to be more or less resilient based on the characteristics of the MFI customers, and the individual choices or situations that support stronger adaptiveness. We also discuss the limitation of this approach.
Learning to Solve AC Optimal Power Flow by Differentiating through Holomorphic Embeddings
Authors: Henning Lange, Bingqing Chen, Mario Berges, Soummya Kar
Date: 2020-12-16 00:00:00

Alternating current optimal power flow (AC-OPF) is one of the fundamental problems in power systems operation. AC-OPF is traditionally cast as a constrained optimization problem that seeks optimal generation set points whilst fulfilling a set of non-linear equality constraints -- the power flow equations. With increasing penetration of renewable generation, grid operators need to solve larger problems at shorter intervals. This motivates the research interest in learning OPF solutions with neural networks, which have fast inference time and is potentially scalable to large networks. The main difficulty in solving the AC-OPF problem lies in dealing with this equality constraint that has spurious roots, i.e. there are assignments of voltages that fulfill the power flow equations that however are not physically realizable. This property renders any method relying on projected-gradients brittle because these non-physical roots can act as attractors. In this paper, we show efficient strategies that circumvent this problem by differentiating through the operations of a power flow solver that embeds the power flow equations into a holomorphic function. The resulting learning-based approach is validated experimentally on a 200-bus system and we show that, after training, the learned agent produces optimized power flow solutions reliably and fast. Specifically, we report a 12x increase in speed and a 40% increase in robustness compared to a traditional solver. To the best of our knowledge, this approach constitutes the first learning-based approach that successfully respects the full non-linear AC-OPF equations.
Predictive K-means with local models
Authors: Vincent Lemaire, Oumaima Alaoui Ismaili, Antoine Cornuéjols, Dominique Gay
Date: 2020-12-16 00:00:00
Tasks: Clustering, Explainable Artificial Intelligence (XAI), Prediction

Supervised classification can be effective for prediction but sometimes weak on interpretability or explainability (XAI). Clustering, on the other hand, tends to isolate categories or profiles that can be meaningful but there is no guarantee that they are useful for labels prediction. Predictive clustering seeks to obtain the best of the two worlds. Starting from labeled data, it looks for clusters that are as pure as possible with regards to the class labels. One technique consists in tweaking a clustering algorithm so that data points sharing the same label tend to aggregate together. With distance-based algorithms, such as k-means, a solution is to modify the distance used by the algorithm so that it incorporates information about the labels of the data points. In this paper, we propose another method which relies on a change of representation guided by class densities and then carries out clustering in this new representation space. We present two new algorithms using this technique and show on a variety of data sets that they are competitive for prediction performance with pure supervised classifiers while offering interpretability of the clusters discovered.
Learning active learning at the crossroads? evaluation and discussion
Authors: Louis Desreumaux, Vincent Lemaire
Date: 2020-12-16 00:00:00
Tasks: Active Learning, Meta-Learning

Active learning aims to reduce annotation cost by predicting which samples are useful for a human expert to label. Although this field is quite old, several important challenges to using active learning in real-world settings still remain unsolved. In particular, most selection strategies are hand-designed, and it has become clear that there is no best active learning strategy that consistently outperforms all others in all applications. This has motivated research into meta-learning algorithms for "learning how to actively learn". In this paper, we compare this kind of approach with the association of a Random Forest with the margin sampling strategy, reported in recent comparative studies as a very competitive heuristic. To this end, we present the results of a benchmark performed on 20 datasets that compares a strategy learned using a recent meta-learning algorithm with margin sampling. We also present some lessons learned and open future perspectives.
From Weakly Supervised Learning to Biquality Learning: an Introduction
Authors: Pierre Nodet, Vincent Lemaire, Alexis Bondu, Antoine Cornuéjols, Adam Ouorou
Date: 2020-12-16 00:00:00
Tasks: Weakly-supervised Learning

The field of Weakly Supervised Learning (WSL) has recently seen a surge of popularity, with numerous papers addressing different types of "supervision deficiencies". In WSL use cases, a variety of situations exists where the collected "information" is imperfect. The paradigm of WSL attempts to list and cover these problems with associated solutions. In this paper, we review the research progress on WSL with the aim to make it as a brief introduction to this field. We present the three axis of WSL cube and an overview of most of all the elements of their facets. We propose three measurable quantities that acts as coordinates in the previously defined cube namely: Quality, Adaptability and Quantity of information. Thus we suggest that Biquality Learning framework can be defined as a plan of the WSL cube and propose to re-discover previously unrelated patches in WSL literature as a unified Biquality Learning literature.
On the Importance of Diversity in Re-Sampling for Imbalanced Data and Rare Events in Mortality Risk Models
Authors: Yuxuan, Yang, Hadi Akbarzadeh Khorshidi, Uwe Aickelin, Aditi Nevgi, Elif Ekinci
Date: 2020-12-15 00:00:00
Tasks: Decision Making, Diversity

Surgical risk increases significantly when patients present with comorbid conditions. This has resulted in the creation of numerous risk stratification tools with the objective of formulating associated surgical risk to assist both surgeons and patients in decision-making. The Surgical Outcome Risk Tool (SORT) is one of the tools developed to predict mortality risk throughout the entire perioperative period for major elective in-patient surgeries in the UK. In this study, we enhance the original SORT prediction model (UK SORT) by addressing the class imbalance within the dataset. Our proposed method investigates the application of diversity-based selection on top of common re-sampling techniques to enhance the classifier's capability in detecting minority (mortality) events. Diversity amongst training datasets is an essential factor in ensuring re-sampled data keeps an accurate depiction of the minority/majority class region, thereby solving the generalization problem of mainstream sampling approaches. We incorporate the use of the Solow-Polasky measure as a drop-in functionality to evaluate diversity, with the addition of greedy algorithms to identify and discard subsets that share the most similarity. Additionally, through empirical experiments, we prove that the performance of the classifier trained over diversity-based dataset outperforms the original classifier over ten external datasets. Our diversity-based re-sampling method elevates the performance of the UK SORT algorithm by 1.4$.
Combating Mode Collapse in GAN training: An Empirical Analysis using Hessian Eigenvalues
Authors: Ricard Durall, Avraam Chatzimichailidis, Peter Labus, Janis Keuper
Date: 2020-12-17 00:00:00
Tasks: Image Generation

Generative adversarial networks (GANs) provide state-of-the-art results in image generation. However, despite being so powerful, they still remain very challenging to train. This is in particular caused by their highly non-convex optimization space leading to a number of instabilities. Among them, mode collapse stands out as one of the most daunting ones. This undesirable event occurs when the model can only fit a few modes of the data distribution, while ignoring the majority of them. In this work, we combat mode collapse using second-order gradient information. To do so, we analyse the loss surface through its Hessian eigenvalues, and show that mode collapse is related to the convergence towards sharp minima. In particular, we observe how the eigenvalues of the $G$ are directly correlated with the occurrence of mode collapse. Finally, motivated by these findings, we design a new optimization algorithm called nudged-Adam (NuGAN) that uses spectral information to overcome mode collapse, leading to empirically more stable convergence properties.
A Generalization of Transformer Networks to Graphs
Authors: Vijay Prakash Dwivedi, Xavier Bresson
Date: 2020-12-17 00:00:00
Tasks: Graph Regression, Inductive Bias, Knowledge Graphs, Link Prediction, Node Classification

We propose a generalization of transformer neural network architecture for arbitrary graphs. The original transformer was designed for Natural Language Processing (NLP), which operates on fully connected graphs representing all connections between the words in a sequence. Such architecture does not leverage the graph connectivity inductive bias, and can perform poorly when the graph topology is important and has not been encoded into the node features. We introduce a graph transformer with four new properties compared to the standard model. First, the attention mechanism is a function of the neighborhood connectivity for each node in the graph. Second, the positional encoding is represented by the Laplacian eigenvectors, which naturally generalize the sinusoidal positional encodings often used in NLP. Third, the layer normalization is replaced by a batch normalization layer, which provides faster training and better generalization performance. Finally, the architecture is extended to edge feature representation, which can be critical to tasks s.a. chemistry (bond type) or link prediction (entity relationship in knowledge graphs). Numerical experiments on a graph benchmark demonstrate the performance of the proposed graph transformer architecture. This work closes the gap between the original transformer, which was designed for the limited case of line graphs, and graph neural networks, that can work with arbitrary graphs. As our architecture is simple and generic, we believe it can be used as a black box for future applications that wish to consider transformer and graphs.
A Note on Optimizing the Ratio of Monotone Supermodular Functions
Authors: Wenxin Li
Date: 2020-12-16 00:00:00

We show that for the problem of minimizing (or maximizing) the ratio of two supermodular functions, no bounded approximation ratio can be achieved via polynomial number of queries, if the two supermodular functions are both monotone non-decreasing or non-increasing.
Understanding the Behaviour of Contrastive Loss
Authors: Feng Wang, Huaping Liu
Date: 2020-12-15 00:00:00
Proceeding: CVPR 2021 1
Tasks: Contrastive Learning

Unsupervised contrastive learning has achieved outstanding success, while the mechanism of contrastive loss has been less studied. In this paper, we concentrate on the understanding of the behaviours of unsupervised contrastive loss. We will show that the contrastive loss is a hardness-aware loss function, and the temperature {\tau} controls the strength of penalties on hard negative samples. The previous study has shown that uniformity is a key property of contrastive learning. We build relations between the uniformity and the temperature {\tau} . We will show that uniformity helps the contrastive learning to learn separable features, however excessive pursuit to the uniformity makes the contrastive loss not tolerant to semantically similar samples, which may break the underlying semantic structure and be harmful to the formation of features useful for downstream tasks. This is caused by the inherent defect of the instance discrimination objective. Specifically, instance discrimination objective tries to push all different instances apart, ignoring the underlying relations between samples. Pushing semantically consistent samples apart has no positive effect for acquiring a prior informative to general downstream tasks. A well-designed contrastive loss should have some extents of tolerance to the closeness of semantically similar samples. Therefore, we find that the contrastive loss meets a uniformity-tolerance dilemma, and a good choice of temperature can compromise these two properties properly to both learn separable features and tolerant to semantically similar samples, improving the feature qualities and the downstream performances.
On Episodes, Prototypical Networks, and Few-shot Learning
Authors: Steinar Laenen, Luca Bertinetto
Date: 2020-12-17 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Few-Shot Learning

Episodic learning is a popular practice among researchers and practitioners interested in few-shot learning. It consists of organising training in a series of learning problems (or episodes), each divided into a small training and validation subset to mimic the circumstances encountered during evaluation. But is this always necessary? In this paper, we investigate the usefulness of episodic learning in methods which use nonparametric approaches, such as nearest neighbours, at the level of the episode. For these methods, we not only show how the constraints imposed by episodic learning are not necessary, but that they in fact lead to a data-inefficient way of exploiting training batches. We conduct a wide range of ablative experiments with Matching and Prototypical Networks, two of the most popular methods that use nonparametric approaches at the level of the episode. Their "non-episodic" counterparts are considerably simpler, have less hyperparameters, and improve their performance in multiple few-shot classification datasets.
Clustering with Semidefinite Programming and Fixed Point Iteration
Authors: Pedro Felzenszwalb, Caroline Klivans, Alice Paul
Date: 2020-12-16 00:00:00
Tasks: Clustering

We introduce a novel method for clustering using a semidefinite programming (SDP) relaxation of the Max k-Cut problem. The approach is based on a new methodology for rounding the solution of an SDP relaxation using iterated linear optimization. We show the vertices of the Max k-Cut relaxation correspond to partitions of the data into at most k sets. We also show the vertices are attractive fixed points of iterated linear optimization. Each step of this iterative process solves a relaxation of the closest vertex problem and leads to a new clustering problem where the underlying clusters are more clearly defined. Our experiments show that using fixed point iteration for rounding the Max k-Cut SDP relaxation leads to significantly better results when compared to randomized rounding.