pwc-archive/papers-with-abstracts huggingface dataset:
Quantum Optimisation of Complex Systems with a Quantum Annealer
Authors: Steve Abel, Andrew Blance, Michael Spannowsky
Date: 2021-05-28 00:00:00

We perform an in-depth comparison of quantum annealing with several classical optimisation techniques, namely thermal annealing, Nelder-Mead, and gradient descent. We begin with a direct study of the 2D Ising model on a quantum annealer, and compare its properties directly with those of the thermal 2D Ising model. These properties include an Ising-like phase transition that can be induced by either a change in 'quantum-ness' of the theory, or by a scaling the Ising couplings up or down. This behaviour is in accord with what is expected from the physical understanding of the quantum system. We then go on to demonstrate the efficacy of the quantum annealer at minimising several increasingly hard two dimensional potentials. For all the potentials we find the general behaviour that Nelder-Mead and gradient descent methods are very susceptible to becoming trapped in false minima, while the thermal anneal method is somewhat better at discovering the true minimum. However, and despite current limitations on its size, the quantum annealer performs a minimisation very markedly better than any of these classical techniques. A quantum anneal can be designed so that the system almost never gets trapped in a false minimum, and rapidly and successfully minimises the potentials.
Towards Deterministic Diverse Subset Sampling
Authors: Joachim Schreurs, Michaël Fanuel, Johan A. K. Suykens
Date: 2021-05-28 00:00:00
Tasks: Document Summarization, Image Retrieval, Point Processes

Determinantal point processes (DPPs) are well known models for diverse subset selection problems, including recommendation tasks, document summarization and image search. In this paper, we discuss a greedy deterministic adaptation of k-DPP. Deterministic algorithms are interesting for many applications, as they provide interpretability to the user by having no failure probability and always returning the same results. First, the ability of the method to yield low-rank approximations of kernel matrices is evaluated by comparing the accuracy of the Nystr\"om approximation on multiple datasets. Afterwards, we demonstrate the usefulness of the model on an image search task.
Efficient Online-Bandit Strategies for Minimax Learning Problems
Authors: Christophe Roux, Elias Wirth, Sebastian Pokutta, Thomas Kerdreux
Date: 2021-05-28 00:00:00

Several learning problems involve solving min-max problems, e.g., empirical distributional robust learning or learning with non-standard aggregated losses. More specifically, these problems are convex-linear problems where the minimization is carried out over the model parameters $w\in\mathcal{W}$ and the maximization over the empirical distribution $p\in\mathcal{K}$ of the training set indexes, where $\mathcal{K}$ is the simplex or a subset of it. To design efficient methods, we let an online learning algorithm play against a (combinatorial) bandit algorithm. We argue that the efficiency of such approaches critically depends on the structure of $\mathcal{K}$ and propose two properties of $\mathcal{K}$ that facilitate designing efficient algorithms. We focus on a specific family of sets $\mathcal{S}_{n,k}$ encompassing various learning applications and provide high-probability convergence guarantees to the minimax values.
Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks
Authors: Dong-Young Lim, Sotirios Sabanis
Date: 2021-05-28 00:00:00
Tasks: Deep Learning, Stochastic Optimization

We present a new class of Langevin based algorithms, which overcomes many of the known shortcomings of popular adaptive optimizers that are currently used for the fine tuning of deep learning models. Its underpinning theory relies on recent advances of Euler's polygonal approximations for stochastic differential equations (SDEs) with monotone coefficients. As a result, it inherits the stability properties of tamed algorithms, while it addresses other known issues, e.g. vanishing gradients in neural networks. In particular, we provide a nonasymptotic analysis and full theoretical guarantees for the convergence properties of an algorithm of this novel class, which we named TH$\varepsilon$O POULA (or, simply, TheoPouLa). Finally, several experiments are presented with different types of deep learning models, which show the superior performance of TheoPouLa over many popular adaptive optimization algorithms.
Quantifying and Localizing Usable Information Leakage from Neural Network Gradients
Authors: Fan Mo, Anastasia Borovykh, Mohammad Malekzadeh, Soteris Demetriou, Deniz Gündüz, Hamed Haddadi
Date: 2021-05-28 00:00:00
Tasks: Attribute

In collaborative learning, clients keep their data private and communicate only the computed gradients of the deep neural network being trained on their local data. Several recent attacks show that one can still extract private information from the shared network's gradients compromising clients' privacy. In this paper, to quantify the private information leakage from gradients we adopt usable information theory. We focus on two types of private information: original information in data reconstruction attacks and latent information in attribute inference attacks. Furthermore, a sensitivity analysis over the gradients is performed to explore the underlying cause of information leakage and validate the results of the proposed framework. Finally, we conduct numerical evaluations on six benchmark datasets and four well-known deep models. We measure the impact of training hyperparameters, e.g., batches and epochs, as well as potential defense mechanisms, e.g., dropout and differential privacy. Our proposed framework enables clients to localize and quantify the private information leakage in a layer-wise manner, and enables a better understanding of the sources of information leakage in collaborative learning, which can be used by future studies to benchmark new attacks and defense mechanisms.
Discretization Drift in Two-Player Games
Authors: Mihaela Rosca, Yan Wu, Benoit Dherin, David G. T. Barrett
Date: 2021-05-28 00:00:00
Tasks: Vocal Bursts Valence Prediction

Gradient-based methods for two-player games produce rich dynamics that can solve challenging problems, yet can be difficult to stabilize and understand. Part of this complexity originates from the discrete update steps given by simultaneous or alternating gradient descent, which causes each player to drift away from the continuous gradient flow -- a phenomenon we call discretization drift. Using backward error analysis, we derive modified continuous dynamical systems that closely follow the discrete dynamics. These modified dynamics provide an insight into the notorious challenges associated with zero-sum games, including Generative Adversarial Networks. In particular, we identify distinct components of the discretization drift that can alter performance and in some cases destabilize the game. Finally, quantifying discretization drift allows us to identify regularizers that explicitly cancel harmful forms of drift or strengthen beneficial forms of drift, and thus improve performance of GAN training.
Scalable Frank-Wolfe on Generalized Self-concordant Functions via Simple Steps
Authors: Alejandro Carderera, Mathieu Besançon, Sebastian Pokutta
Date: 2021-05-28 00:00:00
Proceeding: NeurIPS 2021 12

Generalized self-concordance is a key property present in the objective function of many important learning problems. We establish the convergence rate of a simple Frank-Wolfe variant that uses the open-loop step size strategy $\gamma_t = 2/(t+2)$, obtaining a $\mathcal{O}(1/t)$ convergence rate for this class of functions in terms of primal gap and Frank-Wolfe gap, where $t$ is the iteration count. This avoids the use of second-order information or the need to estimate local smoothness parameters of previous work. We also show improved convergence rates for various common cases, e.g., when the feasible region under consideration is uniformly convex or polyhedral.
Video-rate multispectral imaging in laparoscopic surgery: First-in-human application
Authors: Leonardo Ayala, Sebastian Wirkert, Anant Vemuri, Tim Adler, Silvia Seidlitz, Sebastian Pirmann, Christina Engels, Dogu Teber, Lena Maier-Hein
Date: 2021-05-28 00:00:00

Multispectral and hyperspectral imaging (MSI/HSI) can provide clinically relevant information on morphological and functional tissue properties. Application in the operating room (OR), however, has so far been limited by complex hardware setups and slow acquisition times. To overcome these limitations, we propose a novel imaging system for video-rate spectral imaging in the clinical workflow. The system integrates a small snapshot multispectral camera with a standard laparoscope and a clinically commonly used light source, enabling the recording of multispectral images with a spectral dimension of 16 at a frame rate of 25 Hz. An ongoing in patient study shows that multispectral recordings from this system can help detect perfusion changes in partial nephrectomy surgery, thus opening the doors to a wide range of clinical applications.
Volatility Modeling of Stocks from Selected Sectors of the Indian Economy Using GARCH
Authors: Jaydip Sen, Sidra Mehtab, Abhishek Dutta
Date: 2021-05-28 00:00:00
Tasks: Clustering

Volatility clustering is an important characteristic that has a significant effect on the behavior of stock markets. However, designing robust models for accurate prediction of future volatilities of stock prices is a very challenging research problem. We present several volatility models based on generalized autoregressive conditional heteroscedasticity (GARCH) framework for modeling the volatility of ten stocks listed in the national stock exchange (NSE) of India. The stocks are selected from the auto sector and the banking sector of the Indian economy, and they have a significant impact on the sectoral index of their respective sectors in the NSE. The historical stock price records from Jan 1, 2010, to Apr 30, 2021, are scraped from the Yahoo Finance website using the DataReader API of the Pandas module in the Python programming language. The GARCH modules are built and fine-tuned on the training data and then tested on the out-of-sample data to evaluate the performance of the models. The analysis of the results shows that asymmetric GARCH models yield more accurate forecasts on the future volatility of stocks.
Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines
Authors: Aurélien Decelle, Cyril Furtlehner, Beatriz Seoane
Date: 2021-05-28 00:00:00
Proceeding: NeurIPS 2021 12

Training Restricted Boltzmann Machines (RBMs) has been challenging for a long time due to the difficulty of computing precisely the log-likelihood gradient. Over the past decades, many works have proposed more or less successful training recipes but without studying the crucial quantity of the problem: the mixing time, i.e. the number of Monte Carlo iterations needed to sample new configurations from a model. In this work, we show that this mixing time plays a crucial role in the dynamics and stability of the trained model, and that RBMs operate in two well-defined regimes, namely equilibrium and out-of-equilibrium, depending on the interplay between this mixing time of the model and the number of steps, $k$, used to approximate the gradient. We further show empirically that this mixing time increases with the learning, which often implies a transition from one regime to another as soon as $k$ becomes smaller than this time. In particular, we show that using the popular $k$ (persistent) contrastive divergence approaches, with $k$ small, the dynamics of the learned model are extremely slow and often dominated by strong out-of-equilibrium effects. On the contrary, RBMs trained in equilibrium display faster dynamics, and a smooth convergence to dataset-like configurations during the sampling. Finally we discuss how to exploit in practice both regimes depending on the task one aims to fulfill: (i) short $k$ can be used to generate convincing samples in short learning times, (ii) large $k$ (or increasingly large) is needed to learn the correct equilibrium distribution of the RBM. Finally, the existence of these two operational regimes seems to be a general property of energy based models trained via likelihood maximization.
Generative Network-Based Reduced-Order Model for Prediction, Data Assimilation and Uncertainty Quantification
Authors: Vinicius L. S. Silva, Claire E. Heaney, Nenko Nenov, Christopher C. Pain
Date: 2021-05-28 00:00:00
Tasks: Epidemiology, Time Series, Time Series Analysis, Time Series Prediction, Uncertainty Quantification

We propose a new method in which a generative network (GN) integrate into a reduced-order model (ROM) framework is used to solve inverse problems for partial differential equations (PDE). The aim is to match available measurements and estimate the corresponding uncertainties associated with the states and parameters of a numerical physical simulation. The GN is trained using only unconditional simulations of the discretized PDE model. We compare the proposed method with the golden standard Markov chain Monte Carlo. We apply the proposed approaches to a spatio-temporal compartmental model in epidemiology. The results show that the proposed GN-based ROM can efficiently quantify uncertainty and accurately match the measurements and the golden standard, using only a few unconditional simulations of the full-order numerical PDE model.
pRSL: Interpretable Multi-label Stacking by Learning Probabilistic Rules
Authors: Kirchhof Michael, Schmid Lena, Reining Christopher, ten Hompel Michael, Pauly Markus
Date: 2021-05-28 00:00:00
Tasks: Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, Zero-Shot Learning

A key task in multi-label classification is modeling the structure between the involved classes. Modeling this structure by probabilistic and interpretable means enables application in a broad variety of tasks such as zero-shot learning or learning from incomplete data. In this paper, we present the probabilistic rule stacking learner (pRSL) which uses probabilistic propositional logic rules and belief propagation to combine the predictions of several underlying classifiers. We derive algorithms for exact and approximate inference and learning, and show that pRSL reaches state-of-the-art performance on various benchmark datasets. In the process, we introduce a novel multicategorical generalization of the noisy-or gate. Additionally, we report simulation results on the quality of loopy belief propagation algorithms for approximate inference in bipartite noisy-or networks.
Implicit Regularization in Matrix Sensing via Mirror Descent
Authors: Fan Wu, Patrick Rebeschini
Date: 2021-05-28 00:00:00
Proceeding: NeurIPS 2021 12

We study discrete-time mirror descent applied to the unregularized empirical risk in matrix sensing. In both the general case of rectangular matrices and the particular case of positive semidefinite matrices, a simple potential-based analysis in terms of the Bregman divergence allows us to establish convergence of mirror descent -- with different choices of the mirror maps -- to a matrix that, among all global minimizers of the empirical risk, minimizes a quantity explicitly related to the nuclear norm, the Frobenius norm, and the von Neumann entropy. In both cases, this characterization implies that mirror descent, a first-order algorithm minimizing the unregularized empirical risk, recovers low-rank matrices under the same set of assumptions that are sufficient to guarantee recovery for nuclear-norm minimization. When the sensing matrices are symmetric and commute, we show that gradient descent with full-rank factorized parametrization is a first-order approximation to mirror descent, in which case we obtain an explicit characterization of the implicit bias of gradient flow as a by-product.
DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding
Authors: Neil Zeghidour, Olivier Teboul, David Grangier
Date: 2021-05-28 00:00:00
Tasks: speaker-diarization, Speaker Diarization

We introduce DIVE, an end-to-end speaker diarization algorithm. Our neural algorithm presents the diarization task as an iterative process: it repeatedly builds a representation for each speaker before predicting the voice activity of each speaker conditioned on the extracted representations. This strategy intrinsically resolves the speaker ordering ambiguity without requiring the classical permutation invariant training loss. In contrast with prior work, our model does not rely on pretrained speaker representations and optimizes all parameters of the system with a multi-speaker voice activity loss. Importantly, our loss explicitly excludes unreliable speaker turn boundaries from training, which is adapted to the standard collar-based Diarization Error Rate (DER) evaluation. Overall, these contributions yield a system redefining the state-of-the-art on the standard CALLHOME benchmark, with 6.7% DER compared to 7.8% for the best alternative.
GCN-SL: Graph Convolutional Networks with Structure Learning for Graphs under Heterophily
Authors: Mengying Jiang, Guizhong Liu, Yuanchao Su, Xinliang Wu
Date: 2021-05-28 00:00:00
Tasks: Clustering, Node Classification, Representation Learning

In representation learning on the graph-structured data, under heterophily (or low homophily), many popular GNNs may fail to capture long-range dependencies, which leads to their performance degradation. To solve the above-mentioned issue, we propose a graph convolutional networks with structure learning (GCN-SL), and furthermore, the proposed approach can be applied to node classification. The proposed GCN-SL contains two improvements: corresponding to node features and edges, respectively. In the aspect of node features, we propose an efficient-spectral-clustering (ESC) and an ESC with anchors (ESC-ANCH) algorithms to efficiently aggregate feature representations from all similar nodes. In the aspect of edges, we build a re-connected adjacency matrix by using a special data preprocessing technique and similarity learning, and the re-connected adjacency matrix can be optimized directly along with GCN-SL parameters. Considering that the original adjacency matrix may provide misleading information for aggregation in GCN, especially the graphs being with a low level of homophily. The proposed GCN-SL can aggregate feature representations from nearby nodes via re-connected adjacency matrix and is applied to graphs with various levels of homophily. Experimental results on a wide range of benchmark datasets illustrate that the proposed GCN-SL outperforms the stateof-the-art GNN counterparts.
Early Exiting with Ensemble Internal Classifiers
Authors: Tianxiang Sun, Yunhua Zhou, Xiangyang Liu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, Xipeng Qiu
Date: 2021-05-28 00:00:00
Tasks: Diversity, Ensemble Learning

As a simple technique to accelerate inference of large-scale pre-trained models, early exiting has gained much attention in the NLP community. It allows samples to exit early at internal classifiers without passing through the entire model. Most existing work usually trains the internal classifiers independently and employs an exiting strategy to decide whether or not to exit based on the confidence of the current internal classifier. However, none of these works takes full advantage of the fact that the internal classifiers are trained to solve the same task therefore can be used to construct an ensemble. In this paper, we show that a novel objective function for the training of the ensemble internal classifiers can be naturally induced from the perspective of ensemble learning and information theory. The proposed training objective consists of two terms: one for accuracy and the other for the diversity of the internal classifiers. In contrast, the objective used in prior work is exactly the accuracy term of our training objective therefore only optimizes the accuracy but not diversity. Further, we propose a simple voting-based strategy that considers predictions of all the past internal classifiers to infer the correct label and decide whether to exit. Experimental results on various NLP tasks show that our proposed objective function and voting-based strategy can achieve better accuracy-speed trade-offs.
Do not explain without context: addressing the blind spot of model explanations
Authors: Katarzyna Woźnica, Katarzyna Pękala, Hubert Baniecki, Wojciech Kretowicz, Elżbieta Sienkiewicz, Przemysław Biecek
Date: 2021-05-28 00:00:00
Tasks: BIG-bench Machine Learning, Explainable Artificial Intelligence (XAI)

The increasing number of regulations and expectations of predictive machine learning models, such as so called right to explanation, has led to a large number of methods promising greater interpretability. High demand has led to a widespread adoption of XAI techniques like Shapley values, Partial Dependence profiles or permutational variable importance. However, we still do not know enough about their properties and how they manifest in the context in which explanations are created by analysts, reviewed by auditors, and interpreted by various stakeholders. This paper highlights a blind spot which, although critical, is often overlooked when monitoring and auditing machine learning models: the effect of the reference data on the explanation calculation. We discuss that many model explanations depend directly or indirectly on the choice of the referenced data distribution. We showcase examples where small changes in the distribution lead to drastic changes in the explanations, such as a change in trend or, alarmingly, a conclusion. Consequently, we postulate that obtaining robust and useful explanations always requires supporting them with a broader context.
The Feature-First Block Model
Authors: Lawrence Tray, Ioannis Kontoyiannis
Date: 2021-05-28 00:00:00
Tasks: model, Stochastic Block Model

Labelled networks are an important class of data, naturally appearing in numerous applications in science and engineering. A typical inference goal is to determine how the vertex labels (or features) affect the network's structure. In this work, we introduce a new generative model, the feature-first block model (FFBM), that facilitates the use of rich queries on labelled networks. We develop a Bayesian framework and devise a two-level Markov chain Monte Carlo approach to efficiently sample from the relevant posterior distribution of the FFBM parameters. This allows us to infer if and how the observed vertex-features affect macro-structure. We apply the proposed methods to a variety of network data to extract the most important features along which the vertices are partitioned. The main advantages of the proposed approach are that the whole feature-space is used automatically and that features can be rank-ordered implicitly according to impact.