pwc-archive/papers-with-abstracts huggingface dataset:
SNR optimization of multi-span fiber optic communication systems employing EDFAs with non-flat gain and noise figure
Authors: Metodi Plamenov Yankov, Pawel Marcin Kaminski, Henrik Enggaard Hansen, Francesco Da Ros
Date: 2021-06-07 00:00:00

Throughput optimization of optical communication systems is a key challenge for current optical networks. The use of gain-flattening filters (GFFs) simplifies the problem at the cost of insertion loss, higher power consumption and potentially poorer performance. In this work, we propose a component wise model of a multi-span transmission system for signal-to-noise (SNR) optimization. A machine-learning based model is trained for the gain and noise figure spectral profile of a C-band amplifier without a GFF. The model is combined with the Gaussian noise model for nonlinearities in optical fibers including stimulated Raman scattering and the implementation penalty spectral profile measured in back-to-back in order to predict the SNR in each channel of a multi-span wavelength division multiplexed system. All basic components in the system model are differentiable and allow for the gradient descent-based optimization of a system of arbitrary configuration in terms of number of spans and length per span. When the input power profile is optimized for flat and maximized received SNR per channel, the minimum performance in an arbitrary 3-span experimental system is improved by up to 8 dB w.r.t. a system with flat input power profile. An SNR flatness down to 1.2 dB is simultaneously achieved. The model and optimization methods are used to optimize the performance of an example core network, and 0.2 dB of gain is shown w.r.t. solutions that do not take into account nonlinearities. The method is also shown to be beneficial for systems with ideal gain flattening, achieving up to 0.3 dB of gain w.r.t. a flat input power profile.
Deep Canonical Correlation Alignment for Sensor Signals
Authors: Narayan Schütz, Angela Botros, Michael Single, Aileen C. Naef, Philipp Buluschek, Tobias Nef
Date: 2021-06-07 00:00:00
Tasks: Dynamic Time Warping

Sensor technologies are becoming increasingly prevalent in the biomedical field, with applications ranging from telemonitoring of people at risk, to using sensor derived information as objective endpoints in clinical trials. To fully utilize sensor information, signals from distinct sensors often have to be temporally aligned. However, due to imperfect oscillators and significant noise, commonly encountered with biomedical signals, temporal alignment of raw signals is an all but trivial problem, with, to-date, no generally applicable solution. In this work, we present Deep Canonical Correlation Alignment (DCCA), a novel, generally applicable solution for the temporal alignment of raw (biomedical) sensor signals. DCCA allows practitioners to directly align raw signals, from distinct sensors, without requiring deep domain knowledge. On a selection of artificial and real datasets, we demonstrate the performance and utility of DCCA under a variety of conditions. We compare the DCCA algorithm to other warping based methods, DCCA outperforms dynamic time warping and cross correlation based methods by an order of magnitude in terms of alignment error. DCCA performs especially well on almost periodic biomedical signals such as heart-beats and breathing patterns. In comparison to existing approaches, that are not tailored towards raw sensor data, DCCA is not only fast enough to work on signals with billions of data points but also provides automatic filtering and transformation functionalities, allowing it to deal with very noisy and even morphologically distinct signals.
PcDGAN: A Continuous Conditional Diverse Generative Adversarial Network For Inverse Design
Authors: Amin Heyrani Nobari, Wei Chen, Faez Ahmed
Date: 2021-06-07 00:00:00
Tasks: Design Synthesis, Diversity, Generative Adversarial Network, Point Processes

Engineering design tasks often require synthesizing new designs that meet desired performance requirements. The conventional design process, which requires iterative optimization and performance evaluation, is slow and dependent on initial designs. Past work has used conditional generative adversarial networks (cGANs) to enable direct design synthesis for given target performances. However, most existing cGANs are restricted to categorical conditions. Recent work on Continuous conditional GAN (CcGAN) tries to address this problem, but still faces two challenges: 1) it performs poorly on non-uniform performance distributions, and 2) the generated designs may not cover the entire design space. We propose a new model, named Performance Conditioned Diverse Generative Adversarial Network (PcDGAN), which introduces a singular vicinal loss combined with a Determinantal Point Processes (DPP) based loss function to enhance diversity. PcDGAN uses a new self-reinforcing score called the Lambert Log Exponential Transition Score (LLETS) for improved conditioning. Experiments on synthetic problems and a real-world airfoil design problem demonstrate that PcDGAN outperforms state-of-the-art GAN models and improves the conditioning likelihood by 69% in an airfoil generation task and up to 78% in synthetic conditional generation tasks and achieves greater design space coverage. The proposed method enables efficient design synthesis and design space exploration with applications ranging from CAD model generation to metamaterial selection.
High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning
Authors: Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander I. Cowen-Rivers, Lin Yang, Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, Jan Peters, Haitham Bou-Ammar
Date: 2021-06-07 00:00:00
Tasks: Bayesian Optimisation, Metric Learning, Molecular Graph Generation, Vocal Bursts Intensity Prediction

We introduce a method combining variational autoencoders (VAEs) and deep metric learning to perform Bayesian optimisation (BO) over high-dimensional and structured input spaces. By adapting ideas from deep metric learning, we use label guidance from the blackbox function to structure the VAE latent space, facilitating the Gaussian process fit and yielding improved BO performance. Importantly for BO problem settings, our method operates in semi-supervised regimes where only few labelled data points are available. We run experiments on three real-world tasks, achieving state-of-the-art results on the penalised logP molecule generation benchmark using just 3% of the labelled data required by previous approaches. As a theoretical contribution, we present a proof of vanishing regret for VAE BO.
Deep Neural Network Modeling of Unknown Partial Differential Equations in Nodal Space
Authors: Zhen Chen, Victor Churchill, Kailiang Wu, Dongbin Xiu
Date: 2021-06-07 00:00:00

We present a numerical framework for deep neural network (DNN) modeling of unknown time-dependent partial differential equations (PDE) using their trajectory data. Unlike the recent work of [Wu and Xiu, J. Comput. Phys. 2020], where the learning takes place in modal/Fourier space, the current method conducts the learning and modeling in physical space and uses measurement data as nodal values. We present a DNN structure that has a direct correspondence to the evolution operator of the underlying PDE, thus establishing the existence of the DNN model. The DNN model also does not require any geometric information of the data nodes. Consequently, a trained DNN defines a predictive model for the underlying unknown PDE over structureless grids. A set of examples, including linear and nonlinear scalar PDE, system of PDEs, in both one dimension and two dimensions, over structured and unstructured grids, are presented to demonstrate the effectiveness of the proposed DNN modeling. Extension to other equations such as differential-integral equations is also discussed.
Beyond Bandit Feedback in Online Multiclass Classification
Authors: Dirk van der Hoeven, Federico Fusco, Nicolò Cesa-Bianchi
Date: 2021-06-07 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: 2k, Classification

We study the problem of online multiclass classification in a setting where the learner's feedback is determined by an arbitrary directed graph. While including bandit feedback as a special case, feedback graphs allow a much richer set of applications, including filtering and label efficient classification. We introduce Gappletron, the first online multiclass algorithm that works with arbitrary feedback graphs. For this new algorithm, we prove surrogate regret bounds that hold, both in expectation and with high probability, for a large class of surrogate losses. Our bounds are of order $B\sqrt{\rho KT}$, where $B$ is the diameter of the prediction space, $K$ is the number of classes, $T$ is the time horizon, and $\rho$ is the domination number (a graph-theoretic parameter affecting the amount of exploration). In the full information case, we show that Gappletron achieves a constant surrogate regret of order $B^2K$. We also prove a general lower bound of order $\max\big\{B^2K,\sqrt{T}\big\}$ showing that our upper bounds are not significantly improvable. Experiments on synthetic data show that for various feedback graphs, our algorithm is competitive against known baselines.
Learning Combinatorial Node Labeling Algorithms
Authors: Lukas Gianinazzi, Maximilian Fries, Nikoli Dryden, Tal Ben-Nun, Maciej Besta, Torsten Hoefler
Date: 2021-06-07 00:00:00
Tasks: BIG-bench Machine Learning, Graph Attention, Reinforcement Learning (RL)

We present a novel neural architecture to solve graph optimization problems where the solution consists of arbitrary node labels, allowing us to solve hard problems like graph coloring. We train our model using reinforcement learning, specifically policy gradients, which gives us both a greedy and a probabilistic policy. Our architecture builds on a graph attention network and uses several inductive biases to improve solution quality. Our learned deterministic heuristics for graph coloring give better solutions than classical degree-based greedy heuristics and only take seconds to apply to graphs with tens of thousands of vertices. Moreover, our probabilistic policies outperform all greedy state-of-the-art coloring baselines and a machine learning baseline. Finally, we show that our approach also generalizes to other problems by evaluating it on minimum vertex cover and outperforming two greedy heuristics.
Calibrating multi-dimensional complex ODE from noisy data via deep neural networks
Authors: Kexuan Li, Fangfang Wang, Ruiqi Liu, Fan Yang, Zuofeng Shang
Date: 2021-06-07 00:00:00

Ordinary differential equations (ODEs) are widely used to model complex dynamics that arises in biology, chemistry, engineering, finance, physics, etc. Calibration of a complicated ODE system using noisy data is generally very difficult. In this work, we propose a two-stage nonparametric approach to address this problem. We first extract the de-noised data and their higher order derivatives using boundary kernel method, and then feed them into a sparsely connected deep neural network with ReLU activation function. Our method is able to recover the ODE system without being subject to the curse of dimensionality and complicated ODE structure. When the ODE possesses a general modular structure, with each modular component involving only a few input variables, and the network architecture is properly chosen, our method is proven to be consistent. Theoretical properties are corroborated by an extensive simulation study that demonstrates the validity and effectiveness of the proposed method. Finally, we use our method to simultaneously characterize the growth rate of Covid-19 infection cases from 50 states of the USA.
Nonparametric adaptive control and prediction: theory and randomized algorithms
Authors: Nicholas M. Boffi, Stephen Tu, Jean-Jacques E. Slotine
Date: 2021-06-07 00:00:00
Tasks: Prediction

A key assumption in the theory of nonlinear adaptive control is that the uncertainty of the system can be expressed in the linear span of a set of known basis functions. While this assumption leads to efficient algorithms, it limits applications to very specific classes of systems. We introduce a novel nonparametric adaptive algorithm that estimates an infinite-dimensional density over parameters online to learn an unknown dynamics in a reproducing kernel Hilbert space. Surprisingly, the resulting control input admits an analytical expression that enables its implementation despite its underlying infinite-dimensional structure. While this adaptive input is rich and expressive - subsuming, for example, traditional linear parameterizations - its computational complexity grows linearly with time, making it comparatively more expensive than its parametric counterparts. Leveraging the theory of random Fourier features, we provide an efficient randomized implementation that recovers the complexity of classical parametric methods while provably retaining the expressivity of the nonparametric input. In particular, our explicit bounds only depend polynomially on the underlying parameters of the system, allowing our proposed algorithms to efficiently scale to high-dimensional systems. As an illustration of the method, we demonstrate the ability of the randomized approximation algorithm to learn a predictive model of a 60-dimensional system consisting of ten point masses interacting through Newtonian gravitation. By reinterpretation as a gradient flow on a specific loss, we conclude with a natural extension of our kernel-based adaptive algorithms to deep neural networks. We show empirically that the extra expressivity afforded by deep representations can lead to improved performance at the expense of closed-loop stability that is rigorously guaranteed and consistently observed for kernel machines.
Forward Looking Best-Response Multiplicative Weights Update Methods for Bilinear Zero-sum Games
Authors: Michail Fasoulakis, Evangelos Markakis, Yannis Pantazis, Constantinos Varsos
Date: 2021-06-07 00:00:00

Our work focuses on extra gradient learning algorithms for finding Nash equilibria in bilinear zero-sum games. The proposed method, which can be formally considered as a variant of Optimistic Mirror Descent \cite{DBLP:conf/iclr/MertikopoulosLZ19}, uses a large learning rate for the intermediate gradient step which essentially leads to computing (approximate) best response strategies against the profile of the previous iteration. Although counter-intuitive at first sight due to the irrationally large, for an iterative algorithm, intermediate learning step, we prove that the method guarantees last-iterate convergence to an equilibrium. Particularly, we show that the algorithm reaches first an $\eta^{1/\rho}$-approximate Nash equilibrium, with $\rho > 1$, by decreasing the Kullback-Leibler divergence of each iterate by at least $\Omega(\eta^{1+\frac{1}{\rho}})$, for sufficiently small learning rate, $\eta$, until the method becomes a contracting map, and converges to the exact equilibrium. Furthermore, we perform experimental comparisons with the optimistic variant of the multiplicative weights update method, by \cite{Daskalakis2019LastIterateCZ} and show that our algorithm has significant practical potential since it offers substantial gains in terms of accelerated convergence.
How Tight Can PAC-Bayes be in the Small Data Regime?
Authors: Andrew Y. K. Foong, Wessel P. Bruinsma, David R. Burt, Richard E. Turner
Date: 2021-06-07 00:00:00
Proceeding: NeurIPS 2021 12

In this paper, we investigate the question: Given a small number of datapoints, for example N = 30, how tight can PAC-Bayes and test set bounds be made? For such small datasets, test set bounds adversely affect generalisation performance by withholding data from the training procedure. In this setting, PAC-Bayes bounds are especially attractive, due to their ability to use all the data to simultaneously learn a posterior and bound its generalisation risk. We focus on the case of i.i.d. data with a bounded loss and consider the generic PAC-Bayes theorem of Germain et al. While their theorem is known to recover many existing PAC-Bayes bounds, it is unclear what the tightest bound derivable from their framework is. For a fixed learning algorithm and dataset, we show that the tightest possible bound coincides with a bound considered by Catoni; and, in the more natural case of distributions over datasets, we establish a lower bound on the best bound achievable in expectation. Interestingly, this lower bound recovers the Chernoff test set bound if the posterior is equal to the prior. Moreover, to illustrate how tight these bounds can be, we study synthetic one-dimensional classification tasks in which it is feasible to meta-learn both the prior and the form of the bound to numerically optimise for the tightest bounds possible. We find that in this simple, controlled scenario, PAC-Bayes bounds are competitive with comparable, commonly used Chernoff test set bounds. However, the sharpest test set bounds still lead to better guarantees on the generalisation error than the PAC-Bayes bounds we consider.
Graph Neural Networks in Network Neuroscience
Authors: Alaa Bessadok, Mohamed Ali Mahjoub, Islem Rekik
Date: 2021-06-07 00:00:00
Tasks: Graph Neural Network

Noninvasive medical neuroimaging has yielded many discoveries about the brain connectivity. Several substantial techniques mapping morphological, structural and functional brain connectivities were developed to create a comprehensive road map of neuronal activities in the human brain -namely brain graph. Relying on its non-Euclidean data type, graph neural network (GNN) provides a clever way of learning the deep graph structure and it is rapidly becoming the state-of-the-art leading to enhanced performance in various network neuroscience tasks. Here we review current GNN-based methods, highlighting the ways that they have been used in several applications related to brain graphs such as missing brain graph synthesis and disease classification. We conclude by charting a path toward a better application of GNN models in network neuroscience field for neurological disorder diagnosis and population graph integration. The list of papers cited in our work is available at https://github.com/basiralab/GNNs-in-Network-Neuroscience.
SizeFlags: Reducing Size and Fit Related Returns in Fashion E-Commerce
Authors: Andrea Nestler, Nour Karessli, Karl Hajjar, Rodrigo Weffer, Reza Shirvany
Date: 2021-06-07 00:00:00
Tasks: Articles

E-commerce is growing at an unprecedented rate and the fashion industry has recently witnessed a noticeable shift in customers' order behaviour towards stronger online shopping. However, fashion articles ordered online do not always find their way to a customer's wardrobe. In fact, a large share of them end up being returned. Finding clothes that fit online is very challenging and accounts for one of the main drivers of increased return rates in fashion e-commerce. Size and fit related returns severely impact 1. the customers experience and their dissatisfaction with online shopping, 2. the environment through an increased carbon footprint, and 3. the profitability of online fashion platforms. Due to poor fit, customers often end up returning articles that they like but do not fit them, which they have to re-order in a different size. To tackle this issue we introduce SizeFlags, a probabilistic Bayesian model based on weakly annotated large-scale data from customers. Leveraging the advantages of the Bayesian framework, we extend our model to successfully integrate rich priors from human experts feedback and computer vision intelligence. Through extensive experimentation, large-scale A/B testing and continuous evaluation of the model in production, we demonstrate the strong impact of the proposed approach in robustly reducing size-related returns in online fashion over 14 countries.
Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniques
Authors: Bokun Wang, Mher Safaryan, Peter Richtárik
Date: 2021-06-07 00:00:00
Tasks: BIG-bench Machine Learning, Distributed Optimization, Quantization

To address the high communication costs of distributed machine learning, a large body of work has been devoted in recent years to designing various compression strategies, such as sparsification and quantization, and optimization algorithms capable of using them. Recently, Safaryan et al. (2021) pioneered a dramatically different compression design approach: they first use the local training data to form local smoothness matrices and then propose to design a compressor capable of exploiting the smoothness information contained therein. While this novel approach leads to substantial savings in communication, it is limited to sparsification as it crucially depends on the linearity of the compression operator. In this work, we generalize their smoothness-aware compression strategy to arbitrary unbiased compression operators, which also include sparsification. Specializing our results to stochastic quantization, we guarantee significant savings in communication complexity compared to standard quantization. In particular, we prove that block quantization with $n$ blocks theoretically outperforms single block quantization, leading to a reduction in communication complexity by an $\mathcal{O}(n)$ factor, where $n$ is the number of nodes in the distributed system. Finally, we provide extensive numerical evidence with convex optimization problems that our smoothness-aware quantization strategies outperform existing quantization schemes as well as the aforementioned smoothness-aware sparsification strategies with respect to three evaluation metrics: the number of iterations, the total amount of bits communicated, and wall-clock time.
Top-KAST: Top-K Always Sparse Training
Authors: Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae, Simon Osindero, Erich Elsen
Date: 2021-06-07 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Language Modeling, Language Modelling

Sparse neural networks are becoming increasingly important as the field seeks to improve the performance of existing models by scaling them up, while simultaneously trying to reduce power consumption and computational footprint. Unfortunately, most existing methods for inducing performant sparse models still entail the instantiation of dense parameters, or dense gradients in the backward-pass, during training. For very large models this requirement can be prohibitive. In this work we propose Top-KAST, a method that preserves constant sparsity throughout training (in both the forward and backward-passes). We demonstrate the efficacy of our approach by showing that it performs comparably to or better than previous works when training models on the established ImageNet benchmark, whilst fully maintaining sparsity. In addition to our ImageNet results, we also demonstrate our approach in the domain of language modeling where the current best performing architectures tend to have tens of billions of parameters and scaling up does not yet seem to have saturated performance. Sparse versions of these architectures can be run with significantly fewer resources, making them more widely accessible and applicable. Furthermore, in addition to being effective, our approach is straightforward and can easily be implemented in a wide range of existing machine learning frameworks with only a few additional lines of code. We therefore hope that our contribution will help enable the broader community to explore the potential held by massive models, without incurring massive computational cost.