pwc-archive/papers-with-abstracts huggingface dataset:
Bijective-Contrastive Estimation
Authors: Jae Hyun Lim, Chin-wei Huang, Aaron Courville, Christopher Pal
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Classification

In this work, we propose Bijective-Contrastive Estimation (BCE), a classification-based learning criterion for energy-based models. We generate a collection of contrasting distributions using bijections, and solve all the classification problems between the original data distribution and the distributions induced by the bijections using a classifier parameterized by an energy model. We show that if the classification objective is minimized, the energy function will uniquely recover the data density up to a normalizing constant. This has the benefit of not having to explicitly specify a contrasting distribution, like noise contrastive estimation. Experimentally, we demonstrate that the proposed method works well on 2D synthetic datasets. We discuss the difficulty in high dimensional cases, and propose potential directions to explore for future work.
Parametric Bootstrap Ensembles as Variational Inference
Authors: Dimitrios Milios, Pietro Michiardi, Maurizio Filippone
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Bayesian Inference, Variational Inference

In this paper, we employ variational arguments to establish a connection between ensemble methods for Neural Networks and Bayesian inference. We consider an ensemble-based scheme where each model/particle corresponds to a perturbation of the data by means of parametric bootstrap and a perturbation of the prior. Our goal is to characterize the ensemble distribution in terms of the the Bayesian posterior. We derive conditions under which any optimization steps of the particles makes the associated distribution reduce its divergence to the posterior over model parameters.
Generalized Doubly-Reparameterized Gradient Estimators
Authors: Matthias Bauer, andriy mnih
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1

Efficient low-variance gradient estimation enabled by the reparameterization trick (RT) has been essential to the success of variational autoencoders. Doubly-reparameterized gradients (DReGs) improve on the RT for multi-sample variational bounds by applying reparameterization a second time for an additional reduction in variance. Here, we develop two generalizations of the DReGs estimator and show that they can be used to train conditional and hierarchical VAEs on image modelling tasks more effectively. We first extend the estimator to hierarchical models with several stochastic layers by showing how to treat additional score function terms due to the hierarchical variational posterior. We then generalize DReGs to score functions of arbitrary distributions instead of just those of the sampling distribution, which makes the estimator applicable to the parameters of the prior in addition to those of the posterior.
Functional Priors for Bayesian Neural Networks through Wasserstein Distance Minimization to Gaussian Processes
Authors: Ba-Hien Tran, Dimitrios Milios, Simone Rossi, Maurizio Filippone
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Gaussian Processes

The Bayesian treatment of neural networks dictates that a prior distribution is considered over the weight and bias parameters of the network. The non-linear nature of the model implies that any distribution of the parameters has an unpredictable effect on the distribution of the function output. Gaussian processes offer a rigorous framework to define prior distributions over the space of functions. Our proposal is to impose such functional priors on well-established architectures of neural networks by means of minimising the Wasserstein distance between samples of stochastic processes. Early experimental results demonstrate the potential of functional priors for Bayesian neural networks.
HWA: Hyperparameters Weight Averaging in Bayesian Neural Networks
Authors: Belhal Karimi, Ping Li
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1

Bayesian neural networks attempt to combine the strong predictive performance of neural networks with formal quantification of uncertainty of the predicted output in the Bayesian framework.
In deterministic deep neural network, the confidence of the model and the predictions at inference time are left alone. 
Applying randomness and Bayes Rule to the weights of a deep neural network is a step towards achieving this goal. 
Current state of the art optimization methods for training Bayesian Neural Networks are relatively slow and inefficient, compared to their deterministic counterparts. 
In this paper, we propose HWA (Hyperparameters Weight Averaging) algorithm that exploits an averaging procedure in order to optimize faster and achieve better accuracy. 
We develop our main algorithm using the simple averaging heuristic and demonstrate its effectiveness on the space of the hyperparameters of the networks random weights. 
Numerical applications are presented to confirm the empirical benefits of our method.
Conditional Input Gated Low-Rank Perturbations for Continual Learning
Authors: Anonymous
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Continual Learning, Split-MNIST

We address the problem of learning convolution neural networks (CNN) in the continual setting when tasks arrive sequentially, and only the data of the current task is available. In this setting CNNs are prone to reduce their quality on all old task drastically. In this work, we extend the idea of the \citep{abati} of data-conditional expansion of the CNN architecture. We propose to use low-rank and hence more weak additive perturbations for CNN filters, which is enough due to the composition structure of the CNN layers.  Such low-rank adaptation modules allow us to reduce computational costs and promote sparseness in the adaptation of CNN to new tasks. We validate our approach empirically on the split MNIST and CIFAR10 tasks.
Rethinking Function-Space Variational Inference in Bayesian Neural Networks
Authors: Tim G. J. Rudner, Zonghao Chen, Yarin Gal
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Variational Inference

Bayesian neural networks (BNNs) define distributions over functions induced by distributions over parameters.
In practice, this model specification makes it difficult to define and use meaningful prior distributions over functions that could aid in training. What's more, previous attempts at defining an explicit function-space variational objective for approximate inference in BNNs require approximations that do not scale to high-dimensional data. We propose a new function-space approach to variational inference in BNNs and derive a tractable variational by linearizing the BNN's posterior predictive distribution about its mean parameters, allowing function-space variational inference to be scaled to large and high-dimensional datasets. We evaluate this approach empirically and show that it leads to models with competitive predictive accuracy and significantly improved predictive uncertainty estimates compared to parameter-space variational inference.
Argmax Flows: Learning Categorical Distributions with Normalizing Flows
Authors: Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, Max Welling
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Image Segmentation, Semantic Segmentation

This paper introduces a new method to define and train continuous distributions such as normalizing flows directly on categorical data, for example text and image segmentation. The generative model is defined by a composition of a normalizing flow and an argmax function. To optimize this model, we dequantize the argmax using a distribution that is a probabilistic right-inverse to the argmax. This distribution lifts the categorical data to a continuous space on which the flow can be trained. We demonstrate that applying existing dequantization techniques naïvely to categorical data leads to suboptimal solutions. In addition, the model is fast both in generative (for sampling) and inference direction (for training), as opposed to autoregressive models.
Metro Passenger Flow Prediction via Dynamic Hypergraph Convolution Networks
Authors: Jingcheng Wang, Yong Zhang, Yun Wei, Yongli Hu, Xinglin Piao, BaoCai Yin
Date: 2021-04-21 00:00:00
Proceeding: IEEE Transactions on Intelligent Transportation Systems 2021 4
Tasks: Prediction

Metro passenger flow prediction is a strategically necessary demand in an intelligent transportation system to alleviate traffic pressure, coordinate operation schedules, and plan future constructions. Graph-based neural networks have been widely used in traffic flow prediction problems. Graph Convolutional Neural Networks (GCN) captures spatial features according to established connections but ignores the high-order relationships between stations and the travel patterns of passengers. In this paper, we utilize a novel representation to tackle this issue - hypergraph. A dynamic spatio-temporal hypergraph neural network to forecast passenger flow is proposed. In the prediction framework, the primary hypergraph is constructed from metro system topology and then extended with advanced hyperedges discovered from pedestrian travel patterns of multiple time spans. Furthermore, hypergraph convolution and spatiotemporal blocks are proposed to extract spatial and temporal features to achieve node-level prediction. Experiments on historical datasets of Beijing and Hangzhou validate the effectiveness of the proposed method, and superior performance of prediction accuracy is achieved compared with the state-of-the-arts.
Prediction of Prognosis and Survival of Patients with Gastric Cancer by Weighted Improved Random Forest Model
Authors: Cheng Xu, Jing Wang, TianLong Zheng, Yue Cao, Fan Ye
Date: 2021-04-10 00:00:00
Proceeding: Archives of Medical Science 2021 4
Tasks: Epidemiology, Prognosis

Introduction:
It’s very necessary to predict the survival status of patients based on their prognosis. This can assist physicians in evaluating treatment decisions. Random Forest is an excellent machine learning algorithm even without any modification. We propose a new Random Forest weighting method and apply it to the gastric cancer patient data from the Surveillance, Epidemiology, and End Results (SEER) program, and then evaluated the generalization ability of this weighted Random Forest algorithm on 10 public medical datasets. Furthermore, for the same weighting mode, the difference between using out-of-bag (OOB) data and all training sets as the weighting basis is explored.

Material and methods:
110697 cases of gastric cancer patients diagnosed between 1975 and 2016 obtained from the SEER database were contained in the experiment. In addition, 10 public medical datasets are used for the generalization ability evaluation of this weighted Random Forest algorithm.

Results:
Through experimental verification, on the SEER gastric cancer patient data, the weighted Random Forest algorithm improves the accuracy by 0.79% compared with the original Random Forest. In AUC, Macro-averaging increased by 2.32% and Micro-averaging increased by 0.51% on average. Among the 10 public datasets, the Random Forest weighted in accuracy has the best performance on 6 datasets, with an average increase of 1.44% in accuracy and an average increase of 1.2% in AUC.

Conclusions:
Compared with the original Random Forest, the weighted Random Forest model has a significant improvement in performance, and the effect of using all training data as the weighting basis is better than using OOB data.
Dimension Reduction with Prior Information for Knowledge Discovery
Authors: Anh Tuan Bui
Date: 2021-11-26 00:00:00
Tasks: Dimensionality Reduction

This paper addresses the problem of mapping high-dimensional data to a low-dimensional space, in the presence of other known features. This problem is ubiquitous in science and engineering as there are often controllable/measurable features in most applications. To solve this problem, this paper proposes a broad class of methods, which is referred to as conditional multidimensional scaling (MDS). An algorithm for optimizing the objective function of conditional MDS is also developed. The convergence of this algorithm is proven under mild assumptions. Conditional MDS is illustrated with kinship terms, facial expressions, textile fabrics, car-brand perception, and cylinder machining examples. These examples demonstrate the advantages of conditional MDS over conventional dimension reduction in improving the estimation quality of the reduced-dimension space and simplifying visualization and knowledge discovery tasks. Computer codes for this work are available in the open-source cml R package.
ManiFest: Manifold Deformation for Few-shot Image Translation
Authors: Fabio Pizzati, Jean-François Lalonde, Raoul de Charette
Date: 2021-11-26 00:00:00
Tasks: Image-to-Image Translation, Translation

Most image-to-image translation methods require a large number of training images, which restricts their applicability. We instead propose ManiFest: a framework for few-shot image translation that learns a context-aware representation of a target domain from a few images only. To enforce feature consistency, our framework learns a style manifold between source and proxy anchor domains (assumed to be composed of large numbers of images). The learned manifold is interpolated and deformed towards the few-shot target domain via patch-based adversarial and feature statistics alignment losses. All of these components are trained simultaneously during a single end-to-end loop. In addition to the general few-shot translation task, our approach can alternatively be conditioned on a single exemplar image to reproduce its specific style. Extensive experiments demonstrate the efficacy of ManiFest on multiple tasks, outperforming the state-of-the-art on all metrics and in both the general- and exemplar-based scenarios. Our code is available at https://github.com/cv-rits/Manifest .
Amazon SageMaker Model Monitor: A System for Real-Time Insights into Deployed Machine Learning Models
Authors: David Nigenda, Zohar Karnin, Muhammad Bilal Zafar, Raghu Ramesha, Alan Tan, Michele Donini, Krishnaram Kenthapadi
Date: 2021-11-26 00:00:00
Tasks: BIG-bench Machine Learning

With the increasing adoption of machine learning (ML) models and systems in high-stakes settings across different industries, guaranteeing a model's performance after deployment has become crucial. Monitoring models in production is a critical aspect of ensuring their continued performance and reliability. We present Amazon SageMaker Model Monitor, a fully managed service that continuously monitors the quality of machine learning models hosted on Amazon SageMaker. Our system automatically detects data, concept, bias, and feature attribution drift in models in real-time and provides alerts so that model owners can take corrective actions and thereby maintain high quality models. We describe the key requirements obtained from customers, system design and architecture, and methodology for detecting different types of drift. Further, we provide quantitative evaluations followed by use cases, insights, and lessons learned from more than two years of production deployment.
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs
Authors: Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, Srinivasan Iyer
Date: 2021-11-26 00:00:00

Do language models have beliefs about the world? Dennett (1995) famously argues that even thermostats have beliefs, on the view that a belief is simply an informational state decoupled from any motivational state. In this paper, we discuss approaches to detecting when models have beliefs about the world, and we improve on methods for updating model beliefs to be more truthful, with a focus on methods based on learned optimizers or hypernetworks. Our main contributions include: (1) new metrics for evaluating belief-updating methods that focus on the logical consistency of beliefs, (2) a training objective for Sequential, Local, and Generalizing model updates (SLAG) that improves the performance of learned optimizers, and (3) the introduction of the belief graph, which is a new form of interface with language models that shows the interdependencies between model beliefs. Our experiments suggest that models possess belief-like qualities to only a limited extent, but update methods can both fix incorrect model beliefs and greatly improve their consistency. Although off-the-shelf optimizers are surprisingly strong belief-updating baselines, our learned optimizers can outperform them in more difficult settings than have been considered in past work. Code is available at https://github.com/peterbhase/SLAG-Belief-Updating
Latent Space Smoothing for Individually Fair Representations
Authors: Momchil Peychev, Anian Ruoss, Mislav Balunović, Maximilian Baader, Martin Vechev
Date: 2021-11-26 00:00:00
Tasks: Fairness, Representation Learning

Fair representation learning transforms user data into a representation that ensures fairness and utility regardless of the downstream application. However, learning individually fair representations, i.e., guaranteeing that similar individuals are treated similarly, remains challenging in high-dimensional settings such as computer vision. In this work, we introduce LASSI, the first representation learning method for certifying individual fairness of high-dimensional data. Our key insight is to leverage recent advances in generative modeling to capture the set of similar individuals in the generative latent space. This enables us to learn individually fair representations that map similar individuals close together by using adversarial training to minimize the distance between their representations. Finally, we employ randomized smoothing to provably map similar individuals close together, in turn ensuring that local robustness verification of the downstream application results in end-to-end fairness certification. Our experimental evaluation on challenging real-world image data demonstrates that our method increases certified individual fairness by up to 90% without significantly affecting task utility.
DP-SGD vs PATE: Which Has Less Disparate Impact on GANs?
Authors: Georgi Ganev
Date: 2021-11-26 00:00:00

Generative Adversarial Networks (GANs) are among the most popular approaches to generate synthetic data, especially images, for data sharing purposes. Given the vital importance of preserving the privacy of the individual data points in the original data, GANs are trained utilizing frameworks with robust privacy guarantees such as Differential Privacy (DP). However, these approaches remain widely unstudied beyond single performance metrics when presented with imbalanced datasets. To this end, we systematically compare GANs trained with the two best-known DP frameworks for deep learning, DP-SGD, and PATE, in different data imbalance settings from two perspectives -- the size of the classes in the generated synthetic data and their classification performance. Our analyses show that applying PATE, similarly to DP-SGD, has a disparate effect on the under/over-represented classes but in a much milder magnitude making it more robust. Interestingly, our experiments consistently show that for PATE, unlike DP-SGD, the privacy-utility trade-off is not monotonically decreasing but is much smoother and inverted U-shaped, meaning that adding a small degree of privacy actually helps generalization. However, we have also identified some settings (e.g., large imbalance) where PATE-GAN completely fails to learn some subparts of the training data.
Predicting Document Coverage for Relation Extraction
Authors: Sneha Singhania, Simon Razniewski, Gerhard Weikum
Date: 2021-11-26 00:00:00
Tasks: Information Retrieval, Knowledge Base Construction, Relation, Relation Extraction, Retrieval

This paper presents a new task of predicting the coverage of a text document for relation extraction (RE): does the document contain many relational tuples for a given entity? Coverage predictions are useful in selecting the best documents for knowledge base construction with large input corpora. To study this problem, we present a dataset of 31,366 diverse documents for 520 entities. We analyze the correlation of document coverage with features like length, entity mention frequency, Alexa rank, language complexity and information retrieval scores. Each of these features has only moderate predictive power. We employ methods combining features with statistical models like TF-IDF and language models like BERT. The model combining features and BERT, HERB, achieves an F1 score of up to 46%. We demonstrate the utility of coverage predictions on two use cases: KB construction and claim refutation.
Asian Giant Hornet Control based on Image Processing and Biological Dispersal
Authors: Changjie Lu, Shen Zheng, Hailu Qiu
Date: 2021-11-26 00:00:00
Tasks: Feature Importance

The Asian giant hornet (AGH) appeared in Washington State appears to have a potential danger of bioinvasion. Washington State has collected public photos and videos of detected insects for verification and further investigation. In this paper, we analyze AGH using data analysis,statistics, discrete mathematics, and deep learning techniques to process the data to controlAGH spreading.First, we visualize the geographical distribution of insects in Washington State. Then we investigate insect populations to varying months of the year and different days of a month.Third, we employ wavelet analysis to examine the periodic spread of AGH. Fourth, we apply ordinary differential equations to examine AGH numbers at the different natural growthrate and reaction speed and output the potential propagation coefficient. Next, we leverage cellular automaton combined with the potential propagation coefficient to simulate the geographical spread under changing potential propagation. To update the model, we use delayed differential equations to simulate human intervention. We use the time difference between detection time and submission time to determine the unit of time to delay time. After that, we construct a lightweight CNN called SqueezeNet and assess its classification performance. We then relate several non-reference image quality metrics, including NIQE, image gradient, entropy, contrast, and TOPSIS to judge the cause of misclassification. Furthermore, we build a Random Forest classifier to identify positive and negative samples based on image qualities only. We also display the feature importance and conduct an error analysis. Besides, we present sensitivity analysis to verify the robustness of our models. Finally, we show the strengths and weaknesses of our model and derives the conclusions.
Using Fictitious Class Representations to Boost Discriminative Zero-Shot Learners
Authors: Mohammed Dabbah, Ran El-Yaniv
Date: 2021-11-26 00:00:00
Tasks: Attribute, Generalized Zero-Shot Learning, Zero-Shot Learning

Focusing on discriminative zero-shot learning, in this work we introduce a novel mechanism that dynamically augments during training the set of seen classes to produce additional fictitious classes. These fictitious classes diminish the model's tendency to fixate during training on attribute correlations that appear in the training set but will not appear in newly exposed classes. The proposed model is tested within the two formulations of the zero-shot learning framework; namely, generalized zero-shot learning (GZSL) and classical zero-shot learning (CZSL). Our model improves the state-of-the-art performance on the CUB dataset and reaches comparable results on the other common datasets, AWA2 and SUN. We investigate the strengths and weaknesses of our method, including the effects of catastrophic forgetting when training an end-to-end zero-shot model.