pwc-archive/papers-with-abstracts huggingface dataset:
Proactive DP: A Multple Target Optimization Framework for DP-SGD
Authors: Marten van Dijk, Nhuong V. Nguyen, Toan N. Nguyen, Lam M. Nguyen, Phuong Ha Nguyen
Date: 2021-02-17 00:00:00
Tasks: 2k

We introduce a multiple target optimization framework for DP-SGD referred to as pro-active DP. In contrast to traditional DP accountants, which are used to track the expenditure of privacy budgets, the pro-active DP scheme allows one to {\it a-priori} select parameters of DP-SGD based on a fixed privacy budget (in terms of $\epsilon$ and $\delta$) in such a way to optimize the anticipated utility (test accuracy) the most. To achieve this objective, we first propose significant improvements to the moment account method, presenting a closed-form $(\epsilon,\delta)$-DP guarantee that connects all parameters in the DP-SGD setup. Generally, DP-SGD is $(\epsilon\leq 1/2,\delta=1/N)$-DP if $\sigma=\sqrt{2(\epsilon +\ln(1/\delta))/\epsilon}$ with $T$ at least $\approx 2k^2/\epsilon$ and $(2/e)^2k^2-1/2\geq \ln(N)$, where $T$ is the total number of rounds, and $K=kN$ is the total number of gradient computations where $k$ measures $K$ in number of epochs of size $N$ of the local data set. We prove that our expression is close to tight in that if $T$ is more than a constant factor $\approx 4$ smaller than the lower bound $\approx 2k^2/\epsilon$, then the $(\epsilon,\delta)$-DP guarantee is violated. Our enhanced DP theory allows us to create a utility graph and DP calculator. These tools link privacy and utility objectives and search for optimal experiment setups, efficiently taking into account both accuracy and privacy objectives, as well as implementation goals. We furnish a comprehensive implementation flow of our proactive DP, with rigorous experiments to showcase the proof-of-concept.
Joint Continuous and Discrete Model Selection via Submodularity
Authors: Jonathan Bunton, Paulo Tabuada
Date: 2021-02-17 00:00:00
Tasks: model, Model Selection

In model selection problems for machine learning, the desire for a well-performing model with meaningful structure is typically expressed through a regularized optimization problem. In many scenarios, however, the meaningful structure is specified in some discrete space, leading to difficult nonconvex optimization problems. In this paper, we connect the model selection problem with structure-promoting regularizers to submodular function minimization with continuous and discrete arguments. In particular, we leverage the theory of submodular functions to identify a class of these problems that can be solved exactly and efficiently with an agnostic combination of discrete and continuous optimization routines. We show how simple continuous or discrete constraints can also be handled for certain problem classes and extend these ideas to a robust optimization framework. We also show how some problems outside of this class can be embedded within the class, further extending the class of problems our framework can accommodate. Finally, we numerically validate our theoretical results with several proof-of-concept examples with synthetic and real-world data, comparing against state-of-the-art algorithms.
Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm
Authors: Bin Gu, Guodong Liu, yanfu Zhang, Xiang Geng, Heng Huang
Date: 2021-02-17 00:00:00
Tasks: Hyperparameter Optimization

Modern machine learning algorithms usually involve tuning multiple (from one to thousands) hyperparameters which play a pivotal role in terms of model generalizability. Black-box optimization and gradient-based algorithms are two dominant approaches to hyperparameter optimization while they have totally distinct advantages. How to design a new hyperparameter optimization technique inheriting all benefits from both approaches is still an open problem. To address this challenging problem, in this paper, we propose a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG). Specifically, we first exactly formulate hyperparameter optimization as an A-based constrained optimization problem, where A is a black-box optimization algorithm (such as deep neural network). Then, we use the average zeroth-order hyper-gradients to update hyperparameters. We provide the feasibility analysis of using HOZOG to achieve hyperparameter optimization. Finally, the experimental results on three representative hyperparameter (the size is from 1 to 1250) optimization tasks demonstrate the benefits of HOZOG in terms of simplicity, scalability, flexibility, effectiveness and efficiency compared with the state-of-the-art hyperparameter optimization methods.
Deep Learning Approaches for Forecasting Strawberry Yields and Prices Using Satellite Images and Station-Based Soil Parameters
Authors: Mohita Chaudhary, Mohamed Sadok Gastli, Lobna Nassar, Fakhri Karray
Date: 2021-02-17 00:00:00
Tasks: Deep Learning, Time Series Analysis

Computational tools for forecasting yields and prices for fresh produce have been based on traditional machine learning approaches or time series modelling. We propose here an alternate approach based on deep learning algorithms for forecasting strawberry yields and prices in Santa Barbara county, California. Building the proposed forecasting model comprises three stages: first, the station-based ensemble model (ATT-CNN-LSTM-SeriesNet_Ens) with its compound deep learning components, SeriesNet with Gated Recurrent Unit (GRU) and Convolutional Neural Network LSTM with Attention layer (Att-CNN-LSTM), are trained and tested using the station-based soil temperature and moisture data of SantaBarbara as input and the corresponding strawberry yields or prices as output. Secondly, the remote sensing ensemble model (SIM_CNN-LSTM_Ens), which is an ensemble model of Convolutional NeuralNetwork LSTM (CNN-LSTM) models, is trained and tested using satellite images of the same county as input mapped to the same yields and prices as output. These two ensembles forecast strawberry yields and prices with minimal forecasting errors and highest model correlation for five weeks ahead forecasts.Finally, the forecasts of these two models are ensembled to have a final forecasted value for yields and prices by introducing a voting ensemble. Based on an aggregated performance measure (AGM), it is found that this voting ensemble not only enhances the forecasting performance by 5% compared to its best performing component model but also outperforms the Deep Learning (DL) ensemble model found in literature by 33% for forecasting yields and 21% for forecasting prices
Estimate Three-Phase Distribution Line Parameters With Physics-Informed Graphical Learning Method
Authors: Wenyu Wang, Nanpeng Yu
Date: 2021-02-17 00:00:00

Accurate estimates of network parameters are essential for modeling, monitoring, and control in power distribution systems. In this paper, we develop a physics-informed graphical learning algorithm to estimate network parameters of three-phase power distribution systems. Our proposed algorithm uses only readily available smart meter data to estimate the three-phase series resistance and reactance of the primary distribution line segments. We first develop a parametric physics-based model to replace the black-box deep neural networks in the conventional graphical neural network (GNN). Then we derive the gradient of the loss function with respect to the network parameters and use stochastic gradient descent (SGD) to estimate the physical parameters. Prior knowledge of network parameters is also considered to further improve the accuracy of estimation. Comprehensive numerical study results show that our proposed algorithm yields high accuracy and outperforms existing methods.
Improving Hierarchical Adversarial Robustness of Deep Neural Networks
Authors: Avery Ma, Aladin Virmaux, Kevin Scaman, Juwei Lu
Date: 2021-02-17 00:00:00
Tasks: Adversarial Defense, Adversarial Robustness, Autonomous Driving, General Classification

Do all adversarial examples have the same consequences? An autonomous driving system misclassifying a pedestrian as a car may induce a far more dangerous -- and even potentially lethal -- behavior than, for instance, a car as a bus. In order to better tackle this important problematic, we introduce the concept of hierarchical adversarial robustness. Given a dataset whose classes can be grouped into coarse-level labels, we define hierarchical adversarial examples as the ones leading to a misclassification at the coarse level. To improve the resistance of neural networks to hierarchical attacks, we introduce a hierarchical adversarially robust (HAR) network design that decomposes a single classification task into one coarse and multiple fine classification tasks, before being specifically trained by adversarial defense techniques. As an alternative to an end-to-end learning approach, we show that HAR significantly improves the robustness of the network against $\ell_2$ and $\ell_{\infty}$ bounded hierarchical attacks on the CIFAR-10 and CIFAR-100 dataset.
BORE: Bayesian Optimization by Density-Ratio Estimation
Authors: Louis C. Tiao, Aaron Klein, Matthias Seeger, Edwin V. Bonilla, Cedric Archambeau, Fabio Ramos
Date: 2021-02-17 00:00:00
Tasks: Bayesian Optimization, Binary Classification, Density Ratio Estimation

Bayesian optimization (BO) is among the most effective and widely-used blackbox optimization methods. BO proposes solutions according to an explore-exploit trade-off criterion encoded in an acquisition function, many of which are computed from the posterior predictive of a probabilistic surrogate model. Prevalent among these is the expected improvement (EI) function. The need to ensure analytical tractability of the predictive often poses limitations that can hinder the efficiency and applicability of BO. In this paper, we cast the computation of EI as a binary classification problem, building on the link between class-probability estimation and density-ratio estimation, and the lesser-known link between density-ratios and EI. By circumventing the tractability constraints, this reformulation provides numerous advantages, not least in terms of expressiveness, versatility, and scalability.
An Efficient Diagnosis Algorithm for Inconsistent Constraint Sets
Authors: Alexander Felfernig, Monika Schubert, Christoph Zehentner
Date: 2021-02-17 00:00:00

Constraint sets can become inconsistent in different contexts. For example, during a configuration session the set of customer requirements can become inconsistent with the configuration knowledge base. Another example is the engineering phase of a configuration knowledge base where the underlying constraints can become inconsistent with a set of test cases. In such situations we are in the need of techniques that support the identification of minimal sets of faulty constraints that have to be deleted in order to restore consistency. In this paper we introduce a divide-and-conquer based diagnosis algorithm (FastDiag) which identifies minimal sets of faulty constraints in an over-constrained problem. This algorithm is specifically applicable in scenarios where the efficient identification of leading (preferred) diagnoses is crucial. We compare the performance of FastDiag with the conflict-directed calculation of hitting sets and present an in-depth performance analysis that shows the advantages of our approach.
Domain Impression: A Source Data Free Domain Adaptation Method
Authors: Vinod K Kurmi, Venkatesh K Subramanian, Vinay P Namboodiri
Date: 2021-02-17 00:00:00
Tasks: Domain Adaptation, Unsupervised Domain Adaptation

Unsupervised Domain adaptation methods solve the adaptation problem for an unlabeled target set, assuming that the source dataset is available with all labels. However, the availability of actual source samples is not always possible in practical cases. It could be due to memory constraints, privacy concerns, and challenges in sharing data. This practical scenario creates a bottleneck in the domain adaptation problem. This paper addresses this challenging scenario by proposing a domain adaptation technique that does not need any source data. Instead of the source data, we are only provided with a classifier that is trained on the source data. Our proposed approach is based on a generative framework, where the trained classifier is used for generating samples from the source classes. We learn the joint distribution of data by using the energy-based modeling of the trained classifier. At the same time, a new classifier is also adapted for the target domain. We perform various ablation analysis under different experimental setups and demonstrate that the proposed approach achieves better results than the baseline models in this extremely novel scenario.
Mobile Computational Photography: A Tour
Authors: Mauricio Delbracio, Damien Kelly, Michael S. Brown, Peyman Milanfar
Date: 2021-02-17 00:00:00
Tasks: Super-Resolution

The first mobile camera phone was sold only 20 years ago, when taking pictures with one's phone was an oddity, and sharing pictures online was unheard of. Today, the smartphone is more camera than phone. How did this happen? This transformation was enabled by advances in computational photography -the science and engineering of making great images from small form factor, mobile cameras. Modern algorithmic and computing advances, including machine learning, have changed the rules of photography, bringing to it new modes of capture, post-processing, storage, and sharing. In this paper, we give a brief history of mobile computational photography and describe some of the key technological components, including burst photography, noise reduction, and super-resolution. At each step, we may draw naive parallels to the human visual system.
One-shot action recognition in challenging therapy scenarios
Authors: Alberto Sabater, Laura Santos, Jose Santos-Victor, Alexandre Bernardino, Luis Montesano, Ana C. Murillo
Date: 2021-02-17 00:00:00
Tasks: Action Recognition, One-Shot 3D Action Recognition

One-shot action recognition aims to recognize new action categories from a single reference example, typically referred to as the anchor example. This work presents a novel approach for one-shot action recognition in the wild that computes motion representations robust to variable kinematic conditions. One-shot action recognition is then performed by evaluating anchor and target motion representations. We also develop a set of complementary steps that boost the action recognition performance in the most challenging scenarios. Our approach is evaluated on the public NTU-120 one-shot action recognition benchmark, outperforming previous action recognition models. Besides, we evaluate our framework on a real use-case of therapy with autistic people. These recordings are particularly challenging due to high-level artifacts from the patient motion. Our results provide not only quantitative but also online qualitative measures, essential for the patient evaluation and monitoring during the actual therapy.
Using Distance Correlation for Efficient Bayesian Optimization
Authors: Takuya Kanazawa
Date: 2021-02-17 00:00:00
Tasks: Bayesian Optimization, Gaussian Processes

The need to collect data via expensive measurements of black-box functions is prevalent across science, engineering and medicine. As an example, hyperparameter tuning of a large AI model is critical to its predictive performance but is generally time-consuming and unwieldy. Bayesian optimization (BO) is a collection of methods that aim to address this issue by means of Bayesian statistical inference. In this work, we put forward a BO scheme named BDC, which integrates BO with a statistical measure of association of two random variables called Distance Correlation. BDC balances exploration and exploitation automatically, and requires no manual hyperparameter tuning. We evaluate BDC on a range of benchmark tests and observe that it performs on per with popular BO methods such as the expected improvement and max-value entropy search. We also apply BDC to optimization of sequential integral observations of an unknown terrain and confirm its utility.
Generalization in Quantum Machine Learning: a Quantum Information Perspective
Authors: Leonardo Banchi, Jason Pereira, Stefano Pirandola
Date: 2021-02-17 00:00:00
Tasks: BIG-bench Machine Learning, Classification, Quantum Machine Learning

Quantum classification and hypothesis testing are two tightly related subjects, the main difference being that the former is data driven: how to assign to quantum states $\rho(x)$ the corresponding class $c$ (or hypothesis) is learnt from examples during training, where $x$ can be either tunable experimental parameters or classical data "embedded" into quantum states. Does the model generalize? This is the main question in any data-driven strategy, namely the ability to predict the correct class even of previously unseen states. Here we establish a link between quantum machine learning classification and quantum hypothesis testing (state and channel discrimination) and then show that the accuracy and generalization capability of quantum classifiers depend on the (R\'enyi) mutual informations $I(C{:}Q)$ and $I_2(X{:}Q)$ between the quantum state space $Q$ and the classical parameter space $X$ or class space $C$. Based on the above characterization, we then show how different properties of $Q$ affect classification accuracy and generalization, such as the dimension of the Hilbert space, the amount of noise, and the amount of neglected information from $X$ via, e.g., pooling layers. Moreover, we introduce a quantum version of the Information Bottleneck principle that allows us to explore the various tradeoffs between accuracy and generalization. Finally, in order to check our theoretical predictions, we study the classification of the quantum phases of an Ising spin chain, and we propose the Variational Quantum Information Bottleneck (VQIB) method to optimize quantum embeddings of classical data to favor generalization.
BEDS: Bagging ensemble deep segmentation for nucleus segmentation with testing stage stain augmentation
Authors: Xing Li, Haichun Yang, Jiaxin He, Aadarsh Jha, Agnes B. Fogo, Lee E. Wheless, Shilin Zhao, Yuankai Huo
Date: 2021-02-17 00:00:00
Tasks: Deep Learning, Ensemble Learning, Medical Image Analysis, Segmentation

Reducing outcome variance is an essential task in deep learning based medical image analysis. Bootstrap aggregating, also known as bagging, is a canonical ensemble algorithm for aggregating weak learners to become a strong learner. Random forest is one of the most powerful machine learning algorithms before deep learning era, whose superior performance is driven by fitting bagged decision trees (weak learners). Inspired by the random forest technique, we propose a simple bagging ensemble deep segmentation (BEDs) method to train multiple U-Nets with partial training data to segment dense nuclei on pathological images. The contributions of this study are three-fold: (1) developing a self-ensemble learning framework for nucleus segmentation; (2) aggregating testing stage augmentation with self-ensemble learning; and (3) elucidating the idea that self-ensemble and testing stage stain augmentation are complementary strategies for a superior segmentation performance. Implementation Detail: https://github.com/xingli1102/BEDs.
Automated Detection of Equine Facial Action Units
Authors: Zhenghong Li, Sofia Broomé, Pia Haubro Andersen, Hedvig Kjellström
Date: 2021-02-17 00:00:00
Tasks: Facial Action Unit Detection

The recently developed Equine Facial Action Coding System (EquiFACS) provides a precise and exhaustive, but laborious, manual labelling method of facial action units of the horse. To automate parts of this process, we propose a Deep Learning-based method to detect EquiFACS units automatically from images. We use a cascade framework; we firstly train several object detectors to detect the predefined Region-of-Interest (ROI), and secondly apply binary classifiers for each action unit in related regions. We experiment with both regular CNNs and a more tailored model transferred from human facial action unit recognition. Promising initial results are presented for nine action units in the eye and lower face regions. Code for the project is publicly available.
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Authors: Soravit Changpinyo, Piyush Sharma, Nan Ding, Radu Soricut
Date: 2021-02-17 00:00:00
Proceeding: CVPR 2021 1
Tasks: Caption Generation, Diversity, Image Captioning, Question Answering, Visual Question Answering, Visual Question Answering (VQA)

The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pre-training. However, these datasets are often collected with overrestrictive requirements inherited from their original target tasks (e.g., image caption generation), which limit the resulting dataset scale and diversity. We take a step further in pushing the limits of vision-and-language pre-training data by relaxing the data collection pipeline used in Conceptual Captions 3M (CC3M) [Sharma et al. 2018] and introduce the Conceptual 12M (CC12M), a dataset with 12 million image-text pairs specifically meant to be used for vision-and-language pre-training. We perform an analysis of this dataset and benchmark its effectiveness against CC3M on multiple downstream tasks with an emphasis on long-tail visual recognition. Our results clearly illustrate the benefit of scaling up pre-training data for vision-and-language tasks, as indicated by the new state-of-the-art results on both the nocaps and Conceptual Captions benchmarks.
EEG-based Texture Roughness Classification in Active Tactile Exploration with Invariant Representation Learning Networks
Authors: Ozan Ozdenizci, Safaa Eldeeb, Andac Demir, Deniz Erdogmus, Murat Akcakaya
Date: 2021-02-17 00:00:00
Tasks: EEG, Electroencephalogram (EEG), General Classification, Representation Learning

During daily activities, humans use their hands to grasp surrounding objects and perceive sensory information which are also employed for perceptual and motor goals. Multiple cortical brain regions are known to be responsible for sensory recognition, perception and motor execution during sensorimotor processing. While various research studies particularly focus on the domain of human sensorimotor control, the relation and processing between motor execution and sensory processing is not yet fully understood. Main goal of our work is to discriminate textured surfaces varying in their roughness levels during active tactile exploration using simultaneously recorded electroencephalogram (EEG) data, while minimizing the variance of distinct motor exploration movement patterns. We perform an experimental study with eight healthy participants who were instructed to use the tip of their dominant hand index finger while rubbing or tapping three different textured surfaces with varying levels of roughness. We use an adversarial invariant representation learning neural network architecture that performs EEG-based classification of different textured surfaces, while simultaneously minimizing the discriminability of motor movement conditions (i.e., rub or tap). Results show that the proposed approach can discriminate between three different textured surfaces with accuracies up to 70%, while suppressing movement related variability from learned representations.