pwc-archive/papers-with-abstracts huggingface dataset:
Bayesian Imbalanced Regression Debiasing
Authors: Jiawei Ren, Mingyuan Zhang, Cunjun Yu, Ziwei Liu
Date: 2021-09-29 00:00:00
Tasks: Age Estimation, imbalanced classification, Pose Estimation, regression

Imbalanced regression, where the training data has an uneven distribution on its range, is widely encountered in the real world, e.g., age estimation (uni-dimensional regression) and pose estimation (multi-dimensional regression). Compared to imbalanced and long-tailed classification, imbalanced regression has its unique challenges as the regression label space can be continuous, boundless, and high-dimensional. In this work, we present a principled framework, Bayesian Posterior Debiasing (Bayesian-PD), for re-balancing the regression among frequent and rare observations. Our key insight is that a balanced posterior can be obtained by debiasing the conditional probability with a regression label space prior. Importantly, through a normalization reparameterization technique, we derive a general debiasing function between the empirical posterior and the balanced posterior without relying on task-specific assumptions. We show that the Bayesian-PD framework has multiple instantiations in both training and testing time, with either closed-form or numerical implementations. We further uncover that several existing methods in imbalanced classification/regression serve as special cases of our Bayesian-PD framework. Extensive experiments on both uni- and multi-dimensional regression benchmarks demonstrate the effectiveness of the Bayesian-PD framework on various real-world tasks. Notably, Bayesian-PD exhibits strong robustness to different skewness of the training distributions.
Agnostic Personalized Federated Learning with Kernel Factorization
Authors: Wonyong Jeong, Sung Ju Hwang
Date: 2021-09-29 00:00:00
Tasks: Federated Learning, Personalized Federated Learning, Semantic Similarity, Semantic Textual Similarity

Considering the futuristic scenarios of federated learning at a worldwide scale, it is highly probable that local participants can have their own personalized labels, which might not be compatible with each other even for the same class, and can be also possibly from a variety of multiple domains. Nevertheless, they should be benefited from others while selectively taking helpful knowledge. Toward such extreme scenarios of federated learning, however, most existing approaches are limited in that they often assume: (1) labeling schemes are all synchronized amongst clients; (2) the local data is from the same single dataset (domain). In this sense, we introduce an intensively realistic problem of federated learning, namely Agnostic Personalized Federated Learning (APFL), where any clients, regardless of what they have learned with their personalized labels, can collaboratively learn while benefiting each other. We then study two essential challenges of the agnostic personalized federated learning, which are (1) Label Heterogeneity where local clients learn from the same single domain but labeling schemes are not synchronized with each other and (2) Domain Heterogeneity where the clients learn from the different datasets which can be semantically similar or dissimilar for each other. To tackle these problems, we propose our novel method, namely Similarity Matching and Kernel Factorization (SimFed). Our method measures semantic similarity/dissimilarity between locally learned knowledge and matches/aggregates the relevant ones that are beneficial to each other. Furthermore, we factorize our model parameters into two basis vectors and the sparse masks to effectively capture permutation-robust representations and reduce information loss when aggregating the heterogeneous knowledge. We exhaustively validate our method on both single- and multi-domain datasets, showing that our method outperforms the current state-of-the-art federated learning methods.
Multi-objective optimization for Hardware-aware Neural Architecture Search
Authors: Taehee Jeong, Elliott Delaye
Date: 2021-09-29 00:00:00
Tasks: CPU, Hardware Aware Neural Architecture Search, Neural Architecture Search

Hardware-aware Neural Architecture Search (HW-NAS) has been drawing increasing attention since it can automatically design deep neural networks optimized in a resource-constrained device. However, existing methods may be not optimal in terms of multi-object (accuracy, hardware-metrics). Thus, we propose a new multi-objective optimization method for searching promising architectures in HW-NAS. Our method addresses the architecture selection process in NAS. An architecture population is divided to small cells by given hardware-cost metrics, then, top-ranked architecture is selected within each cell. The selected ones play knobs to guide the direction of evolution in NAS. Despite its simplicity, this method leads to promising results, improving both accuracy and hardware metrics. Using latency as a hardware metric, we demonstrated that the optimized architecture extended its top accuracy to a much lower inference latency regime. We can also significantly reduce computing cost of search using both accuracy predictor and latency estimator and sharing pre-trained weights of super-network. This makes HW-NAS research more reproducible and accessible to the public. For a target hardware, we experimented on both CPU and Field Programmable Gate Array (FPGA). The codes are available at https://anonymous.4open.science/r/multi-objective-optimization-0E27/README.md.
Linear algebra with transformers
Authors: Francois Charton
Date: 2021-09-29 00:00:00
Tasks: Automated Theorem Proving, Few-Shot Learning

Most applications of transformers to mathematics, from integration to theorem proving, focus on symbolic computation. In this paper, we show that transformers can be trained to perform numerical calculations with high accuracy. We consider problems of linear algebra:  matrix transposition, addition, multiplication, eigenvalues and vectors, singular value decomposition, and inversion. Training small transformers (up to six layers) over datasets of random matrices, we achieve high accuracies (over 90%) on all problems. We also show that trained models can generalize out of their training distribution, and that out-of-domain accuracy can be greatly improved by working from more diverse datasets (in particular, by training from matrices with non-independent and identically distributed coefficients). Finally, we show that few-shot learning can be leveraged to retrain models to solve larger problems.
Fast Deterministic Stackelberg Actor-Critic
Authors: Runsheng Yu, Xinrun Wang, James Kwok
Date: 2021-09-29 00:00:00

Most advanced Actor-Critic (AC) approaches update the actor and critic concurrently through (stochastic) Gradient Descents (GD),  which may be trapped into bad local optimality due to the instability of these simultaneous updating schemes.  
Stackelberg AC learning scheme alleviates these limitations by adding a compensated indirect gradient terms to the GD. However, the indirect gradient terms are time-consuming to calculate, and the convergence rate is also relatively slow.  To alleviates these challenges, we find that in the Deterministic Policy Gradient family, by removing the terms that contain Hessian matrices and adopting the block diagonal approximation technique to approximate the remaining inverse matrices, we can construct an approximated Stackelberg AC learning scheme that is easy to compute and fast to converge. Experiments reveal that ours outperform SOTAs in terms of average returns under acceptable training time.
A molecular hypergraph convolutional network with functional group information
Authors: Junyoung Park, Fangying Chen, Jinkyoo Park
Date: 2021-09-29 00:00:00
Tasks: Property Prediction

We propose a Molecular Hypergraph Convolutional Network (MolHGCN) that predicts the molecular properties of a molecule using the atom and functional group information as inputs. Molecules can contain many types of functional groups, which will affect the properties the molecules. For example, the toxicity of a molecule is associated with toxicophores, such as nitroaromatic groups and thiourea. Conventional graph-based methods that consider the pair-wise interactions between nodes are inefficient in expressing the complex relationship between multiple nodes in a graph flexibly, and applying multi-hops may result in oversmoothing and overfitting problems. Hence, we propose MolHGCN to capture the substructural difference between molecules using the atom and functional group information. MolHGCN constructs a hypergraph representation of a molecule using functional group information from the input SMILES strings, extracts hidden representation using a two-stage message passing process (atom and functional group message passing), and predicts the properties of the molecules using the extracted hidden representation. We evaluate the performance of our model using Tox21, ClinTox, SIDER, BBBP, BACE, ESOL, FreeSolv and Lipophilicity datasets. We show that our model is able to outperform other baseline methods for most of the datasets. We particularly show that incorporating functional group information along with atom information results in better separability in the latent space, thus increasing the prediction accuracy of the molecule property prediction.
Exploiting Minimum-Variance Policy Evaluation for Policy Optimization
Authors: Alberto Maria Metelli, Samuele Meta, Marcello Restelli
Date: 2021-09-29 00:00:00

Off-policy methods are the basis of a large number of effective Policy Optimization (PO) algorithms. In this setting, Importance Sampling (IS) is typically employed as a what-if analysis tool, with the goal of estimating the performance of a target policy, given samples collected with a different behavioral policy. However, in Monte Carlo simulation, IS represents a variance minimization approach. In this field, a suitable behavioral distribution is employed for sampling, allowing diminishing the variance of the estimator below the one achievable when sampling from the target distribution. In this paper, we analyze IS in these two guises, showing the connections between the two objectives. We illustrate that variance minimization can be used as a performance improvement tool, with the advantage, compared with direct off-policy learning, of implicitly enforcing a trust region. We make use of these theoretical findings to build a PO algorithm, Policy Optimization via Optimal Policy Evaluation (PO2PE), that employs variance minimization as an inner loop. Finally, we present empirical evaluations on continuous RL benchmarks, with a particular focus on the robustness to small batch sizes.
A Fair Generative Model Using Total Variation Distance
Authors: Soobin Um, Changho Suh
Date: 2021-09-29 00:00:00
Tasks: Fairness

We explore a fairness-related challenge that arises in generative models. The challenge is that biased training data with imbalanced representations of demographic groups may yield a high asymmetry in size of generated samples across distinct groups. We focus on practically-relevant scenarios wherein demographic labels are not available and therefore the design of a fair generative model is particularly challenging. In this paper, we propose an optimization framework that regulates such unfairness by employing one prominent statistical notion, total variation distance (TVD). We quantify the degree of unfairness via the TVD between the generated samples and balanced-yet-small reference samples. We take a variational optimization approach to faithfully implement the TVD-based measure. Experiments on benchmark real datasets demonstrate that the proposed framework can significantly improve the fairness performance while maintaining realistic sample quality for a wide range of the reference set size all the way down to 1% relative to training set.
Learning Global Spatial Information for Multi-View Object-Centric Models
Authors: Yuya Kobayashi, Masahiro Suzuki, Yutaka Matsuo
Date: 2021-09-29 00:00:00
Tasks: Novel View Synthesis, Object

Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of a scene and infer object-centric representations from several observation views. In general, multi-object scenes can be uniquely determined if both the properties of individual objects and the spatial arrangement of objects are specified; however, existing multi-view object-centric models only infer object-level representations and lack spatial information. This insufficient modeling can degrade novel-view synthesis quality and make it difficult to generate novel scenes. We can model both spatial information and object representations by introducing hierarchical probabilistic model, which contains a global latent variable on top of object-level latent variables. However, how to execute inference and training with that hierarchical multi-view object-centric model is unclear. Therefore, we introduce several crucial components which help inference and training with the proposed model. We show that the proposed method achieves good inference quality and can also generate novel scenes.
Automatic Tuning of Federated Learning Hyper-Parameters from System Perspective
Authors: Huanle Zhang, Mi Zhang, Xin Liu, Prasant Mohapatra, Michael DeLucia
Date: 2021-09-29 00:00:00
Tasks: Federated Learning


Federated Learning (FL) is a distributed model training paradigm that preserves clients' data privacy. 
FL hyper-parameters significantly affect the training overheads in terms of time, computation, and communication.
However, the current practice of manually selecting FL hyper-parameters puts a high burden on FL practitioners since various applications prefer different training preferences. In this paper, we propose FedTuning, an automatic FL hyper-parameter tuning algorithm tailored to applications' diverse system requirements of FL training. FedTuning is lightweight and flexible, achieving an average of 41% improvement for different training preferences on time, computation, and communication compared to fixed FL hyper-parameters.
Stochastic Variance Reduced Ensemble Adversarial Attack
Authors: Jiadong Lin, Yifeng Xiong, Min Zhang, John E. Hopcroft, Kun He
Date: 2021-09-29 00:00:00
Tasks: Adversarial Attack

Black-box adversarial attack has attracted much attention for its practical use in deep learning applications, and it is very challenging as there is no access to the architecture and weights of the target model. Based on the hypothesis that if an example remains adversarial for multiple models, then it is more likely to transfer to other models, the ensemble-based attack methods are efficient and widely used in the black-box setting.  Nevertheless, existing ensemble-based approaches simply aggregate the outputs of all models but ignore the variance of different models, leading to a rather poor local optimum. To address this issue, we propose a stochastic variance reduced ensemble attack method to boost the performance of black-box adversarial attacks. By integrating the stochastic variance reduced gradient technique into the model ensemble attack, our method can balance the gradient of different models and leads to a better local maximum, resulting in highly transferable adversarial examples. Empirical results on the standard ImageNet dataset demonstrate that our method can boost the ensemble attack performance and significantly improve the transferability of the generated adversarial examples.
Efficient Computation of Deep Nonlinear Infinite-Width Neural Networks that Learn Features
Authors: Greg Yang, Michael Santacroce, Edward J Hu
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4

While a popular limit of infinite-width neural networks, the Neural Tangent Kernel (NTK) often exhibits performance gaps from finite-width neural networks on standard datasets, due to lack of feature learning. Although the feature learning *maximal update limit*, or *μ-limit* (Yang and Hu, 2020) of wide networks has closed the gap for 1-hidden-layer linear models, no one has been able to demonstrate this for deep nonlinear multi-layer perceptrons (MLP) because of μ-limit’s computational difficulty in this setting.  Here, we solve this problem by proposing a novel feature learning limit, the *π-limit*, that bypasses the computational issues. The π-limit, in short, is the limit of a form of projected gradient descent, and the π-limit of an MLP is roughly another MLP where gradients are appended to weights during training. We prove its almost sure convergence with width using the Tensor Programs technique. We evaluate it on CIFAR10 and Omniglot against NTK as well as finite networks, finding the π-limit outperform finite-width models trained normally (without projection) in both settings, closing the performance gap between finite- and infinite-width neural networks previously left by NTK.
Knowledge Graph Completion as Tensor Decomposition: A Genreal Form and Tensor N-rank Regularization
Authors: Changyi Xiao, Xiangnan He, Yixin Cao
Date: 2021-09-29 00:00:00
Tasks: Form, Knowledge Graph Completion, Tensor Decomposition

Knowledge graph completion (KGC) is a 3rd-order binary tensor completion task. Tensor decomposition based (TDB) models have shown great performance in KGC. In this paper, we summarize existing TDB models and derive a general form for them. Based on the general form, we show the principles of model design to satisfy logical rules. However, these models suffer from the overfitting problem severely. Therefore, we propose a regularization term based on the tensor $n$-rank which enforces the low-rankness of the tensor. First, we relax the tensor $n$-rank to the sum of the nuclear norms of the unfolding matrix along each mode of the tensor. In order to be computationally efficient, we further give an upper bound of the sum of the nuclear norms. Finally, we use the upper bound as the regularization term to achieve low-rank matrix decomposition of each unfolding matrix. Experiments show that our model achieves state-of-the-art performance on benchmark datasets.
Representation Disentanglement in Generative Models with Contrastive Learning
Authors: Shentong Mo, Zhun Sun, Shumin Han
Date: 2021-09-29 00:00:00
Tasks: Contrastive Learning, Disentanglement, image-classification, Image Classification

Contrastive learning has shown its effectiveness in image classification and generation. Recent works apply the contrastive learning on the discriminator of the Generative Adversarial Networks, and there exists little work on exploring if contrastive learning can be applied on encoders to learn disentangled representations. In this work, we propose a simple yet effective method via incorporating contrastive learning into latent optimization, where we name it $\textbf{\texttt{ContraLORD}}$. Specifically, we first use a generator to learn discriminative and disentangled embeddings via latent optimization. Then an encoder and two momentum encoders are applied to dynamically learn disentangled information across large amount of samples with content-level and residual-level contrastive loss. In the meanwhile, we tune the encoder with the learned embeddings in an amortized manner. We evaluate our approach on ten benchmarks in terms of representation disentanglement and linear classification. Extensive experiments demonstrate the effectiveness of our ContraLORD on learning both discriminative and generative representations.
Open-sampling: Re-balancing Long-tailed Datasets with Out-of-Distribution Data
Authors: Hongxin Wei, Lue Tao, Renchunzi Xie, Lei Feng, Bo An
Date: 2021-09-29 00:00:00

Deep neural networks usually perform poorly when the training dataset suffers from extreme class imbalance. To handle this issue, popular re-sampling methods generally require in-distribution data to balance the class priors. However, obtaining suitable in-distribution data with precise labels for selected classes is challenging. In this paper, we theoretically show that out-of-distribution data (i.e., open-set samples) could be leveraged to augment the minority classes from a Bayesian perspective. Based on this motivation, we propose a novel method called Open-sampling, which utilizes open-set noisy labels to re-balance the class priors of the training dataset. For each open-set instance, the label is sampled from our pre-defined distribution that is complementary to the original class priors. Furthermore, class-dependent weights are generated to provide stronger regularization on the minority classes than on the majority classes. 
We empirically show that Open-sampling not only re-balances the class prior but also encourages the neural network to learn separable representations. Extensive experiments on benchmark datasets demonstrate that our proposed method significantly outperforms existing data re-balancing methods and can be easily incorporated into existing state-of-the-art methods to enhance their performance.
Discrepancy-Optimal Meta-Learning for Domain Generalization
Authors: Chen Jia
Date: 2021-09-29 00:00:00
Tasks: Bilevel Optimization, Domain Generalization, Meta-Learning, Style Generalization

This work attempts to tackle the problem of domain generalization (DG) via learning to reduce domain shift with an episodic training procedure. In particular, we measure the domain shift with $\mathcal{Y}$-discrepancy and learn to optimize $\mathcal{Y}$-discrepancy between the unseen target domain and source domains only using source-domain samples. Theoretically, we give a PAC-style generalization bound for discrepancy-optimal meta-learning and further make comparisons with other DG bounds including ERM and domain-invariant learning. The theoretical analyses show that there is a tradeoff between classification performance and computational complexity for discrepancy-optimal meta-learning. The theoretical results also shed light on a bilevel optimization algorithm for DG. Empirically, we evaluate the algorithm with DomainBed and achieves state-of-the-art results on two DG benchmarks.
Generalizable Learning to Optimize into Wide Valleys
Authors: Junjie Yang, Tianlong Chen, Mingkang Zhu, Fengxiang He, DaCheng Tao, Yingbin Liang, Zhangyang Wang
Date: 2021-09-29 00:00:00

Learning to optimize (L2O) has gained increasing popularity in various optimization tasks, since classical optimizers usually require laborious, problem-specific design and hyperparameter tuning. However, current L2O approaches are designed for fast minimization of the objective function value (i.e., training error), hence often suffering from poor generalization ability such as in training deep neural networks (DNNs), including ($i$) disappointing performance across unseen optimizees $\textit{(optimizer generalization)}$; ($ii$) unsatisfactory test-set accuracy of trained DNNs ($\textit{optmizee generalization}$). To overcome the limitations, this paper introduces $\textit{flatness-aware}$ regularizers into L2O for shaping the local geometry of optimizee's loss landscape. Specifically, it guides optimizee to locate well-generalizable minimas in large flat regions of loss surface, while tending to avoid sharp valleys. Such optimizee generalization abilities of $\textit{flatness-aware}$ regularizers have been proved theoretically. Extensive experiments consistently validate the effectiveness of our proposals with substantially improved generalization on multiple sophisticated L2O models and diverse optimizees. Our theoretical and empirical results solidify the foundation for L2O's practically usage. All codes and pre-trained models will be shared upon acceptance.