pwc-archive/papers-with-abstracts huggingface dataset:
In Search of Robust Measures of Generalization
Authors: Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, Ioannis Mitliagkas, Daniel M. Roy
Date: 2020-10-22 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Generalization Bounds

One of the principal scientific challenges in deep learning is explaining generalization, i.e., why the particular way the community now trains networks to achieve small training error also leads to small error on held-out data from the same population. It is widely appreciated that some worst-case theories -- such as those based on the VC dimension of the class of predictors induced by modern neural network architectures -- are unable to explain empirical performance. A large volume of work aims to close this gap, primarily by developing bounds on generalization error, optimization error, and excess risk. When evaluated empirically, however, most of these bounds are numerically vacuous. Focusing on generalization bounds, this work addresses the question of how to evaluate such bounds empirically. Jiang et al. (2020) recently described a large-scale empirical study aimed at uncovering potential causal relationships between bounds/measures and generalization. Building on their study, we highlight where their proposed methods can obscure failures and successes of generalization measures in explaining generalization. We argue that generalization measures should instead be evaluated within the framework of distributional robustness.
CoinDICE: Off-Policy Confidence Interval Estimation
Authors: Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, Dale Schuurmans
Date: 2020-10-22 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Off-policy evaluation, Reinforcement Learning (RL), valid

We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning, where the goal is to estimate a confidence interval on a target policy's value, given only access to a static experience dataset collected by unknown behavior policies. Starting from a function space embedding of the linear program formulation of the $Q$-function, we obtain an optimization problem with generalized estimating equation constraints. By applying the generalized empirical likelihood method to the resulting Lagrangian, we propose CoinDICE, a novel and efficient algorithm for computing confidence intervals. Theoretically, we prove the obtained confidence intervals are valid, in both asymptotic and finite-sample regimes. Empirically, we show in a variety of benchmarks that the confidence interval estimates are tighter and more accurate than existing methods.
F-Siamese Tracker: A Frustum-based Double Siamese Network for 3D Single Object Tracking
Authors: Hao Zou, Jinhao Cui, Xin Kong, Chujuan Zhang, Yong liu, Feng Wen, Wanlong Li
Date: 2020-10-22 00:00:00
Tasks: 3D Single Object Tracking, Object, Object Tracking

This paper presents F-Siamese Tracker, a novel approach for single object tracking prominently characterized by more robustly integrating 2D and 3D information to reduce redundant search space. A main challenge in 3D single object tracking is how to reduce search space for generating appropriate 3D candidates. Instead of solely relying on 3D proposals, firstly, our method leverages the Siamese network applied on RGB images to produce 2D region proposals which are then extruded into 3D viewing frustums. Besides, we perform an online accuracy validation on the 3D frustum to generate refined point cloud searching space, which can be embedded directly into the existing 3D tracking backbone. For efficiency, our approach gains better performance with fewer candidates by reducing search space. In addition, benefited from introducing the online accuracy validation, for occasional cases with strong occlusions or very sparse points, our approach can still achieve high precision, even when the 2D Siamese tracker loses the target. This approach allows us to set a new state-of-the-art in 3D single object tracking by a significant margin on a sparse outdoor dataset (KITTI tracking). Moreover, experiments on 2D single object tracking show that our framework boosts 2D tracking performance as well.
Performance Prediction for Convolutional Neural Networks in Edge Devices
Authors: Halima Bouzidi, Hamza Ouarnoughi, Smail Niar, Abdessamad Ait El Cadi
Date: 2020-10-21 00:00:00
Tasks: GPU, Prediction

Running Convolutional Neural Network (CNN) based applications on edge devices near the source of data can meet the latency and privacy challenges. However due to their reduced computing resources and their energy constraints, these edge devices can hardly satisfy CNN needs in processing and data storage. For these platforms, choosing the CNN with the best trade-off between accuracy and execution time while respecting Hardware constraints is crucial. In this paper, we present and compare five (5) of the widely used Machine Learning based methods for execution time prediction of CNNs on two (2) edge GPU platforms. For these 5 methods, we also explore the time needed for their training and tuning their corresponding hyperparameters. Finally, we compare times to run the prediction models on different platforms. The utilization of these methods will highly facilitate design space exploration by providing quickly the best CNN on a target edge GPU. Experimental results show that eXtreme Gradient Boosting (XGBoost) provides a less than 14.73% average prediction error even for unexplored and unseen CNN models' architectures. Random Forest (RF) depicts comparable accuracy but needs more effort and time to be trained. The other 3 approaches (OLS, MLP and SVR) are less accurate for CNN performances estimation.
Forming native shortcut networks to simulate protein folding
Authors: Susan Khor
Date: 2019-02-17 00:00:00
Tasks: Protein Folding

Native shortcut networks (SCN0) are sub-graphs of native protein residue networks (PRN0). In this paper, we propose the Network Dynamics (ND) model, which reconstructs a PRN0 by adding back its edges according to some recipe, while its nodes are fixed at native (PDB) locations. A PRN0 reconstruction will eventually reconstruct its SCN0, but only after exploring several non-native shortcut network (SCN) configurations. It is these other SCN configurations that are of interest to us, as they produce the statistics to evaluate the different recipes. The recipes vary from each other slightly to investigate the effect of different edge orderings on protein folding. An edge ordering is deemed more successful if it produces a stronger correlation with experimental folding rates. Over proteins of different chain fold types, this basic requirement is best satisfied when a recipe favours earlier restoration of edges with smaller sequence separation, and earlier restoration of edges incident on nodes with larger remaining degree within the 1-hop neighborhood of a previously selected node. Further, this recipe generated more route-like trajectories over SCN space and a wider range of simulated folding rates, both of which signal cooperative two-state folding behavior. It also produced better correspondence between ND calculated phi-values and experimental phi-values obtained from a set of ten transition state ensembles (TSE). Also introduced is the local centrality measure which uses centrality of initial fold substructures to yield calculated phi-values from PRN0s with even better overall correspondence than ND calculated phi-values. Preliminary investigation with mixed ND recipes by chain fold type suggests that concocting further context-sensitive recipes by chain fold type and combining them could improve ND. This search task may be better undertaken by machine learning.
AI-lead Court Debate Case Investigation
Authors: Changzhen Ji, Xin Zhou, Conghui Zhu, Tiejun Zhao
Date: 2020-10-22 00:00:00
Tasks: Question Generation, Question-Generation, Text Generation

The multi-role judicial debate composed of the plaintiff, defendant, and judge is an important part of the judicial trial. Different from other types of dialogue, questions are raised by the judge, The plaintiff, plaintiff's agent defendant, and defendant's agent would be to debating so that the trial can proceed in an orderly manner. Question generation is an important task in Natural Language Generation. In the judicial trial, it can help the judge raise efficient questions so that the judge has a clearer understanding of the case. In this work, we propose an innovative end-to-end question generation model-Trial Brain Model (TBM) to build a Trial Brain, it can generate the questions the judge wants to ask through the historical dialogue between the plaintiff and the defendant. Unlike prior efforts in natural language generation, our model can learn the judge's questioning intention through predefined knowledge. We do experiments on real-world datasets, the experimental results show that our model can provide a more accurate question in the multi-role court debate scene.
Identifying Learning Rules From Neural Network Observables
Authors: Aran Nayebi, Sanjana Srivastava, Surya Ganguli, Daniel L. K. Yamins
Date: 2020-10-22 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Open-Ended Question Answering

The brain modifies its synaptic strengths during learning in order to better adapt to its environment. However, the underlying plasticity rules that govern learning are unknown. Many proposals have been suggested, including Hebbian mechanisms, explicit error backpropagation, and a variety of alternatives. It is an open question as to what specific experimental measurements would need to be made to determine whether any given learning rule is operative in a real biological system. In this work, we take a "virtual experimental" approach to this problem. Simulating idealized neuroscience experiments with artificial neural networks, we generate a large-scale dataset of learning trajectories of aggregate statistics measured in a variety of neural network architectures, loss functions, learning rule hyperparameters, and parameter initializations. We then take a discriminative approach, training linear and simple non-linear classifiers to identify learning rules from features based on these observables. We show that different classes of learning rules can be separated solely on the basis of aggregate statistics of the weights, activations, or instantaneous layer-wise activity changes, and that these results generalize to limited access to the trajectory and held-out architectures and learning curricula. We identify the statistics of each observable that are most relevant for rule identification, finding that statistics from network activities across training are more robust to unit undersampling and measurement noise than those obtained from the synaptic strengths. Our results suggest that activation patterns, available from electrophysiological recordings of post-synaptic activities on the order of several hundred units, frequently measured at wider intervals over the course of learning, may provide a good basis on which to identify learning rules.
Learning Dual Semantic Relations with Graph Attention for Image-Text Matching
Authors: Keyu Wen, Xiaodong Gu, Qingrong Cheng
Date: 2020-10-22 00:00:00
Tasks: Cross-Modal Retrieval, Graph Attention, Image-text matching, Text Matching

Image-Text Matching is one major task in cross-modal information processing. The main challenge is to learn the unified visual and textual representations. Previous methods that perform well on this task primarily focus on not only the alignment between region features in images and the corresponding words in sentences, but also the alignment between relations of regions and relational words. However, the lack of joint learning of regional features and global features will cause the regional features to lose contact with the global context, leading to the mismatch with those non-object words which have global meanings in some sentences. In this work, in order to alleviate this issue, it is necessary to enhance the relations between regions and the relations between regional and global concepts to obtain a more accurate visual representation so as to be better correlated to the corresponding text. Thus, a novel multi-level semantic relations enhancement approach named Dual Semantic Relations Attention Network(DSRAN) is proposed which mainly consists of two modules, separate semantic relations module and the joint semantic relations module. DSRAN performs graph attention in both modules respectively for region-level relations enhancement and regional-global relations enhancement at the same time. With these two modules, different hierarchies of semantic relations are learned simultaneously, thus promoting the image-text matching process by providing more information for the final visual representation. Quantitative experimental results have been performed on MS-COCO and Flickr30K and our method outperforms previous approaches by a large margin due to the effectiveness of the dual semantic relations learning scheme. Codes are available at https://github.com/kywen1119/DSRAN.
SlimIPL: Language-Model-Free Iterative Pseudo-Labeling
Authors: Tatiana Likhomanenko, Qiantong Xu, Jacob Kahn, Gabriel Synnaeve, Ronan Collobert
Date: 2020-10-22 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Language Modeling, Language Modelling, model, Pseudo Label, speech-recognition, Speech Recognition

Recent results in end-to-end automatic speech recognition have demonstrated the efficacy of pseudo-labeling for semi-supervised models trained both with Connectionist Temporal Classification (CTC) and Sequence-to-Sequence (seq2seq) losses. Iterative Pseudo-Labeling (IPL), which continuously trains a single model using pseudo-labels iteratively re-generated as the model learns, has been shown to further improve performance in ASR. We improve upon the IPL algorithm: as the model learns, we propose to iteratively re-generate transcriptions with hard labels (the most probable tokens), that is, without a language model. We call this approach Language-Model-Free IPL (slimIPL) and give a resultant training setup for low-resource settings with CTC-based models. slimIPL features a dynamic cache for pseudo-labels which reduces sensitivity to changes in relabeling hyperparameters and results in improves training stability. slimIPL is also highly-efficient and requires 3.5-4x fewer computational resources to converge than other state-of-the-art semi/self-supervised approaches. With only 10 hours of labeled audio, slimIPL is competitive with self-supervised approaches, and is state-of-the-art with 100 hours of labeled audio without the use of a language model both at test time and during pseudo-label generation.
Unsupervised Representation Learning for Speaker Recognition via Contrastive Equilibrium Learning
Authors: Sung Hwan Mun, Woo Hyun Kang, Min Hyun Han, Nam Soo Kim
Date: 2020-10-22 00:00:00
Tasks: Representation Learning, Speaker Recognition, Speaker Verification

In this paper, we propose a simple but powerful unsupervised learning method for speaker recognition, namely Contrastive Equilibrium Learning (CEL), which increases the uncertainty on nuisance factors latent in the embeddings by employing the uniformity loss. Also, to preserve speaker discriminability, a contrastive similarity loss function is used together. Experimental results showed that the proposed CEL significantly outperforms the state-of-the-art unsupervised speaker verification systems and the best performing model achieved 8.01% and 4.01% EER on VoxCeleb1 and VOiCES evaluation sets, respectively. On top of that, the performance of the supervised speaker embedding networks trained with initial parameters pre-trained via CEL showed better performance than those trained with randomly initialized parameters.
Synthesis of High-Purity Solid SiO2 Nanodumbbells via Induced Aggregation
Authors: Yu-Hao Deng
Date: 2020-10-21 00:00:00
Tasks: Vocal Bursts Intensity Prediction

Optically levitated nanodumbbells in vacuum are excellent candidates for thermodynamics, macroscopic quantum mechanics, precision measurements, and quantum sensing. Silica (SiO2) material, with extremely low absorption of near-infrared light and rubust mechanical strength, has been the most potential material for this system. Here we synthesized high-purity solid SiO2 nanodumbbells by introducing acetone for the induced aggregation of SiO2 nanospheres. The nanodumbbells show high uniformity and the sizes are tunable. Previous experimental results demonstrated that the synthetic nanodumbbells can be applied in GHz nanomechanical rotors and can withstand the tensile strength of over 13 GPa. This work engineers a new material platform to advance levitated optomechanics.
Contrastive Self-Supervised Learning for Wireless Power Control
Authors: Navid Naderializadeh
Date: 2020-10-22 00:00:00
Tasks: Contrastive Learning, Self-Supervised Learning

We propose a new approach for power control in wireless networks using self-supervised learning. We partition a multi-layer perceptron that takes as input the channel matrix and outputs the power control decisions into a backbone and a head, and we show how we can use contrastive learning to pre-train the backbone so that it produces similar embeddings at its output for similar channel matrices and vice versa, where similarity is defined in an information-theoretic sense by identifying the interference links that can be optimally treated as noise. The backbone and the head are then fine-tuned using a limited number of labeled samples. Simulation results show the effectiveness of the proposed approach, demonstrating significant gains over pure supervised learning methods in both sum-throughput and sample efficiency.
Distributed Pinning Control Design for Probabilistic Boolean Networks
Authors: Lin Lin, Jinde Cao, Jianquan Lu, Jie Zhong
Date: 2019-12-07 00:00:00

This paper investigates the stabilization of probabilistic Boolean networks (PBNs) via a novel pinning control strategy based on network structure. In a PBN, each node needs to choose a Boolean function from candidate Boolean function set at each time instance with certain probability. Owing to the stochasticity, the uniform state feedback controllers, which is independent of switching signal, might be out of work. Thereby, a criterion is derived to determine that under what condition uniform controllers can be applied, otherwise non-uniform controllers need to be utilized. Accordingly, an algorithm is designed to find a series of state feedback pinning controllers, under which such a PBN is stabilized to a prespecified steady state. It is worth pointing out that the pinning control used in this paper only requires local in-neighbors' information, rather than global information. Hence, it is also termed as distributed pinning control and reduces the computational complexity to a large extent. Profiting from this, it provides a potential to deal with some large-scale networks. Finally, the mammalian cell-cycle encountering a mutated phenotype is described as a PBN, and presented to demonstrate the obtained results.
Differentially-Private Federated Linear Bandits
Authors: Abhimanyu Dubey, Alex Pentland
Date: 2020-10-22 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Federated Learning

The rapid proliferation of decentralized learning systems mandates the need for differentially-private cooperative learning. In this paper, we study this in context of the contextual linear bandit: we consider a collection of agents cooperating to solve a common contextual bandit, while ensuring that their communication remains private. For this problem, we devise \textsc{FedUCB}, a multiagent private algorithm for both centralized and decentralized (peer-to-peer) federated learning. We provide a rigorous technical analysis of its utility in terms of regret, improving several results in cooperative bandit learning, and provide rigorous privacy guarantees as well. Our algorithms provide competitive performance both in terms of pseudoregret bounds and empirical benchmark performance in various multi-agent settings.
Joint Use of Node Attributes and Proximity for Semi-Supervised Classification on Graphs
Authors: Arpit Merchant, Michael Mathioudakis
Date: 2020-10-22 00:00:00
Tasks: General Classification, Node Classification

The task of node classification is to infer unknown node labels, given the labels for some of the nodes along with the network structure and other node attributes. Typically, approaches for this task assume homophily, whereby neighboring nodes have similar attributes and a node's label can be predicted from the labels of its neighbors or other proximate (i.e., nearby) nodes in the network. However, such an assumption may not always hold -- in fact, there are cases where labels are better predicted from the individual attributes of each node rather than the labels of its proximate nodes. Ideally, node classification methods should flexibly adapt to a range of settings wherein unknown labels are predicted either from labels of proximate nodes, or individual node attributes, or partly both. In this paper, we propose a principled approach, JANE, based on a generative probabilistic model that jointly weighs the role of attributes and node proximity via embeddings in predicting labels. Our experiments on a variety of network datasets demonstrate that JANE exhibits the desired combination of versatility and competitive performance compared to standard baselines.
Multi-agent active perception with prediction rewards
Authors: Mikko Lauri, Frans A. Oliehoek
Date: 2020-10-22 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Prediction

Multi-agent active perception is a task where a team of agents cooperatively gathers observations to compute a joint estimate of a hidden variable. The task is decentralized and the joint estimate can only be computed after the task ends by fusing observations of all agents. The objective is to maximize the accuracy of the estimate. The accuracy is quantified by a centralized prediction reward determined by a centralized decision-maker who perceives the observations gathered by all agents after the task ends. In this paper, we model multi-agent active perception as a decentralized partially observable Markov decision process (Dec-POMDP) with a convex centralized prediction reward. We prove that by introducing individual prediction actions for each agent, the problem is converted into a standard Dec-POMDP with a decentralized prediction reward. The loss due to decentralization is bounded, and we give a sufficient condition for when it is zero. Our results allow application of any Dec-POMDP solution algorithm to multi-agent active perception problems, and enable planning to reduce uncertainty without explicit computation of joint estimates. We demonstrate the empirical usefulness of our results by applying a standard Dec-POMDP algorithm to multi-agent active perception problems, showing increased scalability in the planning horizon.
MinMax Methods for Optimal Transport and Beyond: Regularization, Approximation and Numerics
Authors: Luca De Gennaro Aquino, Stephan Eckstein
Date: 2020-10-22 00:00:00
Proceeding: NeurIPS 2020 12

We study MinMax solution methods for a general class of optimization problems related to (and including) optimal transport. Theoretically, the focus is on fitting a large class of problems into a single MinMax framework and generalizing regularization techniques known from classical optimal transport. We show that regularization techniques justify the utilization of neural networks to solve such problems by proving approximation theorems and illustrating fundamental issues if no regularization is used. We further study the relation to the literature on generative adversarial nets, and analyze which algorithmic techniques used therein are particularly suitable to the class of problems studied in this paper. Several numerical experiments showcase the generality of the setting and highlight which theoretical insights are most beneficial in practice.