pwc-archive/papers-with-abstracts huggingface dataset:
On the Global Optimality of Model-Agnostic Meta-Learning
Authors: Lingxiao Wang, Qi Cai, Zhuoran Yang, Zhaoran Wang
Date: 2020-06-23 00:00:00
Proceeding: ICML 2020 1
Tasks: Bilevel Optimization, Meta-Learning

Model-agnostic meta-learning (MAML) formulates meta-learning as a bilevel optimization problem, where the inner level solves each subtask based on a shared prior, while the outer level searches for the optimal shared prior by optimizing its aggregated performance over all the subtasks. Despite its empirical success, MAML remains less understood in theory, especially in terms of its global optimality, due to the nonconvexity of the meta-objective (the outer-level objective). To bridge such a gap between theory and practice, we characterize the optimality gap of the stationary points attained by MAML for both reinforcement learning and supervised learning, where the inner-level and outer-level problems are solved via first-order optimization methods. In particular, our characterization connects the optimality gap of such stationary points with (i) the functional geometry of inner-level objectives and (ii) the representation power of function approximators, including linear models and neural networks. To the best of our knowledge, our analysis establishes the global optimality of MAML with nonconvex meta-objectives for the first time.
Simple and Effective VAE Training with Calibrated Decoders
Authors: Oleh Rybkin, Kostas Daniilidis, Sergey Levine
Date: 2020-06-23 00:00:00
Tasks: Decoder

Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. However, training VAEs often requires considerable hyperparameter tuning to determine the optimal amount of information retained by the latent variable. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution and can determine this amount of information automatically, on the VAE performance. While many methods for learning calibrated decoders have been proposed, many of the recent papers that employ VAEs rely on heuristic hyperparameters and ad-hoc modifications instead. We perform the first comprehensive comparative analysis of calibrated decoder and provide recommendations for simple and effective VAE training. Our analysis covers a range of image and video datasets and several single-image and sequential VAE models. We further propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically. We observe empirically that using heuristic modifications is not necessary with our method. Project website is at https://orybkin.github.io/sigma-vae/
On the Fundamental Limits of Coded Data Shuffling for Distributed Machine Learning
Authors: Adel Elmahdy, Soheil Mohajer
Date: 2018-07-11 00:00:00
Tasks: BIG-bench Machine Learning

We consider the data shuffling problem in a distributed learning system, in which a master node is connected to a set of worker nodes, via a shared link, in order to communicate a set of files to the worker nodes. The master node has access to a database of files. In every shuffling iteration, each worker node processes a new subset of files, and has excess storage to partially cache the remaining files, assuming the cached files are uncoded. The caches of the worker nodes are updated every iteration, and they should be designed to satisfy any possible unknown permutation of the files in subsequent iterations. For this problem, we characterize the exact load-memory trade-off for worst-case shuffling by deriving the minimum communication load for a given storage capacity per worker node. As a byproduct, the exact load-memory trade-off for any shuffling is characterized when the number of files is equal to the number of worker nodes. We propose a novel deterministic coded shuffling scheme, which improves the state of the art, by exploiting the cache memories to create coded functions that can be decoded by several worker nodes. Then, we prove the optimality of our proposed scheme by deriving a matching lower bound and showing that the placement phase of the proposed coded shuffling scheme is optimal over all shuffles.
Airfoil Design Parameterization and Optimization using Bézier Generative Adversarial Networks
Authors: Wei Chen, Kevin Chiu, Mark Fuge
Date: 2020-06-21 00:00:00
Tasks: global-optimization

Global optimization of aerodynamic shapes usually requires a large number of expensive computational fluid dynamics simulations because of the high dimensionality of the design space. One approach to combat this problem is to reduce the design space dimension by obtaining a new representation. This requires a parametric function that compactly and sufficiently describes useful variation in shapes. We propose a deep generative model, B\'ezier-GAN, to parameterize aerodynamic designs by learning from shape variations in an existing database. The resulted new parameterization can accelerate design optimization convergence by improving the representation compactness while maintaining sufficient representation capacity. We use the airfoil design as an example to demonstrate the idea and analyze B\'ezier-GAN's representation capacity and compactness. Results show that B\'ezier-GAN both (1) learns smooth and realistic shape representations for a wide range of airfoils and (2) empirically accelerates optimization convergence by at least two times compared to state-of-the-art parameterization methods.
The GCE in a New Light: Disentangling the $γ$-ray Sky with Bayesian Graph Convolutional Neural Networks
Authors: Florian List, Nicholas L. Rodd, Geraint F. Lewis, Ishaan Bhat
Date: 2020-06-22 00:00:00

A fundamental question regarding the Galactic Center Excess (GCE) is whether the underlying structure is point-like or smooth. This debate, often framed in terms of a millisecond pulsar or annihilating dark matter (DM) origin for the emission, awaits a conclusive resolution. In this work we weigh in on the problem using Bayesian graph convolutional neural networks. In simulated data, our neural network (NN) is able to reconstruct the flux of inner Galaxy emission components to on average $\sim$0.5%, comparable to the non-Poissonian template fit (NPTF). When applied to the actual $\textit{Fermi}$-LAT data, we find that the NN estimates for the flux fractions from the background templates are consistent with the NPTF; however, the GCE is almost entirely attributed to smooth emission. While suggestive, we do not claim a definitive resolution for the GCE, as the NN tends to underestimate the flux of point-sources peaked near the 1$\sigma$ detection threshold. Yet the technique displays robustness to a number of systematics, including reconstructing injected DM, diffuse mismodeling, and unmodeled north-south asymmetries. So while the NN is hinting at a smooth origin for the GCE at present, with further refinements we argue that Bayesian Deep Learning is well placed to resolve this DM mystery.
Deep Belief Network based representation learning for lncRNA-disease association prediction
Authors: Manu Madhavan, Gopakumar G
Date: 2020-06-22 00:00:00
Tasks: Graph Neural Network, Representation Learning

Background: The expanding research in the field of long non-coding RNAs(lncRNAs) showed abnormal expression of lncRNAs in many complex diseases. Accurately identifying lncRNA-disease association is essential in understanding lncRNA functionality and disease mechanism. There are many machine learning techniques involved in the prediction of lncRNA-disease association which use different biological interaction networks and associated features. Feature learning from the network structured data is one of the limiting factors of machine learning-based methods. Graph neural network based techniques solve this limitation by unsupervised feature learning. Deep belief networks (DBN) are recently used in biological network analysis to learn the latent representations of network features. Method: In this paper, we propose a DBN based lncRNA-disease association prediction model (DBNLDA) from lncRNA, disease and miRNA interactions. The architecture contains three major modules-network construction, DBN based feature learning and neural network-based prediction. First, we constructed three heterogeneous networks such as lncRNA-miRNA similarity (LMS), disease-miRNA similarity (DMS) and lncRNA-disease association (LDA) network. From the node embedding matrices of similarity networks, lncRNA-disease representations were learned separately by two DBN based subnetworks. The joint representation of lncRNA-disease was learned by a third DBN from outputs of the two subnetworks mentioned. This joint feature representation was used to predict the association score by an ANN classifier. Result: The proposed method obtained AUC of 0.96 and AUPR of 0.967 when tested against standard dataset used by the state-of-the-art methods. Analysis on breast, lung and stomach cancer cases also affirmed the effectiveness of DBNLDA in predicting significant lncRNA-disease associations.
Artificial Intelligence-Assisted Energy and Thermal Comfort Control for Sustainable Buildings: An Extended Representation of the Systematic Review
Authors: Ghezlane Halhoul Merabet, Mohamed Essaaidi, Mohamed Ben-Haddou, Basheer Qolomany, Junaid Qadir, Muhammad Anan, Ala Al-Fuqaha, Riduan Mohamed Abid, Driss Benhaddou
Date: 2020-06-22 00:00:00
Tasks: Articles, Systematic Literature Review

Different factors such as thermal comfort, humidity, air quality, and noise have significant combined effects on the acceptability and quality of the activities performed by the building occupants who spend most of their times indoors. Among the factors cited, thermal comfort, which contributes to the human well-being because of its connection with the thermoregulation of the human body. Therefore, the creation of thermally comfortable and energy efficient environments is of great importance in the design of the buildings and hence the heating, ventilation and air-conditioning systems. Recent works have been directed towards more advanced control strategies, based mainly on artificial intelligence which has the ability to imitate human behavior. This systematic literature review aims to provide an overview of the intelligent control strategies inside building and to investigate their ability to balance thermal comfort and energy efficiency optimization in indoor environments. Methods. A systematic literature review examined the peer-reviewed research works using ACM Digital Library, Scopus, Google Scholar, IEEE Xplore (IEOL), Web of Science, and Science Direct (SDOL), besides other sources from manual search. With the following string terms: thermal comfort, comfort temperature, preferred temperature, intelligent control, advanced control, artificial intelligence, computational intelligence, building, indoors, and built environment. Inclusion criteria were: English, studies monitoring, mainly, human thermal comfort in buildings and energy efficiency simultaneously based on control strategies using the intelligent approaches. Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines were used. Initially, 1,077 articles were yielded, and 120 ultimately met inclusion criteria and were reviewed.
A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence
Authors: Changhao Chen, Bing Wang, Chris Xiaoxuan Lu, Niki Trigoni, Andrew Markham
Date: 2020-06-22 00:00:00
Tasks: Deep Learning, Scene Understanding, Simultaneous Localization and Mapping

Deep learning based localization and mapping has recently attracted significant attention. Instead of creating hand-designed algorithms through exploitation of physical models or geometric theories, deep learning based solutions provide an alternative to solve the problem in a data-driven way. Benefiting from ever-increasing volumes of data and computational power, these methods are fast evolving into a new area that offers accurate and robust systems to track motion and estimate scenes and their structure for real-world applications. In this work, we provide a comprehensive survey, and propose a new taxonomy for localization and mapping using deep learning. We also discuss the limitations of current models, and indicate possible future directions. A wide range of topics are covered, from learning odometry estimation, mapping, to global localization and simultaneous localization and mapping (SLAM). We revisit the problem of perceiving self-motion and scene understanding with on-board sensors, and show how to solve it by integrating these modules into a prospective spatial machine intelligence system (SMIS). It is our hope that this work can connect emerging works from robotics, computer vision and machine learning communities, and serve as a guide for future researchers to apply deep learning to tackle localization and mapping problems.
A Causally Formulated Hazard Ratio Estimation through Backdoor Adjustment on Structural Causal Model
Authors: Riddhiman Adib, Paul Griffin, Sheikh Iqbal Ahamed, Mohammad Adibuzzaman
Date: 2020-06-22 00:00:00
Tasks: Causal Inference, Survival Analysis

Identifying causal relationships for a treatment intervention is a fundamental problem in health sciences. Randomized controlled trials (RCTs) are considered the gold standard for identifying causal relationships. However, recent advancements in the theory of causal inference based on the foundations of structural causal models (SCMs) have allowed the identification of causal relationships from observational data, under certain assumptions. Survival analysis provides standard measures, such as the hazard ratio, to quantify the effects of an intervention. While hazard ratios are widely used in clinical and epidemiological studies for RCTs, a principled approach does not exist to compute hazard ratios for observational studies with SCMs. In this work, we review existing approaches to compute hazard ratios as well as their causal interpretation, if it exists. We also propose a novel approach to compute hazard ratios from observational studies using backdoor adjustment through SCMs and do-calculus. Finally, we evaluate the approach using experimental data for Ewing's sarcoma.
Articulatory-WaveNet: Autoregressive Model For Acoustic-to-Articulatory Inversion
Authors: Narjes Bozorg, Michael T. Johnson
Date: 2020-06-22 00:00:00
Tasks: Speech Synthesis

This paper presents Articulatory-WaveNet, a new approach for acoustic-to-articulator inversion. The proposed system uses the WaveNet speech synthesis architecture, with dilated causal convolutional layers using previous values of the predicted articulatory trajectories conditioned on acoustic features. The system was trained and evaluated on the ElectroMagnetic Articulography corpus of Mandarin Accented English (EMA-MAE),consisting of 39 speakers including both native English speakers and native Mandarin speakers speaking English. Results show significant improvement in both correlation and RMSE between the generated and true articulatory trajectories for the new method, with an average correlation of 0.83, representing a 36% relative improvement over the 0.61 correlation obtained with a baseline Hidden Markov Model (HMM)-Gaussian Mixture Model (GMM) inversion framework. To the best of our knowledge, this paper presents the first application of a point-by-point waveform synthesis approach to the problem of acoustic-to-articulatory inversion and the results show improved performance compared to previous methods for speaker dependent acoustic to articulatory inversion.
Phase space learning with neural networks
Authors: Jaime Lopez Garcia, Angel Rivero Jimenez
Date: 2020-06-22 00:00:00

This work proposes an autoencoder neural network as a non-linear generalization of projection-based methods for solving Partial Differential Equations (PDEs). The proposed deep learning architecture presented is capable of generating the dynamics of PDEs by integrating them completely in a very reduced latent space without intermediate reconstructions, to then decode the latent solution back to the original space. The learned latent trajectories are represented and their physical plausibility is analyzed. It is shown the reliability of properly regularized neural networks to learn the global characteristics of a dynamical system's phase space from the sample data of a single path, as well as its ability to predict unseen bifurcations.
Adaptive County Level COVID-19 Forecast Models: Analysis and Improvement
Authors: Stewart W Doe, Tyler Russell Seekins, David Fitzpatrick, Dawsin Blanchard, Salimeh Yasaei Sekeh
Date: 2020-06-16 00:00:00
Tasks: Data Augmentation

Accurately forecasting county level COVID-19 confirmed cases is crucial to optimizing medical resources. Forecasting emerging outbreaks pose a particular challenge because many existing forecasting techniques learn from historical seasons trends. Recurrent neural networks (RNNs) with LSTM-based cells are a logical choice of model due to their ability to learn temporal dynamics. In this paper, we adapt the state and county level influenza model, TDEFSI-LONLY, proposed in Wang et a. [l2020] to national and county level COVID-19 data. We show that this model poorly forecasts the current pandemic. We analyze the two week ahead forecasting capabilities of the TDEFSI-LONLY model with combinations of regularization techniques. Effective training of the TDEFSI-LONLY model requires data augmentation, to overcome this challenge we utilize an SEIR model and present an inter-county mixing extension to this model to simulate sufficient training data. Further, we propose an alternate forecast model, {\it County Level Epidemiological Inference Recurrent Network} (\alg{}) that trains an LSTM backbone on national confirmed cases to learn a low dimensional time pattern and utilizes a time distributed dense layer to learn individual county confirmed case changes each day for a two weeks forecast. We show that the best, worst, and median state forecasts made using CLEIR-Net model are respectively New York, South Carolina, and Montana.
A Bayesian incorporated linear non-Gaussian acyclic model for multiple directed graph estimation to study brain emotion circuit development in adolescence
Authors: Aiying Zhang, Gemeng Zhang, Biao Cai, Tony W. Wilson, Julia M. Stephen, Vince D. Calhoun, Yu-Ping Wang
Date: 2020-06-16 00:00:00
Tasks: Functional Connectivity

Emotion perception is essential to affective and cognitive development which involves distributed brain circuits. The ability of emotion identification begins in infancy and continues to develop throughout childhood and adolescence. Understanding the development of brain's emotion circuitry may help us explain the emotional changes observed during adolescence. Our previous study delineated the trajectory of brain functional connectivity (FC) from late childhood to early adulthood during emotion identification tasks. In this work, we endeavour to deepen our understanding from association to causation. We proposed a Bayesian incorporated linear non-Gaussian acyclic model (BiLiNGAM), which incorporated our previous association model into the prior estimation pipeline. In particular, it can jointly estimate multiple directed acyclic graphs (DAGs) for multiple age groups at different developmental stages. Simulation results indicated more stable and accurate performance over various settings, especially when the sample size was small (high-dimensional cases). We then applied to the analysis of real data from the Philadelphia Neurodevelopmental Cohort (PNC). This included 855 individuals aged 8-22 years who were divided into five different adolescent stages. Our network analysis revealed the development of emotion-related intra- and inter- modular connectivity and pinpointed several emotion-related hubs. We further categorized the hubs into two types: in-hubs and out-hubs, as the center of receiving and distributing information. Several unique developmental hub structures and group-specific patterns were also discovered. Our findings help provide a causal understanding of emotion development in the human brain.
Good Classifiers are Abundant in the Interpolating Regime
Authors: Ryan Theisen, Jason M. Klusowski, Michael W. Mahoney
Date: 2020-06-22 00:00:00
Tasks: Learning Theory

Within the machine learning community, the widely-used uniform convergence framework has been used to answer the question of how complex, over-parameterized models can generalize well to new data. This approach bounds the test error of the worst-case model one could have fit to the data, but it has fundamental limitations. Inspired by the statistical mechanics approach to learning, we formally define and develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers from several model classes. We apply our method to compute this distribution for several real and synthetic datasets, with both linear and random feature classification models. We find that test errors tend to concentrate around a small typical value $\varepsilon^*$, which deviates substantially from the test error of the worst-case interpolating model on the same datasets, indicating that "bad" classifiers are extremely rare. We provide theoretical results in a simple setting in which we characterize the full asymptotic distribution of test errors, and we show that these indeed concentrate around a value $\varepsilon^*$, which we also identify exactly. We then formalize a more general conjecture supported by our empirical findings. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice, and that approaches based on the statistical mechanics of learning may offer a promising alternative.
Exploring Software Naturalness through Neural Language Models
Authors: Luca Buratti, Saurabh Pujar, Mihaela Bornea, Scott McCarley, Yunhui Zheng, Gaetano Rossiello, Alessandro Morari, Jim Laredo, Veronika Thost, Yufan Zhuang, Giacomo Domeniconi
Date: 2020-06-22 00:00:00
Tasks: Language Modeling, Language Modelling

The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks. Present approaches to code analysis depend heavily on features derived from the Abstract Syntax Tree (AST) while our transformer-based language models work on raw source code. This work is the first to investigate whether such language models can discover AST features automatically. To achieve this, we introduce a sequence labeling task that directly probes the language models understanding of AST. Our results show that transformer based language models achieve high accuracy in the AST tagging task. Furthermore, we evaluate our model on a software vulnerability identification task. Importantly, we show that our approach obtains vulnerability identification results comparable to graph based approaches that rely heavily on compilers for feature extraction.