pwc-archive/papers-with-abstracts huggingface dataset:
Automation for Interpretable Machine Learning Through a Comparison of Loss Functions to Regularisers
Authors: Amy Parkes, Adam Sobey, Josef Camilleri, Dominic Hudson
Date: 2021-05-21 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: BIG-bench Machine Learning, Interpretable Machine Learning, regression

To increase the ubiquity of machine learning it needs to be automated. Automation is cost-effective as it allows experts to spend less time tuning the approach, which leads to shorter development times. However, while this automation produces highly accurate architectures, they can be uninterpretable, acting as `black-boxes' which produce low conventional errors but fail to model the underlying input-output relationships---the ground truth. This paper explores the use of the Fit to Median Error measure in machine learning regression automation, using evolutionary computation in order to improve the approximation of the ground truth. When used alongside conventional error measures it improves interpretability by regularising learnt input-output relationships to the conditional median. It is  compared to traditional regularisers to illustrate that the use of the Fit to Median Error produces regression neural networks which model more consistent input-output relationships. The problem considered is ship power prediction using a fuel-saving air lubrication system, which is highly stochastic in nature. The networks optimised for their Fit to Median Error are shown to approximate the ground truth more consistently, without sacrificing conventional Minkowski-r error values.
Auxiliary learning induced graph convolutional networks
Authors: Gengchen Duan, Taisong Jin, Rongrong Ji, Ling Shao, Baochang Zhang, Feiyue Huang, Yongjian Wu
Date: 2021-05-21 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Auxiliary Learning, Classification, Link Prediction, Meta-Learning, Node Classification, Pseudo Label

In this article, we propose a novel auxiliary learning induced graph convolutional network in a multi-task fashion. Specifically, both the link prediction and pseudo label generation are used as two auxiliary tasks to complement the primary task of node classification. Those two auxiliary tasks are jointly trained with the primary node classification task via the graph meta-learning strategy. The experimental results demonstrate that the proposed method consistently and significantly outperforms the existing methods and achieves state-of-the-art node classification results on the benchmark citation network datasets.
How not to Lie with a Benchmark: Rearranging NLP Learderboards
Authors: Tatiana Shavrina, Valentin Malykh
Date: 2021-05-21 00:00:00
Proceeding: NeurIPS 2021 12

Proper model ranking and comparison with a human level is an essential requirement for every benchmark to be a reliable measurement of the model quality. Nevertheless, the methods for model comparison could have a fundamental flaw - the arithmetic mean of separate metrics is used for all tasks of different complexity, different size of test and training sets.
In this paper, we examine popular NLP benchmarks' overall scoring methods and rearrange the models by geometric and harmonic mean (appropriate for averaging rates) according to their reported results. We analyze several popular benchmarks including GLUE, SuperGLUE, XGLUE, and XTREME. The analysis shows that e.g. human level on SuperGLUE is still not reached, and there is still room for improvement for the current models.
EuclidNets: Combining hardware and architecture design for Efficient Inference and Training
Authors: Mariana Oliveira Prazeres, Xinlin Li, Vahid Partovi Nia, Adam M Oberman
Date: 2021-05-21 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Quantization

In order to deploy deep neural networks on edge devices, compressed (resource efficient) networks need to be developed.  While established compression methods, such as quantization, pruning, and architecture search are designed for conventional hardware,  further gains are possible if compressed architectures are coupled with novel hardware designs.   In this work, we propose EuclidNet, a compressed network designed to be implemented on hardware which replaces multiplication, $wx$, with squared difference $(x-w)^2$.  EuclidNet allows for a low precision hardware implementation which is about twice as efficient (in term of logic gate counts) as the comparable conventional hardware, with acceptably small loss of accuracy.   Moveover, the network can be trained and quantized using standard methods, without requiring additional training time.  Codes and pre-trained models are available at \url{http://github.com/anonymous/}.
A Distance Correlation-based Kernel for Nonlinear Causal Clustering in Heterogeneous Populations
Authors: Alex Markham, Moritz Grosse-Wentrup
Date: 2021-05-21 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Clustering

We consider the problem of causal structure learning in the setting of heterogeneous populations, i.e., populations in which a single causal structure does not adequately represent all population members, as is common in biological and social sciences. To this end, we introduce a distance covariance-based kernel designed specifically to measure the similarity between the underlying nonlinear causal structures of different samples.This kernel enables us to perform clustering to identify the homogeneous subpopulations. Indeed, we prove the corresponding feature map is a statistically consistent estimator of nonlinear independence structure, rendering the kernel itself a statistical test for the hypothesis that sets of samples come from different generating causal structures. We can then use existing methods to learn a causal structure for each of these subpopulations. We demonstrate using our kernel for causal clustering with an application in genetics, allowing us to reason about the latent transcription factor networks regulating measured gene expression levels.
Normalization of Language Embeddings for Cross-Lingual Alignment
Authors: Prince Osei Aboagye, Jeff Phillips, Yan Zheng, Chin-Chia Michael Yeh, Junpeng Wang, Wei zhang, Liang Wang, Hao Yang
Date: 2021-05-21 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Translation

Learning a good transfer function to map the word vectors from two languages into a shared cross-lingual word vector space plays a crucial role in cross-lingual NLP. It is useful in translation tasks and important in allowing complex models built on a high-resource language like English to be directly applied on an aligned low resource language.  While previous approaches have identified Procrustes and other techniques to align language models with some success, it has recently been identified that structural differences (for instance, due to differing word frequency) create different profiles for various monolingual embedding. When these profiles differ across languages, it correlates with how well languages can align and their performance on cross-lingual downstream tasks.  In this work, we develop a very general language embedding normalization procedure, building and subsuming various previous approaches, which removes these structural profiles across languages without destroying their intrinsic meaning.  We demonstrate that meaning is retained, and alignment is improved on similarity, translation, and cross-language classification tasks.  Our proposed normalization clearly outperforms all prior approaches like centering and vector normalization on each task and with each alignment approach.
Unsupervised Dynamic Routing Via Competition Over Network Loss
Authors: Lei Zhao, Lin Wang, Jiuqiao Yu
Date: 2021-05-21 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Philosophy

  This paper proposes a novel neural network architecture that can simultaneously do normal network optimizing while attaining the ability of unsupervised learning. Almost all existing unsupervised learning algorithms are based on doing calculations on the input space or feature space, this paper proposes a new possibility to discover a structure in the functional space without supervision. Using the self-organizing map over the competition of the loss of individual neural column, we route the input to the most appropriate modules dynamically, by doing this we separate the input functional space into different sub spaces which are represented by each individual neural column. At the end of the paper, we propose several possible architectures based on the philosophy of this paper that could build a neural network system block by block.
Mean-Square Analysis with An Application to Optimal Dimension Dependence of Langevin Monte Carlo
Authors: Ruilin Li, Hongyuan Zha, Molei Tao
Date: 2021-05-21 00:00:00
Proceeding: NeurIPS 2021 12

Sampling algorithms based on discretizations of Stochastic Differential Equations (SDEs) compose a rich and popular subset of MCMC methods. This work provides a general framework for the non-asymptotic analysis of sampling error in 2-Wasserstein distance, which also leads to a bound of mixing time. The method applies to any consistent discretization of contractive SDEs. When applied to Langevin Monte Carlo algorithm, it establishes $\widetilde{\mathcal{O}}\left(\frac{\sqrt{d}}{\epsilon}\right)$ mixing time, without warm start, under the common log-smooth and log-strongly-convex conditions, plus a growth condition on the potential of target measures at infinity. This bound improves the best previously known $\widetilde{\mathcal{O}}\left(\frac{d}{\epsilon}\right)$ result and is optimal in both dimension $d$ and accuracy tolerance $\epsilon$ for log-smooth and log-strongly-convex target measures. Our theoretical analysis is further validated by numerical experiments.
Distributional Generalization: Characterizing Classifiers Beyond Test Error
Authors: Preetum Nakkiran, Yamini Bansal
Date: 2021-05-21 00:00:00
Proceeding: NeurIPS 2021 12

We present a new set of empirical properties of interpolating classifiers, including neural networks, kernel machines and decision trees.
Informally, the output distribution of an interpolating classifier matches
the distribution of true labels, when conditioned on certain subgroups of the input space. For example, if we mislabel 30% of images of dogs as cats in the train set of CIFAR-10, then a ResNet trained to interpolation will
in fact mislabel roughly 30% of dogs as cats on the *test set* as well, while leaving other classes unaffected.
These behaviors are not captured by classical generalization, which would only consider the average error over the inputs,
and not *where* these errors occur.
We introduce and experimentally validate a formal conjecture that specifies the subgroups for which we expect this distributional closeness.
Further, we show that these properties can be seen as a new form of generalization, which advances our understanding of the implicit bias of interpolating methods.
ML Agent Safety Mechanisms based on Counterfactual Planning
Authors: Koen Holtman
Date: 2021-05-21 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: BIG-bench Machine Learning, counterfactual, Counterfactual Planning

We present counterfactual planning as a design approach for creating a
range of safety mechanisms for machine learning agents. 
We specifically target the safety problem of keeping control over
hypothetical future AGI agents.
The key step in counterfactual planning is to use the agent's
machine learning system to construct a counterfactual world model,
designed to be different from the real world the agent is in.  A
counterfactual planning agent determines the action that best
maximizes expected utility in this counterfactual planning world, and
then performs the same action in the real world.
The design approach is built around a two-diagram graphical notation
that provides a specific vantage point on the construction of online
machine learning agents, a vantage point designed to make the problem
of control more tractable.
We show two examples where the construction of a counterfactual planning world
acts to suppress certain unsafe agent incentives, incentives for the
agent to take control over its own safety mechanisms.
Adaptive Pseudo-labeling for Quantum Calculations
Authors: Kexin Huang, Vishnu Sresht, Brajesh Rai, Mykola Bordyuh
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: Out-of-Distribution Generalization

Machine learning models have recently shown promise in predicting molecular quantum chemical properties. However, the path to real-life adoption requires (1) learning under low-resource constraint and (2) out-of-distribution generalization to unseen, structurally diverse molecules. We observe that these two challenges originate from label scarcity issue. We hypothesize that pseudo-labeling on vast array of unlabeled molecules can serve as proxies as gold-label to greatly expand the training labeled data. The challenge in pseudo-labeling is to prevent the bad pseudo-labels from biasing the model. We develop a simple and effective strategy Pseudo-Sigma that can assign pseudo-labels, detect bad pseud-labels through evidential uncertainty, and then prevent them from biasing the model using adaptive weighting. Empirically, Pseudo-Sigma improves quantum calculations accuracy across full data, low data and out-of-distribution settings.
Single Reference Frequency Loss for Multi-frequency Wavefield Representation using Physics-Informed Neural Networks
Authors: Xinquan Huang, Tariq Alkhalifah
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12

Physics-informed neural networks (PINNs) can offer approximate multidimensional functional solutions to the Helmholtz equation that are flexible, require low memory, and have no limitations on the shape of the solution space. However, the neural network (NN) training can be costly and the cost dramatically increases as we train for multi-frequency wavefields by adding frequency to the NN multidimensional function, as the variation of the wavefield with frequency adds more complexity to the NN training.  Thus, we propose a new loss function for the NN multidimensional input training that allows us to seamlessly include frequency as a dimension. We specifically utilize the linear relation between frequency and wavenumber (the wavefield space representation) to incorporate a reference frequency scaling to the loss function. As a result, the effective wavenumber of the wavefield solution as a function of frequency remains stationary reducing the learning burden on the NN function. We demonstrate the effectiveness of this modified loss function on a layered model.
Scientific Argument with Supervised Learning
Authors: Jeffrey W Lockhart, Abigail Z Jacobs
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: BIG-bench Machine Learning, scientific discovery

The use of machine learning (ML) for scientific discovery has enabled data-driven approaches to new and old questions alike. We argue that scientific arguments based on algorithms for discovery hold the potential to reinforce existing assumptions about phenomena, under the guise of testing them. Using examples from image-based biological classification, we show how scientific arguments using supervised learning can contribute to unintended, unrealistic, or under-evidenced claims.
Human-in-the-loop for a Disconnection Aware Retrosynthesis
Authors: Andrea Byekwaso, Philippe Schwaller, Alain C. Vaucher, Alessandra Toniato, Teodoro Laino
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: Retrosynthesis, Single-step retrosynthesis

 Retrosynthesis is an approach commonly undertaken when considering the manufacture of novel molecules. During this process, a target molecule is broken down and analyzed by considering the bonds to be changed as well as the functional group interconversion. In modern computer-assisted synthesis planning tools, the predictions of these changes are typically carried out automatically. However there may be some benefit to the decision being guided by those executing the process: typically, chemists have a clear idea where the retrosynthetic change should happen, but not how such a transformation is to be realized. Using a data-driven model, the retrosynthesis task can be further explored by giving chemists the option to explore specific disconnections. In this work, we design an approach to provide this option by adapting a transformer-based model for single-step retrosynthesis. The model takes as input a product SMILES string, in which the atoms where the transformation should occur are tagged accordingly. This model predicts precursors corresponding to a disconnection occurring in the correct location in 88.9\% of the test set reactions. The assessment with a forward prediction model shows that 76\% of the predictions are chemically correct, with 14.1\% perfectly matching the ground truth.
Multiple Sequential Learning Tasks Represented in Recurrent Neural Networks
Authors: Shaonan Wang, Bingyu Liu
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12

Our brain can flexibly perform a variety of sequential learning tasks including music, language, and mathematics, but the underlying mechanism hasn't been elucidated in traditional experimental and modeling studies which were designed for only one task at a time. From the computational perspective, we hypothesize that the working mechanism of a multitask model can provide a possible solution to that of brains. Therefore, we trained a single recurrent neural network to perform 8 sequential learning tasks that depend on working memory, structure extraction, categorization, and other cognitive processes. After training, the model can learn sophisticated information holding and erasing strategies to perform multitasks simultaneously. More interestingly, the model learns to reuse neurons to encode similar task features. Hopefully, this work can provide a computational platform to investigate the neural representations of cognitive sequential learning ability.
Apertures in Agriculture Seeking Attention
Authors: Sanchita Das, Ramya Srinivasan
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: Position

Agriculture is arguably one economic activity that serves as the backbone of human civilization---It is the provider of the most essential lifeline for human survival, namely, food. Today, we stand at a juncture where world over, all key stakeholders involved in this activity, i.e., producers (farmers), consumers, and the planet are facing grave concerns. Formulating these concerns concretely and leveraging AI methods in developing solution strategies can potentially help alleviate major risks to food systems. However, local contexts such as cultural practices, physical terrains, and socio-economic status pose unique challenges in being able to directly employ existing AI techniques across geographies. In this position paper, we highlight some such key challenges, specifically focusing on India and the developing world. We outline the problems of key stakeholders, and identify some gaps that need to be filled in addressing these problems.
Multi-task Learning with Domain Knowledge for Molecular Property Prediction
Authors: Shengchao Liu, Meng Qu, Zuobai Zhang, Huiyu Cai, Jian Tang
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: Drug Discovery, Molecular Property Prediction, Multi-Task Learning, Prediction, Property Prediction, Relation, Structured Prediction

Multi-task learning for molecular property prediction is becoming increasingly important in drug discovery. However, in contrast to other domains, the performance of multi-task learning in drug discovery is still not satisfying as the number of labeled data for each task is too limited, which calls for additional data to complement the data scarcity. In this paper, we study multi-task learning for molecule property prediction in a different setting, where a relation graph between different tasks is available. We first extract a dataset including ~400 tasks as well as a relation graph between different tasks. Then we systematically investigate modeling the relation between different tasks: (1) in the latent space by learning effective task representations on the task relation graph; (2) and in the output space via structured prediction methods (e.g., energy-based methods). Empirical results prove the effectiveness of our proposed approaches.
Identification of Enzymatic Active Sites with Unsupervised Language Modeling
Authors: Loïc Kwate Dassi, Matteo Manica, Daniel Probst, Philippe Schwaller, Yves Gaetan Nana Teukam, Teodoro Laino
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: Language Modeling, Language Modelling

The first decade of genome sequencing saw a surge in the characterization of proteins with unknown functionality. Even still, more than 20% of proteins in well-studied model animals have yet to be identified, making the discovery of their active site one of biology's greatest puzzle.
Herein, we apply a Transformer architecture to a language representation of bio-catalyzed chemical reactions to learn the signal at the base of the substrate-active site atomic interactions. The language representation comprises a reaction simplified molecular-input line-entry system (SMILES) for substrate and products, complemented with amino acid (AA) sequence information for the enzyme. We demonstrate that by creating a custom tokenizer and a score based on attention values, we can capture the substrate-active site interaction signal and utilize it to determine the active site position in unknown protein sequences, unraveling complicated 3D interactions using just 1D representations.
This approach exhibits remarkable results and can recover, with no supervision,  31.51% of the active site when considering co-crystallized substrate-enzyme structures as a ground-truth, vastly outperforming approaches based on sequence similarities only. Our findings are further corroborated by docking simulations on the 3D structure of few enzymes. This work confirms the unprecedented impact of natural language processing and more specifically of the Transformer architecture on domain-specific languages, paving the way to effective solutions for protein functional characterization and bio-catalysis engineering.
Improving the spectral resolution of fMRI signals through the temporal de-correlation approach
Authors: Wenjun Bai, Junichiro Yoshimoto
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: EEG, Electroencephalogram (EEG), Time Series, Time Series Analysis

The inherent infra-slow, narrowband signal thwarts the fMRI modality in considering as an optimal neuroimaging modality to its alternatives, e.g., EEG and MEG, in investigating the spectral character of cortical activities. To enhance the spectral resolution of fMRI signal, we put forward a novel linear transformation approach to encourage both the multivariate fMRI time series and their derived temporal derivatives to be temporal de-correlated with each other. Thorough empirical validations of our temporal de-correlation approach on multiple independent fMRI datasets are presented, along with the attached empirical comparison of several alternative methods. Throughout all employed fMRI datasets, we observe a general increment on spectral resolution of temporal de-correlated fMRI signals in terms of wider frequency bandwidth, and more distinctive spectral characters to the original signals.