pwc-archive/papers-with-abstracts huggingface dataset:
Joint Content-Context Analysis of Scientific Publications: Identifying Opportunities for Collaboration in Cognitive Science
Authors: Lu Cheng, Girish Ganesan, William He, Daniel Silverston, Harlin Lee, Jacob Gates Foster
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: Community Detection

This work studies publications in the field of cognitive science and utilizes mathematical techniques to connect the analysis of the papers' content (abstracts) to the context (citation, journals). We apply hierarchical topic modeling on the abstracts and community detection algorithms on the citation network, and measure content-context discrepancy to find academic fields that study similar topics but do not cite each other or publish in the same venues. These results show a promising, systemic framework to identify opportunities for scientific collaboration in highly interdisciplinary fields such as cognitive science and machine learning.
Regression modeling on DNA encoded libraries
Authors: Ralph Ma, Gabriel Hart Stocker Dreiman, Fiorella Ruggiu, Adam Joseph Riesselman, Bowen Liu, Keith James, Mohammad Sultan, Daphne Koller
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: Drug Discovery, Graph Neural Network, regression

DNA encoded libraries (DELs) are pooled, combinatorial compound collections where each member is tagged with its own unique DNA barcode. DELs are used in drug discovery for early hit finding against protein targets. Recently, several groups have proposed building machine learning models with quantities derived from DEL datasets. However, DEL datasets have a low signal-to-noise ratio which makes modeling them challenging. To that end, we propose a novel graph neural network (GNN) based regression model that directly predicts enrichment scores from raw sequencing counts while accounting for multiple sources of technical variation and intrinsic assay noise. We show that our GNN regression model quantitatively outperforms standard classification approaches and can be used to find diverse sets of molecules in external virtual libraries.
Scalable Bayesian Optimization Accelerates Process Optimization of Penicillin Production
Authors: Qiaohao Liang, Lipeng Lai
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: Bayesian Optimization, Drug Discovery, global-optimization

While Bayesian Optimization (BO) has emerged as sample-efficient optimization method for accelerating drug discovery, it has rarely been applied to the process optimization of pharmaceutical manufacturing, which traditionally has relied on human-intuition, along with trial-and-error and slow cycles of learning. The combinatorial and hierarchical complexity of such process control also introduce challenges related to high-dimensional design spaces and requirements of larger scale observations, in which BO has typically scaled poorly. In this paper, we use penicillin production as a case study to demonstrate the efficacy of BO in accelerating the optimization of typical pharmaceutical manufacturing processes. To overcome the challenges raised by high dimensionality, we apply a trust region BO approach (TuRBO) for global optimization of penicillin yield and empirically show that it outperforms other BO and random baselines. We also extend the study by leveraging BO in the context of multi-objective optimization, allowing us to further evaluate the trade-offs between penicillin yield, production time, and CO$_2$ emission as by-product. Through quantifying the performance of BO across high-dimensional and multi-objective optimization on drug production processes, we hope to popularize application of BO in this field, and encourage closer collaboration between machine learning and broader scientific communities.
This Earthquake Doesn't Exist
Authors: Artemii Novoselov, Krisztina Sinkovics, Goetz Bokelmann
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12

This study applies Conditional Generative Adversarial Networks (cGAN) to the field of seismology. With GAN, realistic seismic waveforms can be created for various applications, such as augmenting limited seismic data or modeling, or generating realistic noise. A potential and alarming application of GAN is to generate realistic seismic signals that can cause disturbances to the international treaty banning nuclear explosions (CTBT). Results show that the generated seismic waves are nearly indistinguishable from real ones.
High-Dimensional Discrete Bayesian Optimization with Self-Supervised Representation Learning for Data-Efficient Materials Exploration
Authors: Masaki Adachi
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12
Tasks: Bayesian Optimization, Representation Learning

A material exploration model based on high-dimensional discrete Bayesian optimization is introduced. Features were extracted from a large-scale database of ab-initio calculations by self-supervised representation learning. Material exploration was carried out based on 100 prior target values from 6,218 candidate materials. As a baseline, ten human experts of materials science were selected and evaluated their exploration efficiency. Under the same conditions, the proposed discrete algorithm was 1.93 times as efficient as human experts on average, while the conventional continuous algorithm could not outperform them.
Multi-modal Self-supervised Pre-training for Large-scale Genome Data
Authors: Shentong Mo, Xi Fu, Chenyang Hong, Yizhen Chen, Yuxuan Zheng, Xiangru Tang, Yanyan Lan, Zhiqiang Shen, Eric Xing
Date: 2021-09-24 00:00:00
Proceeding: NeurIPS Workshop AI4Scien 2021 12

Open genomic regions, being accessible to regulatory proteins, could act as the on/off switch or amplifier/attenuator of gene expression, and thus reflects the defining characteristics of cell types. Many previous models make predictions from the sequence to the regulatory region, but the interaction between regulatory regions and genes could be complex and differ between cell types. Moreover, current models usually only perform well on the cell types in the training set, which are not generalizable to data-scarce scenarios. In this work, we propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call GeneBERT. Specifically, we simultaneously take the 1d sequence of genome data and a 2d matrix of (transcription factors × regions) as the input, where three pre-training tasks are proposed to improve the robustness and generalizability of our model. We pre-train our model on the ATAC-seq dataset with 17 million gene sequences. We evaluate our GeneBERT on various downstream tasks, including promoter prediction, transaction factor binding sites prediction, disease risks estimation, and RNA-Splicing. Extensive experiments demonstrate the effectiveness of multi-modal and self-supervised pre-training for large-scale genome data.
Type Inference as Optimization
Authors: Eirene V. Pandi, Earl T. Barr, Andrew D. Gordon, Charles Sutton
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: valid, Vocal Bursts Type Prediction

Optionally typed dynamic languages can permit multiple valid type assignments. When this happens, developers can prefer one valid type assignment over another because it better reflects how they think about the program and the problem it solves. Natural type inference (NTI) uses natural language text within source code, such as identifiers, to help choose valid programming language types. A growing body of techniques has been proposed for NTI. These techniques predict types; they seek to return natural type assignments (assignments that reflect developer preferences) while striving for correctness. They are empirically effective, but they are not sound by construction: they do not leverage programming language theory to formalize their algorithms and show correctness and termination. Filling this foundational gap is the purpose of this paper. We are the first to present a detailed algorithm for NTI that is validated with theorems and proofs. Valid type assignments obey logical constraints arising from type rules; natural type assignments obey natural constraints arising from the natural language text associated with a variable and its uses.The core intuition of this work is that logical and natural constraints can interact to speed finding a type valuation that 1. type checks (satisfies the logical constraints) and 2. is most natural.We formulate NTI as a joint optimization problem.  To do this, we define a numerical relaxation over boolean logical constraints that give us a condition that we treat as a hard constraint, while simultaneously we minimize distance from natural constraints, which we treat as soft constraints for our optimization problem. Our main result, the first formal proof of soundness for natural type inference, is that our algorithm always terminates, either with an error or with a tuple that is guaranteed to be a type signature for its input.
Staged compilation of tensor expressions
Authors: Marco Zocca
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12

We present our current progress towards a metaprogramming framework for tensor expressions embedded in Haskell; the system offers a high-level syntax for dimension-annotated linear algebra,  and generates specialized source code corresponding to the input expression.
AutumnSynth: Synthesis of Reactive Programs with Structured Latent State
Authors: Ria Das, Joshua B. Tenenbaum, Armando Solar-Lezama, Zenna Tavares
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: Program Synthesis

The human ability to efficiently discover causal theories of their environments from observations is a feat of nature that remains elusive in machines. In this work, we attempt to make progress on this frontier by formulating the challenge of causal mechanism discovery from observed data as one of program synthesis. We focus on the domain of time-varying, Atari-like 2D grid worlds, and represent causal models in this domain using a programming language called Autumn. Discovering the causal structure underlying a sequence of observations is equivalent to identifying the program in the Autumn language that generates the observations. We introduce a novel program synthesis algorithm, called AutumnSynth, that approaches this synthesis challenge by integrating standard methods of synthesizing functions with an automata synthesis approach, used to discover the model's latent state. We evaluate our method on a suite of Autumn programs designed to express the richness of the domain, and our results signal the potential of our formulation.
Are Transformers All That Karel Needs?
Authors: Abhay Garg, Anand Sriraman, Kunal Pagarey, Shirish Karande
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: All, Decoder, Program Synthesis

Recent works have shown the incredible promise of using neural networks for the task of program synthesis from input-output examples. In this paper, we propose using Transformer-based architectures as a baseline for the program synthesis task on the Karel dataset. Specifically, we leverage DistillGPT as our decoder model to perform program synthesis for the Karel DSL. We show that changing the model architecture from an LSTM to a transformer based architecture, we are able to significantly improve on supervised learning approaches obtaining a top-1 generalization accuracy of 82.4%. Further, applying execution guided search on the output beams of the model increases the accuracy of our approach to 89.64%.
AutoCoder: Leveraging Transformers for Automatic Code Synthesis
Authors: Mrinal Anand, Pratik Kayal, Mayank Singh
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: Program Synthesis

Program synthesis from natural language descriptions is a challenging task. This paper explores two variants of transformer models for the task of program synthesis and showcase higher performance than the existing SOTA models. Through the end, we also discuss the differences in learned representation in these two variants. We demonstrate that the vanilla transformer model has a higher capacity to memorize the training data as compared to the other variant.
Learning Rules with Stratified Negation in Differentiable ILP.
Authors: Giri P Krishnan, Frederick Maier, Ramyaa Ramyaa
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: Inductive logic programming, Negation

Differentiable methods to learn rules (logic programs) have the potential to integrate the interpretability, transferability and low data requirements of inductive logic programming with the noise tolerance of non-symbolic learning. While negation is an essential component of reasoning, incorporating it into a logic programming framework poses several problems (hence its central place in the logic programming and nonmonotonic reasoning communities). Current implementations of differentiable rule learners either exclude negation entirely or else treat it only in passing. In this work, we introduce stratified negation into a differentiable inductive logic programming framework, and we  demonstrate that the resulting system can learn recursive programs in which negation plays a central role. We include examples from multiple domains, e.g.,  arithmetic, graph, sets and lists.
PAC Synthesis of Machine Learning Programs
Authors: Osbert Bastani
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: BIG-bench Machine Learning, image-classification, Image Classification, Learning Theory

We study the problem of synthesizing programs that include machine learning components such as deep neural networks (DNNs). We focus on statistical properties, which are properties expected to hold with high probability---e.g., that an image classification model correctly identifies people in images with high probability. We propose novel algorithms for sketching and synthesizing such programs by leveraging ideas from statistical learning theory to provide statistical soundness guarantees. We evaluate our approach on synthesizing list processing programs that include DNN components used to process image inputs, as well as case studies on image classification and on precision medicine. Our results demonstrate that our approach can be used to synthesize programs with probabilistic guarantees.
Learning Adaptive Control Flow in Transformers for Improved Systematic Generalization
Authors: Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: Systematic Generalization

Despite successes across a broad range of applications, Transformers have limited capability in systematic generalization. The situation is especially frustrating for algorithmic tasks, where they often fail to find intuitive solutions that can be simply expressed in terms of attention patterns. Here we propose two modifications to the Transformer architecture, copy gate and geometric attention, which facilitate learning such intuitive and interpretable solutions to algorithmic problems. Our novel Transformer, called Transformer Control Flow (TCF) achieves 100% length generalization accuracy on the classic compositional table lookup task. The resulting attention and gating patterns are interpretable, demonstrating that the model implements adaptive control flow.
Adversarial Robustness of Program Synthesis Models
Authors: Mrinal Anand, Pratik Kayal, Mayank Singh
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: Adversarial Robustness, Program Synthesis

The resurgence of automatic program synthesis has been observed with the rise of deep learning. In this paper, we study the behaviour of the program synthesis model under adversarial settings. Our experiments suggest that these program synthesis models are prone to adversarial attacks. The proposed transformer model have higher adversarial performance than the current state-of-the-art program synthesis model. We specifically experiment with \textsc{AlgoLisp} DSL-based generative models and showcase the existence of significant dataset bias through different classes of adversarial examples.
Synthesizing Video Trajectory Queries
Authors: Stephen Mell, Favyen Bastani, Stephan Zdancewic, Osbert Bastani
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: Active Learning, Object Tracking

We propose a novel framework called Quivr for synthesizing queries to identify events of interest in video data. For instance, Quivr can be used to identify instances of human driving behaviors such as lane changes or left turns, which are important for designing planning algorithms for autonomous cars. Our queries operate over object trajectories predicted by a deep object tracking model. Then, a query consists of regular expression operators used to compose underlying predicates (e.g., whether a car is in a lane), and selects a subset of trajectories. A key challenge is that queries are difficult for end users to develop: queries must reason about complex spatial and temporal patterns in object trajectories in order to select trajectories of interest, and predicates often include real-valued parameters (e.g., whether two cars are within a certain distance) that can be tedious to manually tune. Thus, Quivr automatically synthesizes queries given examples of trajectories that the query should match. To make the synthesis procedure efficient, we use overapproximations to prune invalid branches of the query search space, including using a quantitative variant of our query semantics to efficiently prune the search space over parameter values. We also propose two optimizations for speeding up the execution of our queries. Finally, we leverage an active learning strategy to disambiguate between multiple consistent candidate queries by collecting additional labels from the user. We evaluate Quivr on a benchmark of 11 tasks, and demonstrate that it can synthesize accurate queries for each task given just a few examples, and that our pruning strategy and optimizations substantially reduce synthesis time.
Augmenting Classic Algorithms with Neural Components for Strong Generalisation on Ambiguous and High-Dimensional Data
Authors: Imanol Schlag, Jürgen Schmidhuber
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12

We augment classic algorithms with learned components to adapt them to domains currently dominated by deep learning models. Two traditional sorting algorithms with learnable neural building blocks are applied to visual data with apriori unknown symbols and rules. The models are quickly and reliably trained end-to-end in a supervised setting. Our models learn symbol representations and generalise better than generic neural network models to longer input sequences.
LazyPPL: laziness and types in non-parametric probabilistic programs
Authors: Hugo Paquet, Sam Staton
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: Clustering, Gaussian Processes, Point Processes, Probabilistic Programming

We introduce LazyPPL, a prototype probabilistic programming library for Haskell. The library emphasises the clarifying power of types, and the connection between non-parametric, stochastic processes and lazy (call by need) evaluation. We illustrate the power of the language with natural specifications of infinite structures including Poisson point processes, Gaussian processes, and Dirichlet Process clustering.
Proof Extraction for Logical Neural Networks
Authors: Thabang Lebese, Ndivhuwo Makondo, Cristina Cornelio, Naweed Khan
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: Automated Theorem Proving

Automated Theorem Provers (ATPs) are widely used for the verification of logical statements. Explainability is one of the key advantages of ATPs: providing an expert readable proof path which shows the inference steps taken to conclude correctness. Conversely, Neuro-Symbolic Networks (NSNs) that perform theorem proving, do not have this capability. We propose a proof-tracing and filtering algorithm to provide explainable reasoning in the case of Logical Neural Networks (LNNs), a special type of Neural-Theorem Prover (NTP).
A Genetic Programming Approach To Zero-Shot Neural Architecture Ranking
Authors: Yash Akhauri, Juan Pablo Munoz, Ravishankar Iyer, Nilesh Jain
Date: 2021-10-08 00:00:00
Proceeding: NeurIPS Workshop AIPLANS 2021 12
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), image-classification, Image Classification, Neural Architecture Search, Program Synthesis, speech-recognition, Speech Recognition

Neural networks are becoming increasingly ubiquitous in a wide range of use cases. A primary hurdle in deploying neural networks in many scenarios is the tedious and difficult neural network architectural design process, which was reliant on expert knowledge and iterative design. Neural Architecture Search (NAS) reduces the human effort required for design, but still has considerable resource requirements and is extremely slow. To address the inefficiencies of conventional NAS, Zero-Shot NAS is a new paradigm, which introduces zero shot neural architecture scoring metrics (NASMs) to identify good neural network designs without training them. While applying Zero Shot NASMs is cheap and requires no training resources, we identify that there is a lack of NASMs that generalize well across neural architecture design spaces. In this paper, we present a program representation for NASMs and automate its search with genetic programming. We discover effective NASMs for Image Classification as well as Automatic Speech Recognition. We believe that our work indicates a new direction for NASM design and can greatly benefit from recent advances in program synthesis.
RASL: Relational Algebra in Scikit-Learn Pipelines
Authors: Chirag Sahni, Kiran Kate, Avraham Shinnar, Hoang Thanh Lam, Martin Hirzel
Date: 2021-10-03 00:00:00
Proceeding: NeurIPS Workshop DBAI 2021 12
Tasks: BIG-bench Machine Learning

Integrating data preparation with machine-learning (ML) pipelines has been a long- standing challenge. Prior work tried to solve it by building new data processing platforms such as MapReduce or Spark, and then implementing new libraries of ML algorithms for those. But despite the availability of these platforms, many ML practitioners continue to use scikit-learn instead, owing to its clean design and rich set of algorithms. Therefore, this paper proposes a different approach: instead of extending a data processing platform for ML, extend an ML library for data processing. Specifically, this paper proposes RASL, an open-source library of relational algebra (RA) operators for scikit-learn (SL). We illustrate RASL with a detailed case study involving joins and aggregation across multi-table input data. We hope our approach will lead to cleaner integration of data preparation with machine learning in practice.
DRL-Clusters: Buffer Management with Clustering based Deep Reinforcement Learning
Authors: Kai Li, Qi Zhang, Lei Yu, Hong Min
Date: 2021-10-03 00:00:00
Proceeding: NeurIPS Workshop DBAI 2021 12
Tasks: Clustering, Deep Reinforcement Learning, Management, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Buffer cache has been widely implemented in database systems to reduce disk I/Os. Existing database systems typically use heuristic-based algorithms for buffer replacement, which cannot dynamically adapt to changing workload patterns. This paper proposes a deep reinforcement learning-based approach, DRL-Clusters, to manage the buffer pool when handling changing workloads. DRL-Clusters can dynamically adapt to different workload patterns without incurring high inference overhead and miss ratio with page re-clustering and continuous interactions with the cache environment. Our evaluation results demonstrate that DRL-Clusters can achieve a lower or comparable miss ratio than the heuristic policies while reducing 13.3% - 26.8% page access overhead under changing workloads.