pwc-archive/papers-with-abstracts huggingface dataset:
An Autoregressive Flow Model for 3D Molecular Geometry Generation from Scratch
Authors: Youzhi Luo, Shuiwang Ji
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4

We consider the problem of generating 3D molecular geometries from scratch. While multiple methods have been developed for generating molecular graphs, generating 3D molecular geometries from scratch is largely under-explored. In this work, we propose G-SphereNet, a novel autoregressive flow model for generating 3D molecular geometries. G-SphereNet employs a flexible sequential generation scheme by placing atoms in 3D space step-by-step. Instead of generating 3D coordinates directly, we propose to determine 3D positions of atoms by generating distances, angles and torsion angles, thereby ensuring both invariance and equivariance properties. In addition, we propose to use spherical message passing and attention mechanism for conditional information extraction. Experimental results show that G-SphereNet outperforms previous methods on random molecular geometry generation and targeted molecule discovery tasks.
Topological Vanilla Transfer Learning
Authors: Nicholas George Bishop, Lau Truong Nguyen, Hieu Trung Thai, Thomas Davies, Long Tran-Thanh
Date: 2021-09-29 00:00:00
Tasks: Transfer Learning

In this paper we investigate the connection of topological similarity between source and target tasks with the efficiency of vanilla transfer learning (i.e., transfer learning without retraining) between them. We discuss that while it is necessary to have strong topological similarity between the source and target tasks, the other direction does not hold (i.e., it is not a sufficient condition). To this extent, we further investigate what can be done in order guarantee efficient feature representation transfer that is needed for such vanilla transfer learning.  To answer this, we provide a matrix transformation based homeomorphism (i.e., topology preserving mapping) that significantly improves the transferability measures while keeping the topological properties of the source and target models intact. We prove that while finding such optimal matrix transformation is typically APX-hard, there exists an efficient randomised algorithm that achieves probably correct approximation guarantees. To demonstrate the effectiveness of our approach, we run a number of experiments on transferring features between ImageNet and a number of other datasets (CIFAR-10, CIFAR-100, MNIST, and ISIC 2019) with a variety of pre-trained models (ResNet50, EfficientNetB3, and InceptionV3). These numerical results show that our matrix transformation can increase the performance (measured by F-score) by up to 3-fold.
Amortized Posterior on Latent Variables in Gaussian Process
Authors: Qing Sun
Date: 2021-09-29 00:00:00

Deep neural networks have achieved impressive performance on a variety of domains. However, performing tasks in partially observed, dynamic environments is still an open problem. Gaussian Process (GPs) is well-known for capturing uncertainty in model parameters. However, it simply assumes a fixed Gaussian prior on latent variables. Thus, agents are not able to update their beliefs about latent variables as observing data points. Instead, in this paper, we propose to replace the prior with an amortized posterior, which enables quick adaptation, especially to abrupt changes. Experiments show that our proposed method can adjust behaviors on the fly (e.g., blind “Predator” take 56% more chance to approach “Prey”), correct mistakes to escape bad situations (e.g., 25% ↑ on avoiding repeating to hit objects with negative rewards), and update beliefs quickly (e.g., 9% faster convergence rate on learning new concepts).
Compressing Transformer-Based Sequence to Sequence Models With Pre-trained Autoencoders for Text Summarization
Authors: Ala Alam Falaki, Robin Gras
Date: 2021-09-29 00:00:00
Tasks: Decoder, Text Summarization

We proposed a technique to reduce the decoder’s number of parameters in a sequence to sequence (seq2seq) architecture for automatic text summarization. This approach uses a pre-trained AutoEncoder (AE) trained on top of a pre-trained encoder to reduce the encoder’s output dimension and allow to significantly reduce the size of the decoder. The ROUGE score is used to measure the effectiveness of this method by comparing four different latent space dimensionality reductions: 96%, 66%, 50%, 44%. A few well-known frozen pre-trained encoders (BART, BERT, and DistilBERT) have been tested, paired with the respective frozen pre-trained AEs to test the reduced dimension latent space’s ability to train a 3-layer transformer decoder. We also repeated the same experiments on a small transformer model that has been trained for text summarization. This study shows an increase of the R-1 score by 5% while reducing the model size by 44% using the DistilBERT encoder, and competitive scores for all the other models associated to important size reduction.
GraphEBM: Towards Permutation Invariant and Multi-Objective Molecular Graph Generation
Authors: Meng Liu, Keqiang Yan, Bora Oztekin, Shuiwang Ji
Date: 2021-09-29 00:00:00
Tasks: Drug Discovery, Graph Generation, Molecular Graph Generation

Although significant progress has been made in molecular graph generation recently, permutation invariance and multi-objective generation remain to be important but challenging goals to achieve. In this work, we propose GraphEBM, a molecular graph generation method via energy-based models (EBMs), as an exploratory work to perform permutation invariant and multi-objective molecule generation. Particularly, thanks to the flexibility of EBMs and our parameterized permutation-invariant energy function, our GraphEBM can define a permutation invariant distribution over molecular graphs. We learn the energy function by contrastive divergence and generate samples by Langevin dynamics. In addition, to generate molecules with a specific desirable property, we propose a simple yet effective learning strategy, which pushes down energies with flexible degrees according to the properties of corresponding molecules. Further, we explore to use our GraphEBM for generating molecules towards multiple objectives via compositional generation, which is practically desired in drug discovery. We conduct comprehensive experiments on random, single-objective, and multi-objective molecule generation tasks. The results demonstrate our method is effective.
What’s Wrong with Deep Learning in Tree Search for Combinatorial Optimization
Authors: Maximilian Böther, Otto Kißig, Martin Taraz, Sarel Cohen, Karen Seidel, Tobias Friedrich
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Combinatorial Optimization

Combinatorial optimization lies at the core of many real-world problems. Especially since the rise of graph neural networks (GNNs), the deep learning community has been developing solvers that derive solutions to NP-hard problems by learning the problem-specific solution structure. However, reproducing the results of these publications proves to be difficult. We make three contributions. First, we present an open-source benchmark suite for the NP-hard Maximum Independent Set problem, in both its weighted and unweighted variants. The suite offers a unified interface to various state-of-the-art traditional and machine learning-based solvers. Second, using our benchmark suite, we conduct an in-depth analysis of the popular guided tree search algorithm by Li et al. [NeurIPS 2018], testing various configurations on small and large synthetic and real-world graphs. By re-implementing their algorithm with a focus on code quality and extensibility, we show that the graph convolution network used in the tree search does not learn a meaningful representation of the solution structure, and can in fact be replaced by random values. Instead, the tree search relies on algorithmic techniques like graph kernelization to find good solutions. Thus, the results from the original publication are not reproducible. Third, we extend the analysis to compare the tree search implementations to other solvers, showing that the classical algorithmic solvers often are faster, while providing solutions of similar quality. Additionally, we analyze a recent solver based on reinforcement learning and observe that for this solver, the GNN is responsible for the competitive solution quality.
Learning Stochastic Representations of Physical Systems
Authors: Jakub Rydzewski, Omar Valsson
Date: 2021-09-29 00:00:00
Tasks: Dimensionality Reduction

Learning representations of physical systems is an important problem at the interface of statistical physics and machine learning. Recently, there has been a growing interest in devising methods to analyze high-dimensional simulation data generated by unbiased or biased samplers. As statistical physics systems consisting of $N \gg 1$ objects tend to have many degrees of freedom, dimensionality reduction methods are of particular interest. Here, we use a new method, multiscale reweighted stochastic embedding (MRSE), to analyze handwritten digits data sets and a biased trajectory of alanine tetrapeptide, and show that we can reconstruct low-dimensional representations of these data sets while retaining the most informative characteristics of their high-dimensional representation.
Adaptive Speech Duration Modification using a Deep-Generative Framework
Authors: Ravi Shankar, Archana Venkataraman
Date: 2021-09-29 00:00:00
Tasks: Decoder, Dynamic Time Warping, Voice Conversion

We propose the first method to adaptively modify the duration of a given speechsignal.  Our approach uses a Bayesian framework to define a latent attention mapthat links frames of the input and target utterances. We train a masked convolu-tional encoder-decoder network to generate this attention map via a stochastic ver-sion of the mean absolute error loss function. Our model also predicts the lengthof the target speech signal using the encoder embeddings, which determines thenumber of time steps for the decoding operation. During testing, we generate theattention map as a proxy for the similarity matrix between the given input speechand an unknown target speech signal. Using this similarity matrix, we compute awarping path of alignment between the two signals. Our experiments demonstratethat this adaptive framework produces similar results to dynamic time warping,which relies on a known target signal, on both voice conversion and emotion con-version tasks. We also show that the modified speech utterances achieve high userquality ratings, thus highlighting the practical utility of our method.
Contrastive Clustering to Mine Pseudo Parallel Data for Unsupervised Translation
Authors: Xuan-Phi Nguyen, Hongyu Gong, Yun Tang, Changhan Wang, Philipp Koehn, Shafiq Joty
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Clustering, Machine Translation, Translation, Unsupervised Machine Translation

Modern unsupervised machine translation systems mostly train their models by generating synthetic parallel training data from large unlabeled monolingual corpora of different languages through various means, such as iterative back-translation. However, there may exist small amount of actual parallel data hidden in the sea of unlabeled data, which has not been exploited. We develop a new fine-tuning objective, called Language-Agnostic Constraint for SwAV loss, or LAgSwAV, which enables a pre-trained model to extract such pseudo-parallel data from the monolingual corpora in a fully unsupervised manner. We then propose an effective strategy to utilize the obtained synthetic data to augment unsupervised machine translation. Our method achieves the state of the art in the WMT'14 English-French, WMT'16 German-English and English-Romanian bilingual unsupervised translation tasks, with 40.2, 36.8, 37.0 BLEU, respectively. We also achieve substantial improvements in the FLoRes low-resource English-Nepali and English-Sinhala unsupervised tasks with 5.3 and 5.4 BLEU, respectively.
Two Regimes of Generalization for Non-Linear Metric Learning
Authors: Mark Kozdoba, Shie Mannor
Date: 2021-09-29 00:00:00
Tasks: Generalization Bounds, Metric Learning, Vocal Bursts Valence Prediction

A common approach to metric learning is to seek an embedding of the input data that behaves well with respect to the labels. While generalization bounds for linear embeddings are known, the non-linear case is not well understood.  In this work we fill this gap by 
providing uniform generalization guarantees for the case where the metric is induced by a neural network type embedding of the data. Specifically, we discover and analyze two regimes of behavior of the networks, which are roughly related to the sparsity of the last layer. The bounds corresponding to the first regime are based on the spectral and $(2,1)$-norms of the weight matrices, while the second regime bounds use the $(2,\infty)$-norm at the last layer, and are significantly stronger when the last layer is dense. In addition, we empirically evaluate the behavior of the bounds for networks trained with SGD on the MNIST and 20newsgroups datasets. In particular, we demonstrate that both regimes occur naturally on realistic data.
A precortical module for robust CNNs to light variations
Authors: Rita Fioresi, Janko Petkovic
Date: 2021-09-29 00:00:00
Tasks: image-classification, Image Classification

We present a simple mathematical model for the mammalian low visual pathway,  taking into account its key elements: retina, lateral
geniculate nucleus (LGN), primary visual cortex (V1). The analogies between the cortical level of the visual system and the  structure of popular CNNs, used in image classification tasks, suggests  the introduction of an additional preliminary convolutional module inspired  to  precortical neuronal circuits to improve robustness with respect  to global light intensity and contrast variations in the input images.  We validate our hypothesis on the popular databases MNIST, FashionMNIST and  SVHN, obtaining significantly more robust CNNs with respect  to these variations, once such extra module is added.
Gradient-Guided Importance Sampling for Learning Discrete Energy-Based Models
Authors: Meng Liu, Haoran Liu, Shuiwang Ji
Date: 2021-09-29 00:00:00
Tasks: Graph Generation

Learning energy-based models (EBMs) is known to be difficult especially on discrete data where gradient-based learning strategies cannot be applied directly. Although ratio matching is a sound method to learn discrete EBMs, it suffers from expensive computation and excessive memory requirement, thereby resulting in difficulties for learning EBMs on high-dimensional data. In this study, we propose ratio matching with gradient-guided importance sampling (RMwGGIS) to alleviate the above limitations. Particularly, we leverage the gradient of the energy function w.r.t. the discrete data space to approximately construct the provable optimal proposal distribution, which is subsequently used by importance sampling to efficiently estimate the original ratio matching objective. We perform experiments on density modeling over synthetic discrete data and graph generation to evaluate our proposed method. The experimental results demonstrate that our method can significantly alleviate the limitations of ratio matching and perform more effectively in practice.
Generate Triggers in Neural Relation Extraction
Authors: Liu Yujiang
Date: 2021-09-29 00:00:00
Tasks: Decoder, Relation, Relation Classification, Relation Extraction

In the relation extraction task, the relationship between two entities is determined by some specific words in their source text. These words are called relation triggers, which are the evidence to explain the relationship; other words are called ir-relevant words. The current relationship extraction neural network model aims at identifying the relation type between two entities mentioned in source text by encoding the text and entities. However, these models cannot output the relation triggers, but only gives the result of relation classification. Although models can generate weights for every single word through the improvement of attention mechanism, the weights will be affected by irrelevant words essentially, which are not required by the relation extraction task. In order to output re-lation triggers accurately, we propose a novel training frame-work for Relation Extraction (RE) that reduces the negative effect of irrelevant words on them in the encoding stage. In specific, we leverage Evolutive Mask based Point Network (EMPN) as a decoder to generate relation triggers and encode these words again. For an ordered output in relation triggers, we utilize order loss to constrain the output order in them. Ex-tensive experiment results demonstrate that the effectiveness of our proposed model achieves state-of-the-art performance on three RE benchmark datasets.
A Study of Aggregation of Long Time-series Input for LSTM Neural Networks
Authors: Nitzan Farhi, Yuval Shavitt
Date: 2021-09-29 00:00:00
Tasks: Time Series, Time Series Analysis, Time Series Forecasting

Time series forecasting is the process of using time series data to create a prediction model. 
Long-short term memory (LSTM) models are the state-of-the-art for time-series forecasting.
However, LSTMs can handle limited length input mostly since when the samples enter the model in sequence, 
the oldest samples need to propagate through the LSTM cells self loop for each new sample and thus their data diminishes in this process.

This limits the length of the history that can be used in the training for each time epoch. The common way of handling this problem is by partitioning time records to uniform intervals, averaging each interval, and feeding the LSTM with rather short sequences, but each represents data from a longer history. 
    
In this paper, we show that this common data aggregation method is far from optimal. We generalize the method of partitioning the data, and suggest an Exponential partitioning. We show that non-uniformly partitioning, and especially Exponential partitioning improves LSTM accuracy, significantly. Using other aggregation functions (such as median or maximum) are shown to further improve the accuracy. Overall, using 7 public datasets we show an improvement in accuracy by 6% to 27%.
GCN-SL: Graph Convolutional Network with Structure Learning for Disassortative Graphs
Authors: Mengying Jiang, Guizhong Liu, Yuanchao Su, Xinliang Wu
Date: 2021-09-29 00:00:00
Tasks: Node Classification, Representation Learning

In representation learning on the graph-structured data, many popular GNNs may fail to capture long-range dependencies, which leads to their performance degradation. Furthermore, this weakness will be magnified when the concerned graph is disassortative. To solve the above-mentioned issue, we propose a graph convolutional network with structure learning (GCN-SL), and furthermore, the proposed approach can be applied to node classification. The proposed GCN-SL contains two improvements: corresponding to edges and node features, respectively. Since the original adjacency matrix may provide misleading information for the aggregation process in GNNs, especially in the disassortative graph. We build a re-connected adjacency matrix by structure learning from the perspective of edges. The structure learning module aims to learn an optimized graph structure and corresponding feature representations. Specifically, the re-connected adjacency matrix is built by using a special data preprocessing technique and similarity learning, and can be optimized directly along with GCN-SL parameters. Through structure learning, GCN-SL can search reliable adjacent nodes from the entire graph for aggregation. In the aspect of node features, we propose an efficient-spectral-clustering (ESC) and an ESC with anchors (ESC-ANCH) algorithms. The two algorithms can efficiently aggregate feature representations from similar nodes, no matter how far away these similar nodes are from the target node. Both of the two improvements can help GCN-SL capture long-range dependencies, then make GCN-SL is capable of performing representation learning on both disassortative and assortative graphs. Experimental results on a wide range of benchmark datasets illustrate that the proposed GCN-SL outperforms the state-of-the-art GNN counterparts.
Adaptive Wavelet Transformer Network for 3D Shape Representation Learning
Authors: Hao Huang, Yi Fang
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: 3D Shape Classification, 3D Shape Representation, Representation Learning

We present a novel method for 3D shape representation learning using multi-scale wavelet decomposition. Distinct from previous works that either decompose 3D shapes into complimentary components at a single scale, or naively adopt up-/down-sampling to build hierarchies and treat all points or local regions equally, we decompose 3D shapes into sub-bands components at multiple scales and all scales form a decomposition tree in a principled manner rooted in multi-resolution wavelet analysis. Specifically, we propose Adaptive Wavelet Transformer Network (AWT-Net) that firstly generates approximation or detail wavelet coefficients per point, classifying each point into high or low sub-bands components, using lifting scheme at multiple scales recursively and hierarchically. Then, AWT-Net exploits Transformers that regard the features from different but complementary components as two holistic representations, and fuse them with the original shape features with different attentions. The wavelet coefficients can be learned without direct supervision on coefficients, and AWT-Net is fully differentiable and can be learned in an end-to-end fashion. Extensive experiments demonstrate that AWT-Net achieves state-of-the-arts or competitive performance on 3D shape classification and segmentation benchmarks.
Path-specific Causal Fair Prediction via Auxiliary Graph Structure Learning
Authors: Liuyi Yao, Yaliang Li, Bolin Ding, Jingren Zhou, Jinduo Liu, Mengdi Huai, Jing Gao
Date: 2021-09-29 00:00:00
Tasks: Fairness, Graph structure learning, Prediction

Algorithm fairness has become a trending topic, and it has a great impact on social welfare. Among different fairness definitions, path-specific causal fairness is a widely adopted one with great potentials, as it distinguishes the fair and unfair effects that the sensitive attributes exert on algorithm predictions. Existing methods based on path-specific causal fairness either require graph structure as the prior knowledge or have high complexity in the calculation of path-specific effect. To tackle these challenges, we propose a novel casual graph based fair prediction framework, which integrates graph structure learning into fair prediction to ensure that unfair pathways are excluded in the causal graph. Furthermore, we generalize the proposed framework to the scenarios where sensitive attributes can be non-root nodes and affected by other variables, which is commonly observed in real-world applications but hardly addressed by existing works. We provide theoretical analysis on the generalization bound for the proposed fair prediction method, and conduct a series of experiments on real-world datasets to demonstrate that the proposed framework can provide better prediction performance and algorithm fairness trade-off.