pwc-archive/papers-with-abstracts huggingface dataset:
Multi-level Graph Matching Networks for Deep and Robust Graph Similarity Learning
Authors: Xiang Ling, Lingfei Wu, Saizhuo Wang, Tengfei Ma, Fangli Xu, Alex X. Liu, Chunming Wu, Shouling Ji
Date: 2021-01-01 00:00:00
Tasks: Graph Classification, Graph Matching, Graph Neural Network, Graph Similarity

While the celebrated graph neural networks yield effective representations for individual nodes of a graph, there has been relatively less success in extending to graph similarity learning. Recent works have considered either global-level graph-graph interactions or low-level node-node interactions, ignoring the rich cross-level interactions (e.g., between nodes of a graph and the other whole graph). In this paper, we propose a Multi-level Graph Matching Network (MGMN) for computing the graph similarity between any pair of graph-structured objects in an end-to-end fashion. The proposed MGMN model consists of a node-graph matching network for effectively learning cross-level interactions between nodes of a graph and the other whole graph, and a siamese graph neural network to learn global-level interactions between two graphs. Furthermore, to bridge the gap of the lack of standard graph similarity learning benchmark, we have created and collected a set of datasets for both graph-graph classification and regression tasks with different sizes in order to evaluate the robustness of models. Our comprehensive experiments demonstrate that MGMN consistently outperforms state-of-the-art baselines on these graph similarity learning benchmarks.
Language Controls More Than Top-Down Attention: Modulating Bottom-Up Visual Processing with Referring Expressions
Authors: Ozan Arkan Can, Ilker Kesen, Deniz Yuret
Date: 2021-01-01 00:00:00
Tasks: Referring Expression

How to best integrate linguistic and perceptual processing in multimodal tasks is an important open problem. In this work we argue that the common technique of using language to direct visual attention over high-level visual features may not be optimal. Using language throughout the bottom-up visual pathway, going from pixels to high-level features, may be necessary. Our experiments on several English referring expression datasets show significant improvements when language is used to control the filters for bottom-up visual processing in addition to top-down attention.
Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity
Authors: Anonymous
Date: 2021-01-01 00:00:00
Tasks: Position, Video Reconstruction

Reconstructing complex and dynamic visual perception from brain activity remains a major challenge in machine learning applications to neuroscience. Here we present a new method for reconstructing naturalistic images and videos from very large single-participant functional magnetic resonance data that leverages the recent success of image-to-image transformation networks. This is achieved by exploiting spatial information obtained from retinotopic mappings across the visual system. More specifically, we first determine what position each voxel in a particular region of interest would represent on the visual field based on its corresponding receptive field location. Then, the 2D image representation of the brain activity on the visual field is passed to a fully convolutional image-to-image network trained to recover the original stimuli using VGG feature loss with an adversarial regularizer. In our experiments we show that our method offers a significant improvement over existing techniques.
Automatic Music Production Using Generative Adversarial Networks
Authors: Giorgio Barnabò, Giovanni Trappolini, Lorenzo Lastilla, Cesare Campagnano, Angela Fan, Fabio Petroni, Fabrizio Silvestri
Date: 2021-01-01 00:00:00
Tasks: Image-to-Image Translation, Music Generation

When talking about computer-based music generation, two are the main threads of research: the construction of $\textit{autonomous music-making systems}$, and the design of $\textit{computer-based environments to assist musicians}$. Despite consistent demand from music producers and artists, however, little effort has been done in the field of automatic music arrangement in the audio domain. In this work, we propose a novel framework for $\textit{automatic music arrangement from raw audio in the frequency domain}$. Using several songs converted into Mel-spectrograms -- a two-dimensional time-frequency representation of audio signals -- we were able to automatically generate original arrangements for both bass and voice lines. Treating music pieces as images (Mel-spectrograms) allowed us to reformulate our problem as an $\textit{unpaired image-to-image translation}$ problem, and to tackle it with CycleGAN, a well-established framework. Moreover, the choice to deploy raw audio and Mel spectrograms enabled us to more effectively model long-range dependencies, to better represent how humans perceive music, and to potentially draw sounds for new arrangements from the vast collection of music recordings accumulated in the last century. Our approach was tested on two different downstream tasks: given a bass line creating credible and on-time drums, and given an acapella song arranging it to a full song. In absence of an objective way of evaluating the output of music generative systems, we also defined a possible metric for the proposed task, partially based on human (and expert) judgment.  To the best of our knowledge, we are the first to address the music arrangement task in the audio domain, to treat music pieces as images, and to propose a quantitative approach to evaluate the model results.
Global Node Attentions via Adaptive Spectral Filters
Authors: Shouheng Li, Dongwoo Kim, Qing Wang
Date: 2021-01-01 00:00:00
Tasks: Node Classification

Graph neural networks (GNNs) have been extensively studied for prediction tasks on graphs. Most GNNs assume local homophily, i.e., strong similarities in local neighborhoods. This assumption limits the generalizability of GNNs, which has been demonstrated by recent work on disassortative graphs with weak local homophily. In this paper, we argue that GNN's feature aggregation scheme can be made flexible and adaptive to data without the assumption of local homophily. To demonstrate, we propose a GNN model with a global self-attention mechanism defined using learnable spectral filters, which can attend to any nodes, regardless of distance. We evaluated the proposed model on node classification tasks over six benchmark datasets. The proposed model has been shown to generalize well to both assortative and disassortative graphs. Further, it outperforms all state-of-the-art baselines on disassortative graphs and performs comparably with them on assortative graphs.
Addressing the Topological Defects of Disentanglement
Authors: Diane Bouchacourt, Mark Ibrahim, Stephane Deny
Date: 2021-01-01 00:00:00
Tasks: Disentanglement

A core challenge in Machine Learning is to disentangle natural factors of variation in data (e.g. object shape vs pose).  A popular approach to disentanglement consists in learning to map each of these factors to distinct subspaces of a model's latent representation. However, this approach has shown limited empirical success to date. Here, we show that this approach to disentanglement introduces topological defects (i.e. discontinuities in the encoder) for a broad family of transformations acting on images ---encompassing simple affine transformations such as rotations and translations.  Moreover, motivated by classical results from group representation theory, we propose an alternative, more flexible approach to disentanglement which relies on distributed equivariant operators, potentially acting on the entire latent space. We theoretically and empirically demonstrate the effectiveness of our approach to disentangle affine transformations. Our work lays a theoretical foundation for the recent success of a new generation of models using distributed operators for disentanglement.
Machine Reading Comprehension with Enhanced Linguistic Verifiers
Authors: Xianchao Wu
Date: 2021-01-01 00:00:00
Tasks: Machine Reading Comprehension, Reading Comprehension, TriviaQA

We propose two linguistic verifiers for span-extraction style machine reading comprehension to respectively tackle two challenges: how to evaluate the syntactic completeness of predicted answers and how to utilize the rich context of long documents. Our first verifier rewrites a question through replacing its interrogatives by the predicted answer phrases and then builds a cross-attention scorer between the rewritten question and the segment, so that the answer candidates are scored in a \emph{position-sensitive} context. Our second verifier builds a hierarchical attention network to represent segments in a passage where neighbour segments in long passages are \emph{recurrently connected} and can contribute to current segment-question pair's inference for answerablility classification and boundary determination. We then combine these two verifiers together into a pipeline and apply it to SQuAD2.0, NewsQA and TriviaQA benchmark sets. Our pipeline achieves significantly better improvements of both exact matching and F1 scores than state-of-the-art baselines.
Learning Contextual Perturbation Budgets for Training Robust Neural Networks
Authors: Jing Xu, Zhouxing Shi, huan zhang, JinFeng Yi, Cho-Jui Hsieh, LiWei Wang
Date: 2021-01-01 00:00:00

Existing methods for training robust neural networks generally try to make the models uniformly robust on all input dimensions. However, the importance of different input dimensions is different and should be context-dependent. In this paper, we propose a novel framework to train robust models with non-uniform perturbation budgets on different input dimensions. We insert a perturbation budget generator to generate perturbation budgets, and incorporate it into certified defense. We evaluate our method on MNIST and CIFAR-10 datasets and show that we can achieve lower errors under the same "robustness volume", compared to methods using uniform perturbation budgets. We also demonstrate that the perturbation budget generator can produce semantically-meaningful budgets, which implies that the generator can capture contextual information and the sensitivity of different features in a given image.
Deep Coherent Exploration For Continuous Control
Authors: Yijie Zhang, Herke van Hoof
Date: 2021-01-01 00:00:00
Tasks: continuous-control, Continuous Control, Reinforcement Learning (RL)

In policy search methods for reinforcement learning (RL), exploration is often performed by injecting noise either in action space at each step independently or in parameter space over each full trajectory. In prior work, it has been shown that with linear policies, a more balanced trade-off between these two exploration strategies is beneficial. However, that method did not scale to policies using deep neural networks. In this paper, we introduce Deep Coherent Exploration, a general and scalable exploration framework for deep RL algorithms on continuous control, that generalizes step-based and trajectory-based exploration. This framework models the last layer parameters of the policy network as latent variables and uses a recursive inference step within the policy update to handle these latent variables in a scalable manner. We find that Deep Coherent Exploration improves the speed and stability of learning of A2C, PPO, and SAC on several continuous control tasks.
Median DC for Sign Recovery: Privacy can be Achieved by Deterministic Algorithms
Authors: Jiyuan Tu, Weidong Liu, Xiaojun Mao
Date: 2021-01-01 00:00:00
Tasks: Common Sense Reasoning, Privacy Preserving, regression

Privacy-preserving data analysis becomes prevailing in recent years. It is a common sense in privacy literature that strict differential privacy can only be obtained by imposing additional randomness in the algorithm. In this paper, we study the problem of private sign recovery for sparse mean estimation and sparse linear regression in a distributed setup. By taking a coordinate-wise median among the reported local sign vectors, which can be referred to as a median divide-and-conquer (Med-DC) approach, we can recover the signs of the true parameter with a provable consistency guarantee. Moreover, without adding any extra randomness to the algorithm, our Med-DC method can protect data privacy with high probability. Simulation studies are conducted to demonstrate the effectiveness of our proposed method.
Pseudo Label-Guided Multi Task Learning for Scene Understanding
Authors: Sunkyung Kim, Hyesong Choi, Dongbo Min
Date: 2021-01-01 00:00:00
Tasks: Depth Estimation, Monocular Depth Estimation, Multi-Task Learning, Pseudo Label, Scene Understanding, Segmentation, Semantic Segmentation, Stereo Matching

Multi-task learning (MTL) for scene understanding has been actively studied by exploiting correlation of multiple tasks. This work focuses on improving the performance of the MTL network that infers depth and semantic segmentation maps from a single image. Specifically, we propose a novel MTL architecture, called Pseudo-MTL, that introduces pseudo labels for joint learning of monocular depth estimation and semantic segmentation tasks. The pseudo ground truth depth maps, generated from pretrained stereo matching methods, are leveraged to supervise the monocular depth estimation. More importantly, the pseudo depth labels serve to impose a cross-view consistency on the estimated monocular depth and segmentation maps of two views. This enables for mitigating the mismatch problem incurred by inconsistent prediction results across two views. A thorough ablation study validates that the cross-view consistency leads to a substantial performance gain by ensuring inference-view invariance for the two tasks.
Stochastic Proximal Point Algorithm for Large-scale Nonconvex Optimization: Convergence, Implementation, and Application to Neural Networks
Authors: Aysegul Bumin, Kejun Huang
Date: 2021-01-01 00:00:00

We revisit the stochastic proximal point algorithm (SPPA) for large-scale nonconvex optimization problems. SPPA has been shown to converge faster and more stable than the celebrated stochastic gradient descent (SGD) algorithm, and its many variations, for convex problems. However, the per-iteration update of SPPA is defined abstractly and has long been considered expensive. In this paper, we show that efficient implementation of SPPA can be achieved. If the problem is a nonlinear least squares, each iteration of SPPA can be efficiently implemented by Gauss-Newton; with some linear algebra trick the resulting complexity is in the same order of SGD. For more generic problems, SPPA can still be implemented with L-BFGS or accelerated gradient with high efficiency. Another contribution of this work is the convergence of SPPA to a stationary point in expectation for nonconvex problems. The result is encouraging that it admits more flexible choices of the step sizes under similar assumptions. The proposed algorithm is elaborated for both regression and classification problems using different neural network structures. Real data experiments showcase its effectiveness in terms of convergence and accuracy compared to SGD and its variants.
A Chaos Theory Approach to Understand Neural Network Optimization
Authors: Michele Sasdelli, Thalaiyasingam Ajanthan, Tat-Jun Chin, Gustavo Carneiro
Date: 2021-01-01 00:00:00
Tasks: Second-order methods

Despite the complicated structure of modern deep neural network architectures, they are still optimized with algorithms based on Stochastic Gradient Descent (SGD). However, the reason behind the effectiveness of SGD is not well understood, making its study an active research area. In this paper, we formulate deep neural network optimization as a dynamical system and show that the rigorous theory developed to study chaotic systems can be useful to understand SGD and its variants. In particular, we first observe that the inverse of the instability timescale of SGD optimization, represented by the largest Lyapunov exponent, corresponds to the most negative eigenvalue of the Hessian of the loss. This observation enables the introduction of an efficient method to estimate the largest eigenvalue of the Hessian. Then, we empirically show that for a large range of learning rates, SGD traverses the loss landscape across regions with largest eigenvalue of the Hessian similar to the inverse of the learning rate. This explains why effective learning rates can be found to be within a large range of values and shows that SGD implicitly uses the largest eigenvalue of the Hessian while traversing the loss landscape. This sheds some light on the effectiveness of SGD over more sophisticated second-order methods. We also propose a quasi-Newton method that dynamically estimates an optimal learning rate for the optimization of deep learning models. We demonstrate that our observations and methods are robust across different architectures and loss functions on CIFAR-10 dataset.
Log representation as an interface for log processing applications
Authors: Mohammad Amin Sadeghi, Shameem Parambath, Ji Lucas, Youssef Meguebli, Maguette Toure, Fawaz Al Qahtani, Ting Yu, Sanjay Chawla
Date: 2021-01-01 00:00:00

Log files are files that record events, messages, or transactions. Logs are rich containers of data because they can store a sequence of structured textual and numerical data. Many sequential forms of data including natural languages and temporal signals can be represented as logs.

We propose to represent logs at a few levels of abstraction including field level, log level, and log sequence level. The representation for each level can be computed from the previous level. These representations are in vector format and serve as interfaces to downstream applications. We use a version of transformer networks to encode numerical information as well as textual information that is suitable for log embedding. We show how a number of log processing applications can be readily solved with our representation.
GL-Disen: Global-Local disentanglement for unsupervised learning of graph-level representations
Authors: Thilini Cooray, Ngai-Man Cheung, Wei Lu
Date: 2021-01-01 00:00:00
Tasks: Disentanglement, Graph Representation Learning, Molecular Property Prediction, Property Prediction, Representation Learning

Graph-level representation learning plays a crucial role in a variety of tasks such as molecular property prediction and community analysis. Currently, several models based on mutual information maximization have shown strong performance on the task of unsupervised graph representation learning. In this paper, instead, we consider a disentanglement approach to learn graph-level representations in the unsupervised setting. Our work is the first to study disentanglement learning for graph-level representations. Our key observation is that the formation of many real-world graphs is a complex process with global and  local generative factors. We hypothesize that disentangled representations which capture these global and local generative factors into independent latent units can be highly beneficial. Specifically, for graph-level representation learning, our disentanglement approach can alleviate distraction due to local variations of individual nodes or individual local neighbourhoods. We propose a VAE based learning algorithm to disentangle the global graph-level information, which is common across the entire graph, and local patch-level information, which varies across individual patches (the local subgraphs centered around the nodes). Through extensive experiments and analysis, we show that our method achieves the state-of-the-art performance on the task of unsupervised graph representation learning.
Hard Masking for Explaining Graph Neural Networks
Authors: Thorben Funke, Megha Khosla, Avishek Anand
Date: 2021-01-01 00:00:00
Tasks: Data Compression, Decision Making, Node Classification, Prediction

Graph Neural Networks (GNNs) are a flexible and powerful family of models that build nodes' representations on irregular graph-structured data. This paper focuses on explaining or interpreting the rationale underlying a given prediction of already trained graph neural networks for the node classification task. Existing approaches for interpreting GNNs try to find subsets of important features and nodes by learning a continuous mask. We alternately formulate interpretability inspired from data compression, where we consider an explanation as a compressed form of the original input representation. Our objective is to find discrete masks that are arguably more interpretable while minimizing the expected deviation from the underlying model's prediction. We empirically show that our explanations are both more predictive and sparse. Additionally, we find that multiple diverse explanations are possible, which sufficiently explain a prediction. Finally, we analyze the explanations to find the effect of network homophily on the decision-making process of GNNs.
Single Layers of Attention Suffice to Predict Protein Contacts
Authors: Nick Bhattacharya, Neil Thomas, Roshan Rao, Justas Daupras, Peter K Koo, David Baker, Yun S. Song, Sergey Ovchinnikov
Date: 2021-01-01 00:00:00
Proceeding: ICLR Workshop EBM 2021 5
Tasks: valid

The established approach to protein contact prediction frames the task as one of graph selection, extracting contacts by estimating the parameters of a Potts model. Another approach has recently appeared which leverages large pretrained Transformers, producing contacts by combining attention maps from various heads. In this work, we provide evidence that these approaches are not as different as they initially seem by providing a theoretical connection between attention and Potts models. To do so, we introduce a simplified attention model called \textit{factored attention}. On the one hand, factored attention is a direct simplification of multihead scaled dot-product attention in the Transformer. On the other hand, factored attention defines a valid Pairwise Markov Random Field and includes Potts models as a sparse special case. Examining factored attention allows us to explore the relative merits of each model class when learning contacts from an aligned protein family. We go on to empirically assess the performance of factored attention by training on a wide range of alignments of individual protein families. We further compare to a large pretrained Transformer trained on a corpus of unaligned protein sequences. We find that a single layer of attention is comparable to state-of-the-art Potts model at contact prediction.  Taken together, these results provide motivation for training Transformers on large protein datasets.
Trust, but verify: model-based exploration in sparse reward environments
Authors: Konrad Czechowski, Tomasz Odrzygóźdź, Michał Izworski, Marek Zbysiński, Łukasz Kuciński, Piotr Miłoś
Date: 2021-01-01 00:00:00
Proceeding: NeurIPS Workshop LMCA 2020 12
Tasks: Model-based Reinforcement Learning, reinforcement-learning, Reinforcement Learning (RL)

We propose $\textit{trust-but-verify}$ (TBV) mechanism, a new method which uses model uncertainty estimates to guide  exploration. The mechanism augments graph search planning algorithms by the capacity to deal with learned model's imperfections. We identify certain type of frequent model errors, which we dub $\textit{false loops}$, and which are particularly dangerous for graph search algorithms in discrete environments. These errors impose falsely pessimistic expectations and thus hinder exploration. We confirm this experimentally and show that TBV can effectively alleviate them. TBV combined with MCTS or Best First Search forms an effective model-based reinforcement learning solution, which is able to robustly solve sparse reward problems.