pwc-archive/papers-with-abstracts huggingface dataset:
Grassmann Graph Embedding
Authors: Bingxin Zhou, Xuebin Zheng, Yu Guang Wang, Ming Li, Junbin Gao
Date: 2021-03-08 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Dimensionality Reduction, Graph Embedding

Geometric deep learning that employs the geometric and topological features of data has attracted increasing attention in deep neural networks. Learning the intrinsic structure property of data is a crucial step for dimensionality reduction and effective feature extraction. This paper develops Grassmann graph embedding, which combines graph convolutions to capture the main components within graphs' hidden representations. Each set of featured graph nodes is mapped to a point on a Grassmann matrix manifold through Singular Value Decomposition, which is then embedded into a symmetric matrix space that approximates denoised second-order feature information. The view of treating nodes as a set could inspire many potential applications. In particular, we propose Grassmann (global graph) pooling that can connect with any graph convolution for graph neural networks. The Grassmann pooling achieves state-of-the-art performance on a variety of graph prediction benchmarks.
Chemistry-informed Macromolecule Graph Representation for Similarity Computation and Supervised Learning
Authors: Somesh Mohapatra, Joyce An, Rafael Gomez-Bombarelli
Date: 2021-03-08 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Decision Making, Diversity

Macromolecules are large, complex molecules composed of covalently bonded monomer units, existing in different stereochemical configurations and topologies. As a result of such chemical diversity, representing, comparing, and learning over macromolecules emerge as critical challenges. To address this, we developed a macromolecule graph representation, with monomers and bonds as nodes and edges, respectively. We captured the inherent chemistry of the macromolecule by using molecular fingerprints for node and edge attributes. For the first time, we demonstrated computation of chemical similarity between 2 macromolecules of varying chemistry and topology, using exact graph edit distances and graph kernels. We also trained graph neural networks for a variety of glycan classification tasks, achieving state-of-the-art results. Our work has two-fold implications – it provides a general framework for representation, comparison, and learning of macromolecules; and enables quantitative chemistry-informed decision-making and iterative design in the macromolecular chemical space.
SKILLBERT: “SKILLING” THE BERT TO CLASSIFY SKILLS!
Authors: Anonymous
Date: 2021-03-08 00:00:00
Tasks: Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION

In the age of digital recruitment, job posts can attract a large number of applications, and screening them manually can become a very tedious task. We propose a BERT-based model, SkillBERT, the embeddings of which are used as features for classifying skills into groups referred to as “competency groups”. A competency group is a group of similar skills and it is used as matching criteria (instead of matching on skills) for finding the overlap of skills between the candidates and the jobs. This proxy match takes advantage of the BERT’s capability of deriving meaning from the structure of competency groups present in the skill dataset. The problem that we are trying to solve is a multi-label classification problem, as a single skill can belong to multiple competency groups. To solve multi-label competency group classification using binary classifier, we have paired each skill with each competency group and tried to predict the probability of that skill belonging to that particular competency group. SkillBERT, which is trained from scratch on the skills present in job requisitions, is shown to be better performing than the pre-trained BERT (Devlin et al., 2019) and the Word2Vec (Mikolov et al., 2013). We have also explored K-means clustering (Lloyd, 1982) and spectral clustering (Chung, 1997) on SkillBERT embeddings to generate cluster-based features. Both algorithms provide similar performance benefits. Last, we have experimented with different classification models like Random Forest (Breiman, 2001), XGBoost (Chen & Guestrin, 2016), and a deep learning algorithm Bi-LSTM (Schuster & Paliwal, 1997; Hochreiter & Schmidhuber, 1997) for the tagging of competency groups to skill. We did not observe a significant performance difference among the algorithms, although XGBoost and Bi-LSTM perform slightly better than Random Forest. The features created using SkillBERT are most predictive in the classification task, which demonstrates that the SkillBERT is able to capture the information about the skills’ ontology from the data. We have made the source code, the trained models and the dataset (Electronic Recruitment Records, referred to as ERRs)1 of our experiments publicly available. ERRs are stored in the form of tables in our recruitment database.
Functorial Manifold Learning
Authors: Anonymous
Date: 2021-03-08 00:00:00
Tasks: Clustering

We adapt previous research on category theory and topological unsupervised learning to develop a functorial perspective on manifold learning. We first characterize manifold learning algorithms as functors that map pseudometric spaces to optimization objectives and factor through hierarchical clustering functors.  We then use this characterization to prove refinement bounds on manifold learning loss functions and construct a hierarchy of manifold learning algorithms based on their invariants. We express several popular manifold learning algorithms as functors at different levels of this hierarchy and present bounds on how closely the embeddings these algorithms produce from noisy data approximate the embeddings they would learn from noiseless data.
Learning Irreducible Representations of Noncommutative Lie Groups
Authors: Anonymous
Date: 2021-03-08 00:00:00
Tasks: Object, Object Tracking

Recent work has constructed neural networks that are equivariant to continuous symmetry groups such as 2D and 3D rotations. This is accomplished using explicit group representations to derive the equivariant kernels and nonlinearities. We present two contributions motivated by frontier applications of equivariance beyond rotations and translations. First, we relax the requirement for explicit Lie group representations, presenting a novel algorithm that finds irreducible representations of noncommutative Lie groups given only the structure constants of the associated Lie algebra. Second, we demonstrate that Lorentz-equivariance is a useful prior for object-tracking tasks and construct the first object-tracking model equivariant to the Poincaré group.
Persistent Homology and Graphs Representation Learning
Authors: Anonymous
Date: 2021-03-08 00:00:00
Tasks: Representation Learning

This article aims to study the topological invariant properties encoded in node graph representational embeddings by utilizing tools available in persistent homology. Specifically, given a node embedding representation algorithm, we consider the case when these embeddings are real-valued. By viewing these embeddings as scalar functions on a domain of interest, we can utilize the tools available in persistent homology to study the topological information encoded in these representations. Our construction effectively defines a unique persistence-based graph descriptor, on both the graph and node levels, for every node representation algorithm. To demonstrate the effectiveness of the proposed method, we study the topological descriptors induced by DeepWalk, Node2Vec and Diff2Vec.
Self-supervised representation learning on manifolds
Authors: Eric O Korman
Date: 2021-03-08 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Representation Learning

We explore the use of a topological manifold, represented as a collection of charts, as the target space of neural network based representation learning tasks. This is achieved by a simple adjustment to the output of an encoder's network architecture plus the addition of a maximal mean discrepancy based loss function for regularization. Most algorithms in representation learning are easily adaptable to our framework and we demonstrate its effectiveness by adjusting SimCLR to have a manifold encoding space. Our experiments show that we obtain a substantial performance boost over the baseline for low dimensional encodings. Code for reproducing experiments is provided at https://github.com/ekorman/neurve.
Unsupervised Geometric Disentanglement via CFAN-VAE
Authors: Norman Joseph Tatro, Stefan C Schonsheck, Rongjie Lai
Date: 2021-03-08 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Disentanglement, Pose Transfer

Geometric disentanglement, the separation of latent codes for intrinsic (i.e. identity) and extrinsic (i.e. pose) geometry, is a prominent task for generative models of non-Euclidean data such as 3D deformable models. It provides greater interpretability of the latent space, and leads to more control in generation. This work introduces a mesh feature, the conformal factor and normal feature (CFAN), for use in mesh convolutional autoencoders. We further propose CFAN-VAE, a novel architecture that disentangles identity and pose using the CFAN feature. Requiring no label information on the identity or pose during training, CFAN-VAE achieves geometric disentanglement in an unsupervised way. Our comprehensive experiments, including reconstruction, interpolation, generation, and identity/pose transfer, demonstrate CFAN-VAE achieves state-of-the-art performance on unsupervised geometric disentanglement.
Quantifying barley morphology using the Euler characteristic transform
Authors: Anonymous
Date: 2021-03-08 00:00:00
Tasks: Diversity, Topological Data Analysis

Shape plays a fundamental role in biology. Throughout history, the constant observation and documentation of the shape of multiple organisms has been key to further biological understanding of organism and tissue behavior, function, and origins. To extract, compare, and analyze such data diversity we must wield a quantifiable, robust, and concise method. We thus turn to Topological Data Analysis (TDA) and the Euler Characteristic Transform (ECT). As a study case, we quantify the morphology of X-ray CT scans of barley spikes and seeds using both traditional and topological shape descriptors based. We then successfully train a support vector machine to distinguish and classify 28 different parental genotypes of barley based solely on the 3D shape of their grains. We observe that combining both traditional and topological descriptors produces considerably better classification results compared to the use of exclusively traditional descriptors. This improvement suggests that TDA is thus a powerful complement to describe comprehensively a multitude of shape nuances which are otherwise not picked up by traditional morphometrics methods.
Bootstrapped Representation Learning on Graphs
Authors: Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Remi Munos, Petar Veličković, Michal Valko
Date: 2021-03-08 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Contrastive Learning, Linear evaluation, Representation Learning, Self-Supervised Learning

Current state-of-the-art self-supervised learning methods for graph neural networks are based on contrastive learning. As such, they heavily depend on the construction of augmentations and negative examples. Increasing the number of negative pairs improves performance, thereby requiring quadratic computation and memory cost to achieve peak performance. Inspired by BYOL, a recently introduced method for self-supervised learning that does not require negative pairs, we present Bootstrapped Graph Latents, BGRL, a self-supervised graph representation method that gets rid of this potentially quadratic bottleneck. BGRL outperforms or matches the previous unsupervised state-of-the-art results on several established benchmarks. Moreover, it enables the effective usage of graph attentional (GAT) encoders, allowing us to further improve the state of the art, in particular achieving 70.49% Micro-F1 on the PPI dataset using the linear evaluation protocol.
Topological Features Anchor Visual Shape Representation
Authors: Anonymous
Date: 2021-03-08 00:00:00

Computer vision is outstanding on fixed datasets, but lacks the
effortless generality and flexibility of human perception. 
As a step toward understanding this flexibility, we
show that there are features -- critical contours -- readily detectable in images that have
counterparts -- extremal contours -- on 3D shapes. Both are 
part of a topological structure, Morse-Smale complex, and together they provide an invariant
linking image to shape structure. An open neighborhood around these contours
is related to human perception of shape features, such as
bumps, which suggests that training for these features could provide a
generalization benefit in algorithm development.
GRAPH-BASED FEATURE REPRESENTATION FOR MULTI-CLASS CLASSIFICATIONS USING THE JEFFRIES-MATUSITA DISTANCE
Authors: Anonymous
Date: 2021-03-08 00:00:00
Tasks: feature selection

Visualization of the feature space has gained attention in recent years, especially with the growing
importance of explainability in AI. When processing high-dimensional datasets, a common preprocessing
step is feature selection. Filter-based feature selection algorithms are not tailored to
a specific classification method, but rather rank the relevance of each feature with respect to the
target and the task. The Jeffries-Matusita (JM) distance is a measure for separability between two
distributions that has been used for filter based feature selection. It calculates for each feature the
separability strength between pairs of classes. In this work, a low-dimensional representation of
the JM measures for the feature space is formed by diffusion map. Diffusion maps organizes the
features by their class separation abilities. Moreover, feature elimination is performed based on
the distribution of the points in the low-dimensional space. Experimental results are provided for 3
public datasets and compared with known filter-based feature selection techniques.
Attend to connect: end-to-end brain functional connectivity estimation
Authors: Usman Mahmood, Zening Fu, Vince Calhoun, Sergey Plis
Date: 2021-03-08 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Connectivity Estimation, Functional Connectivity

Functional connectivity (FC) studies have demonstrated the benefits of investigating the brain and its disorders through the undirected weighted graph of fMRI correlation matrix. Most of the work with the FC, however, depends on the way the connectivity is computed and further depends on the manual post-hoc analysis of the FC matrices. In this work, we propose a deep learning architecture (BrainGNN) that learns the connectivity structure while learning to classify subjects. It simultaneously trains a graphical neural network on this graph and learns to select a sparse subset of brain regions important to the prediction task. We demonstrate the model's state-of-the-art classification performance on a schizophrenia fMRI dataset and show how introspection leads to disorder-relevant findings. The graphs learned by the model exhibit strong class discrimination, and the identified sparse subset of relevant regions is consistent with the schizophrenia literature.
On the discriminability of samples in the Hamming space of binarized ReLU activations
Authors: Anonymous
Date: 2021-03-08 00:00:00

Binarized ReLU activations are considered as a metric space equipped with the Hamming distance. While for two layer ReLU networks with random Gaussian weights it can be shown theoretically
that  local metric properties are approximately preserved, we experimentally study the discrimination capability in this Hamming space for deeper ReLU networks and look also at the non-local behavior.
It turns out that the discrimination capability is approximately preserved as expected, but showing small saturation effects that differ from standard metrics based on full activation information.
These effects are explained based on the fact that the binarized activation states induce a tessellation of polyhedral cells in the input space.
Missing Glow Phenomenon: learning disentangled representation of missing data
Authors: Anonymous
Date: 2021-03-08 00:00:00

Learning from incomplete data has been recognized as one of the fundamental challenges in deep learning. There are many more or less complicated methods for processing missing data by neural networks in the literature. 
In this paper, we show that flow-based generative models can work directly on images with missing data to produce full images without missing parts. We name this behavior Missing Glow Phenomenon. We present experiments that document such behaviors and propose theoretical justification of such phenomena.
Scaling Up Graph Homomorphism Features with Efficient Data Structures
Authors: Paul Beaujean, Florian Sikora, Florian Yger
Date: 2021-03-08 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Graph Classification

Typical datasets used in graph classification tasks only contain a few thousand graphs which rarely exceed hundreds of nodes. Graph homomorphism densities are permutation-invariant features that can be directly computed from graph data, and their approximation scales naturally to large graphs. We propose the use of efficient data structures for approximate set membership in the context of a sampling algorithm for graph homomorphism density which enables the use of large-scale datasets containing larger graphs. To validate our findings, we compare this method with existing approaches used for graph homomorphism features in synthetic experiments.
Equivariant Manifold Flows
Authors: Anonymous
Date: 2021-03-08 00:00:00

Tractably modelling distributions over manifolds has long been an important goal in the natural sciences. Recent work has focused on developing general machine learning models to learn such distributions. Though these are significant advances, many problems require the learned distribution to obey symmetries inherent to the manifold. Up until now, the theory of invariant learning on manifolds was underdeveloped—models were unable to incorporate these symmetries while learning. In this paper, we lay the theoretical foundations for learning symmetry invariant distributions on arbitrary manifolds via equivariant manifold flows. We demonstrate the efficacy of our approach in the context of quantum field theory in learning gauge invariant densities over SU(n).
Exploring epithelial-cell calcium signaling with geometric and topological data analysis
Authors: Feng Gao, Jessica Moore, Bastian Rieck, Valentina Greco, Smita Krishnaswamy
Date: 2021-03-08 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Topological Data Analysis

Spatio-temporal calcium imaging is widely used to study neuronal activity. However the function of calcium signaling in epithelial cells is not well understood. We thus explore the underlying fluorescence patterns using a combination of data geometry and topology. We model the epithelial tissue as a graph and use a variant of geometric scattering transform (a multi-level wavelet transform on a graph) to describe the signaling at each time point. We then embed these discriptors using the manifold learning method PHATE in order to  capture the entire time-trajectory of calcium signaling. Finally, we use persistent homology computed on the PHATE embedding to quantitatively characterize the trajectory.  We demonstrate that scattering coefficients can effectively learn time point embeddings while PHATE can represent relationships between time points. Persistent homology provides a way to quantify differences between signaling dynamics as evidenced by the differences in dynamics between wild type cells and cells that are stalled in a particular cell cycle phase.
Simplicial Complex Representation Learning
Authors: Anonymous
Date: 2021-03-08 00:00:00
Tasks: Representation Learning

Simplicial complexes form an important class of topological spaces that are frequently used in many application areas such as computer-aided design, computer graphics, and simulation. Representation learning on graphs, which are just 1-d simplicial complexes, has witnessed a great attention and success recently. Due to the additional complexity higher dimensional simplicial hold, there has not been enough effort to extend representation learning to these objects especially when it comes to entire-simplicial complex representation learning. In this work, we propose a method for simplicial  complex-level representation learning that embeds a simplicial complex to a universal embedding space in a way that complex-to-complex proximity is preserved. Our method utilizes a simplex-level embedding induced by a pre-trained simplicial autoencoder to learn an entire simplicial complex representation. To the best of our knowledge, this work presents the first method for learning simplicial complex-level representation.
Simplicial Regularization
Authors: Jose Gallego-Posada, Patrick Forré
Date: 2021-03-08 00:00:00
Proceeding: ICLR Workshop GTRL 2021 5
Tasks: Data Augmentation, Dimensionality Reduction

Inspired by the fuzzy topological representation of a dataset employed in UMAP (McInnes et al., 2018), we propose a regularization principle for supervised learning based on the preservation of the simplicial complex structure of the data. We analyze the behavior of our proposal in contrast with the mixup (Zhang et al., 2018) framework on dimensionality reduction and classification tasks. Our experiments show how simplicial regularization induces more appropriate learning biases and alleviates some of the shortcomings of state-of-the art methods for regularization based on data augmentation.
Learning where to learn
Authors: Dominic Zhao, Nicolas Zucchet, Joao Sacramento, Johannes von Oswald
Date: 2021-03-13 00:00:00
Proceeding: ICLR Workshop Learning_to_Learn 2021 5
Tasks: Meta-Learning, Sparse Learning

Finding neural network weights that generalize well from small datasets is difficult. A  promising approach is to  (meta-)learn a  weight initialization from  a collection of tasks,  such that a small number of weight changes results in low generalization error. We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn. We find that patterned sparsity emerges from this process. Lower-level features tend to be frozen, while weights close to the output remain plastic. This selective sparsity enables running longer sequences of weight updates with-out overfitting, resulting in better generalization in the miniImageNet benchmark. Our findings shed light on an ongoing debate on whether meta-learning can discover adaptable features, and suggest that sparse learning can outperform simpler feature reuse schemes.
Compositionality as Learning Bias in Generative RNNs solves the Omniglot Challenge
Authors: Sarah Fabi, Sebastian Otte, Martin V. Butz
Date: 2021-03-13 00:00:00
Proceeding: ICLR Workshop Learning_to_Learn 2021 5

One aspect of learning to learn concerns the development of compositional knowledge structures that can be flexibly recombined in a semantically meaningful manner to analogically solve related problems. We focus on learning to learn one-shot/few-shot generation and classification tasks of handwritten character trajectories, as described in the Omniglot challenge. We show that solving the challenge becomes possible, by suitably fostering a generative LSTM network to develop well-structured, compositional encodings, which can be quickly reassembled into new, unseen but related character trajectories. This is a major improvement compared to the original approach, which explicitly provided character components. We believe that the development of similarly compressed, compositional structures may also be highly useful to address related learning to learn challenges in other dynamic processing, prediction, and control domains.