pwc-archive/papers-with-abstracts huggingface dataset:
Weakly Supervised Graph Clustering
Authors: Tian Bian, Tingyang Xu, Yu Rong, Wenbing Huang, Xi Xiao, Peilin Zhao, Junzhou Huang, Hong Cheng
Date: 2021-09-29 00:00:00
Tasks: Clustering, Graph Clustering, Graph Learning

Graph Clustering, which clusters the nodes of a graph given its collection of node features and edge connections in an unsupervised manner, has long been researched in graph learning and is essential in certain applications. While this task is common, more complex cases arise in practice—can we cluster nodes better with some graph-level side information or in a weakly supervised manner as, for example, identifying potential fraud users in a social network given additional labels of fraud communities. This triggers an interesting problem which we define as Weakly Supervised Graph Clustering (WSGC). In this paper, we firstly discuss the various possible settings of WSGC, formally. Upon such discussion, we investigate a particular task of weakly supervised graph clustering by making use of the graph labels and node features, with the assistance of a hierarchical graph that further characterizes the connections between different graphs. To address this task, we propose Gaussian Mixture Graph Convolutional Network (GMGCN), a simple yet effective framework for learning node representations under the supervision of graph labels guided by a proposed consensus loss and then inferring the category of each node via a Gaussian Mixture Layer (GML). Extensive experiments are conducted to test the rationality of the formulation of weakly supervised graph clustering. The experimental results show that, with the assistance of graph labels, the weakly supervised graph clustering method has a great improvement over the traditional graph clustering method.
Pixab-CAM: Attend Pixel, not Channel
Authors: Jaeeun Jang, Seokjun Kim, Hyeoncheol Kim
Date: 2021-09-29 00:00:00
Tasks: Adversarial Attack

To understand the internal behaviors of convolution neural networks (CNNs), many class activation mapping (CAM) based methods, which generate an explanation map by a linear combination of channels and corresponding weights, have been proposed. Previous CAM-based methods have tried to define a channel-wise weight that represents the importance of a channel for the target class. However, these methods have two common limitations. First, all pixels in the channel share a single scalar value. If the pixels are tied to a specific value, some of them are overestimated. Second, since the explanation map is the result of a linear combination of channels in the activation tensor, it is inevitably dependent on the activation tensor. To address these issues, we propose gradient-free Pixel-wise Ablation-CAM (Pixab-CAM), which utilizes pixel-wise weights rather than channel-wise weights to break the link between pixels in a channel. In addition, in order not to generate an explanation map dependent on the activation tensor, the explanation map is generated only with pixel-wise weights without linear combination with the activation tensor. In this paper, we also propose novel evaluation metrics to measure the quality of explanation maps using an adversarial attack. We demonstrate through experiments the qualitative and quantitative superiority of Pixab-CAM.
POI-Transformers: POI Entity Matching through POI Embeddings by Incorporating Semantic and Geographic Information
Authors: Jinbao Zhang, Changwang Zhang, Xiaojuan Liu, Xia Li, Weilin Liao, Penghua Liu, Yao Yao, Jihong Zhang
Date: 2021-09-29 00:00:00

Point of Interest (POI) data is crucial to location-based applications and various user-oriented services. However, three problems are existing in POI entity matching. First, traditional approaches to general entity matching are designed without geographic location information, which ignores the geographic features when performing POI entity matching. Second, existing POI matching methods for feature design are heavily dependent on the experts’ knowledge. Third, current deep learning-based entity matching approaches require a high computational complexity since all the potential POI entity pairs need input to the network. A general and robust POI embedding framework, the POI-Transformers, is initially proposed in this study to address these problems of POI entity matching. The POI-Transformers can generate semantically meaningful POI embeddings through aggregating the text attributes and geographic location, and minimize the inconsistency of a POI entity by measuring the distance between the newly generated POI embeddings. Moreover, the POI entities are matched by the similarity of POI embeddings instead of directly comparing the POI entities, which can greatly reduce the complexity of computation. The implementation of the POI-Transformers achieves a high F1 score of 95.8% on natural scenes data sets (from the Gaode Map and the Tencent Map) in POI entity matching and is comparable to the state-of-the-art (SOTA) entity matching methods of DeepER, DeepMatcher, and Ditto (in entity matching benchmark data set). Compared with the existing deep learning methods, our method reduces the effort for identifying one million pairs from about 20 hours to 228 seconds. These demonstrate that the proposed POI-Transformers framework significantly outstrips traditional methods both in accuracy and efficiency.
Grounding Language Representation with Visual Object Information via Cross Modal Pretraining
Authors: Cong-Duy T Nguyen, Anh Tuan Luu, Tho Quan
Date: 2021-09-29 00:00:00
Tasks: Grounded language learning, Object, Sentence

Previous studies of visual grounded language learning use a convolutional neural network (CNN) to extract features from the whole image for grounding with the sentence description. However, this approach has two main drawbacks: (i) the whole image usually contains more objects and backgrounds than the sentence itself; thus, matching them together will confuse the grounded model; (ii) CNN only extracts the features of the image but not the relationship between objects inside that, limiting the grounded model to learn complicated contexts. To overcome such shortcomings, we propose a novel object-level grounded language learning framework that empowers the language representation with visual object-grounded information. The framework is comprised of three main components: (i) ObjectGroundedBERT captures the visual-object relations and literary portrayals by cross-modal pretraining via a Text-grounding mechanism, (ii) Visual encoder represents a visual relation between objects and (iii) Cross-modal Transformer helps the Visual encoder and ObjectGroundedBERT learn the alignment and representation of image-text context. Experimental results show that our proposed framework consistently outperforms the baseline language models on various language tasks of GLUE and SQuAD datasets.
Large-Scale Adversarial Attacks on Graph Neural Networks via Graph Coarsening
Authors: Jianfu Zhang, Yan Hong, Liqing Zhang, Qibin Zhao
Date: 2021-09-29 00:00:00
Tasks: Adversarial Attack

Graph Neural Networks (GNNs) are fragile to adversarial attacks. However, existing state-of-the-art adversarial attack methods against GNNs are typically constrained by the graph's scale, failing to attack large graphs effectively. In this paper, we propose a novel attack method that attacks the graph in a divide-and-conquer manner to tackle large-scale adversarial attacks on GNNs. Specifically, the nodes are clustered based on node embeddings, coarsened graphs are constructed using the node clusters, and attacks are conducted on the coarsened graphs. Perturbations are selected starting with smaller coarsened graphs and progressing to larger detailed graphs while most of the irrelative nodes remain clustered, significantly reducing the complexity of generating adversarial graphs. Extensive empirical results show that the proposed method can greatly save the computational resources required to attack GNNs on large graphs while maintaining comparable performance on small graphs.
Benchmarking Graph Neural Networks on Dynamic Link Prediction
Authors: Joakim Skarding, Matthew Hellmich, Bogdan Gabrys, Katarzyna Musial-Gabrys
Date: 2021-09-29 00:00:00
Tasks: Benchmarking, Dynamic Link Prediction, Link Prediction, Prediction

Graph neural networks (GNNs) are rapidly becoming the dominant way to learn on graph-structured data.  Link prediction is a near-universal benchmark for new GNN models.  Many advanced models such as Dynamic graph neural networks(DGNNs)  specifically target dynamic link prediction.   However,  these models, particularly DGNNs, are rarely compared to each other or existing heuristics. Different works evaluate their models in different ways,  thus one cannot compare evaluation metrics directly. Motivated by this, we perform a comprehensive comparison study.  We compare link prediction heuristics, GNNs, discrete DGNNs, and continuous DGNNs on dynamic link prediction. We find that simple link prediction heuristics often perform better than GNNs and DGNNs, different sliding window sizes greatly affect performance,  and of all examined graph neural networks, that DGNNs consistently outperform static GNNs.
Can network pruning benefit deep learning under label noise?
Authors: Zheng He, Quanzhi Zhu, Zengchang Qin
Date: 2021-09-29 00:00:00
Tasks: Deep Learning, Network Pruning

Network pruning is a widely-used technique to reduce the computational cost of over-parameterized neural networks. Conventional wisdom also regards pruning as a way to improve generalization: by zeroing out parameters, pruning reduces model capacity and prevents overfitting. However, this wisdom is facing challenges in a line of recent studies, which show that over-parameterization actually helps generalization. In this work, we demonstrate the existence of a novel double descent phenomenon in sparse regimes, namely, in the presence of label noise, medium sparsity induced by pruning hurts model performance, while high sparsity benefits. Through extensive experiments on noisy versions of MNIST, CIFAR-10 and CIFAR-100, We show that proper pruning could consistently promise non-trivial robustness against label noise, which provides a new lens for studying network pruning. Further, we reassess some common beliefs concerning the generalization of sparse networks, and hypothesize it is the distance from initialization that is key to robustness rather than sharpness/flatness. Experimental results correlate with this hypothesis. Together, our study provides valuable insight on whether, when and why network pruning benefits deep learning under label noise.
Robust Learning with Adaptive Sample Credibility Modeling
Authors: Boshen Zhang, Yuxi Li, Yuanpeng Tu, Yabiao Wang, Yang Xiao, Cai Rong Zhao, Chengjie Wang
Date: 2021-09-29 00:00:00
Tasks: Denoising

Training deep neural network (DNN) with noisy labels is practically challenging since inaccurate labels severely degrade the generalization ability of DNN. Previous efforts tend to handle part or full data in a unified denoising flow to mitigate the noisy label problem, while they lack the consideration of intrinsic difference among difficulties of various noisy samples. In this paper, a novel and adaptive end-to-end robust learning method, called CREMA, is proposed. The insight behind is that the credibility of a training sample can be estimated by the joint distribution of its data-label pair, thus to roughly separate clean and noisy samples from original samples, which will be processed with different denoising process in a divide-and-conquer manner. For the clean set, we deliberately design a memory-based modulation scheme to dynamically adjust the contribution of each sample in terms of its historical credibility sequence during training, thus to alleviate the effect from potential hard noisy samples in clean set. Meanwhile, for those samples categorized into noisy set, we try to correct their labels in a selective manner to maximize data utilization and further boost performance. Extensive experiments on mainstream benchmarks, including synthetic (noisy versions of MNIST, CIFAR-10 and CIFAR-100) and real-world (Clothing1M and Animal-10N) noisy datasets demonstrate superiority of the proposed method.
On strong convergence of the two-tower model for recommender system
Authors: SHIRONG XU, Junhui Wang
Date: 2021-09-29 00:00:00
Tasks: Recommendation Systems, Vocal Bursts Valence Prediction

Recommender system is capable of predicting preferred items for a user by integrating information from similar users or items. A popular model in recommender system is the so-called two-tower model, which employs two deep neural networks to embed users and items into a low-dimensional space, and predicts ratings via the geometrical relationship of the embeddings of user and item in the embedded space. Even though it is popularly used for recommendations, its theoretical properties remain largely unknown. In this paper, we establish some asymptotic results of the two-tower model in terms of its strong convergence to the optimal recommender system, showing that it achieves a fast convergence rate depending on the intrinsic dimensions of inputs features. To the best of our knowledge, this is among the first attempts to establish the statistical guarantee of the two-tower model. Through numerical experiments, we also demonstrate that the two-tower model is capable of capturing the effects of users' and items' features on ratings, leading to higher prediction accuracy over its competitors in both simulated examples and a real application data set.
The Infinite Contextual Graph Markov Model
Authors: Daniele Castellana, Federico Errica, Davide Bacciu, Alessio Micheli
Date: 2021-09-29 00:00:00
Tasks: Graph Classification, model, Model Selection

The Contextual Graph Markov Model is a deep, unsupervised, and probabilistic model for graphs that is trained incrementally on a layer-by-layer basis. As with most Deep Graph Networks, an inherent limitation is the lack of an automatic mechanism to choose the size of each layer's latent representation. In this paper, we circumvent the problem by extending the Contextual Graph Markov Model with Hierarchical Dirichlet Processes. The resulting model for graphs can automatically adjust the complexity of each layer without the need to perform an extensive model selection. To improve the scalability of the method, we introduce a novel approximated inference procedure that better deals with larger graph topologies. The quality of the learned unsupervised representations is then evaluated across a set of eight graph classification tasks, showing competitive performances against end-to-end supervised methods. The analysis is complemented by studies on the importance of depth, hyper-parameters, and compression of the graph embeddings. We believe this to be an important step towards the theoretically grounded and automatic construction of deep probabilistic architectures for graphs.
Residual Contrastive Learning: Unsupervised Representation Learning from Residuals
Authors: Nanqing Dong, Matteo Maggioni, Yongxin Yang, Eduardo Pérez-Pellitero, Ales Leonardis, Steven McDonagh
Date: 2021-09-29 00:00:00
Tasks: Contrastive Learning, Image Reconstruction, Image Restoration, Representation Learning, Self-Supervised Learning

In the era of deep learning, supervised residual learning (ResL) has led to many breakthroughs in low-level vision such as image restoration and enhancement tasks. However, the question of how to formalize and take advantage of unsupervised ResL remains open. 
In this paper we consider visual signals with additive noise and propose to build a connection between ResL and self-supervised learning (SSL) via contrastive learning. We present residual contrastive learning (RCL), an unsupervised representation learning framework for downstream low-level vision tasks with noisy inputs. While supervised image reconstruction tasks aim to minimize the residual terms directly, RCL formulates an instance-wise discrimination pretext task by using the residuals as the discriminative feature. Empirical results on low-level vision tasks show that RCL is able to learn more robust and transferable representations in comparison to other SSL frameworks when ingesting noisy images, whilst retaining significantly reduced annotation costs over fully supervised alternatives.
A theoretically grounded characterization of feature representations
Authors: Bharath Hariharan, Cheng Perng Phoo
Date: 2021-09-29 00:00:00
Tasks: Few-Shot Learning, Self-Supervised Learning, Transfer Learning

A large body of work has explored how learned feature representations can be useful for a variety of downstream tasks. This is true even when the downstream tasks differ greatly from the actual objective used to (pre)train the feature representation. This observation underlies the success of, e.g., few-shot learning, transfer learning and self-supervised learning, among others. However, very little is understood about why such transfer is successful, and more importantly, how one should choose the pre-training task. As a first step towards this understanding, we ask: what makes a feature representation good for a target task? We present simple, intuitive measurements of the feature space that are good predictors of downstream task performance. We present theoretical results showing how these measurements can be used to bound the error of the downstream classifiers, and show empirically that these bounds correlate well with actual downstream performance. Finally, we show that our bounds are practically useful for choosing the right pre-trained representation for a target task.
Machine Learning Applications in Forecasting of COVID-19 Based on Patients' Individual Symptoms
Authors: Zhanyang Sun, Rui Ding, Xinyu Zhou
Date: 2021-09-29 00:00:00
Tasks: BIG-bench Machine Learning, Prediction

Predicting the COVID-19 outbreak has been studied by many researchers in recent years. Many
machine learning models have been used for the prediction of the transmission in a country or
region, but few studies aim to predict whether an individual has been infected by COVID-19.
However, due to the gravity of this global pandemic, prediction at an individual level is critical.
The objective of this paper is to predict if an individual has COVID-19 based on the symptoms
and features. The prediction results can help the government better allocate the medical resources
during this pandemic. Data of this study was taken on June 18th from the Israeli Ministry of
Health on COVID-19. The purpose of this study is to compare and analyze different models,
which are Support Vector Machine (SVM), Logistic Regression (LR), Naive Bayesian (NB),
Decision Tree (DT), Random Forest (RF) and Neural Network (NN).
Constituency Tree Representation for Argument Unit Recognition
Authors: Samuel Guilluy, Florian Méhats, Billal Chouli
Date: 2021-09-29 00:00:00
Tasks: Sentence

The extraction of arguments from sentences is usually studied by considering only the neighbourhood dependencies of words. Such a representation does not rely on the syntactic structure of the sentence and can lead to poor results especially in languages where grammatical categories are scattered in the sentence. In this paper, we investigate the advantages of using a constituency tree representation of sentences for argument discourse unit (ADU) prediction. We demonstrate that the constituency structure is more powerful than simple linear dependencies between neighbouring words in the sentence. Our work was organised as follows: First, we compare the maximum depth allowed for our constituency trees. This first step allows us to choose an optimal maximum depth. Secondly, we combine this structure with graph neural networks, which are very successful in neural network tasks. Finally, we evaluate the benefits of adding a conditional random field to model global dependencies between labels, given local dependency rules. We improve the current best models for argument unit recognition at token level and also present an explainability method to evaluate the suitability of our model architecture.
Generative Pseudo-Inverse Memory
Authors: Kha Pham, Hung Le, Man Ngo, Truyen Tran, Bao Ho, Svetha Venkatesh
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Denoising

We propose Generative Pseudo-Inverse Memory (GPM), a class of deep generative memory models that are fast to write in and read out. Memory operations are recast as seeking robust solutions of linear systems, which naturally lead to the use of matrix pseudo-inverses. The pseudo-inverses are iteratively approximated, with practical computation complexity of almost $O(1)$. We prove theoretically and verify empirically that our model can retrieve exactly what have been written to the memory under mild conditions. A key capability of GPM is iterative reading, during which the attractor dynamics towards fixed points are enabled, allowing the model to iteratively improve sample quality in denoising and generating. More impressively, GPM can store a large amount of data while maintaining key abilities of accurate retrieving of stored patterns, denoising of corrupted data and generating novel samples. Empirically we demonstrate the efficiency and versatility of GPM on a comprehensive suite of experiments involving binarized MNIST, binarized Omniglot, FashionMNIST, CIFAR10 & CIFAR100 and CelebA.
Learning to Solve Combinatorial Problems via Efficient Exploration
Authors: Thomas D Barrett, Christopher William Falke Parsonson, Alexandre Laterre
Date: 2021-09-29 00:00:00
Tasks: Efficient Exploration, Reinforcement Learning (RL)

From logistics to the natural sciences, combinatorial optimisation on graphs underpins numerous real-world applications.  Reinforcement learning (RL) has shown particular promise in this setting as it can adapt to specific problem structures and does not require pre-solved instances for these, often NP-hard, problems.  However, state-of-the-art (SOTA) approaches typically suffer from severe scalability issues, primarily due to their reliance on expensive graph neural networks (GNNs) at each decision step.  We introduce ECORD; a novel RL algorithm that alleviates this expense by restricting the GNN to a single pre-processing step, before entering a fast-acting exploratory phase directed by a recurrent unit.  Experimentally, we demonstrate that ECORD achieves a new SOTA for RL algorithms on the Maximum Cut problem, whilst also providing orders of magnitude improvement in speed and scalability.  Compared to the nearest competitor, ECORD reduces the optimality gap by up to 73% on 500 vertex graphs with a decreased wall-clock time.  Moreover, ECORD retains strong performance when generalising to larger graphs with up to 10000 vertices.