pwc-archive/papers-with-abstracts huggingface dataset:
Stationary Diffusion State Neural Estimation for Multiview Clustering
Authors: Chenghua Liu, Zhuolin Liao, Yixuan Ma, Kun Zhan
Date: 2021-12-02 00:00:00
Tasks: Clustering, Graph Neural Network, Multiview Clustering, Multiview Learning

Although many graph-based clustering methods attempt to model the stationary diffusion state in their objectives, their performance limits to using a predefined graph. We argue that the estimation of the stationary diffusion state can be achieved by gradient descent over neural networks. We specifically design the Stationary Diffusion State Neural Estimation (SDSNE) to exploit multiview structural graph information for co-supervised learning. We explore how to design a graph neural network specially for unsupervised multiview learning and integrate multiple graphs into a unified consensus graph by a shared self-attentional module. The view-shared self-attentional module utilizes the graph structure to learn a view-consistent global graph. Meanwhile, instead of using auto-encoder in most unsupervised learning graph neural networks, SDSNE uses a co-supervised strategy with structure information to supervise the model learning. The co-supervised strategy as the loss function guides SDSNE in achieving the stationary state. With the help of the loss and the self-attentional module, we learn to obtain a graph in which nodes in each connected component fully connect by the same weight. Experiments on several multiview datasets demonstrate effectiveness of SDSNE in terms of six clustering evaluation metrics.
Optimal regularizations for data generation with probabilistic graphical models
Authors: Arnaud Fanthomme, F Rizzato, S Cocco, R Monasson
Date: 2021-12-02 00:00:00
Tasks: valid

Understanding the role of regularization is a central question in Statistical Inference. Empirically, well-chosen regularization schemes often dramatically improve the quality of the inferred models by avoiding overfitting of the training data. We consider here the particular case of L 2 and L 1 regularizations in the Maximum A Posteriori (MAP) inference of generative pairwise graphical models. Based on analytical calculations on Gaussian multivariate distributions and numerical experiments on Gaussian and Potts models we study the likelihoods of the training, test, and 'generated data' (with the inferred models) sets as functions of the regularization strengths. We show in particular that, at its maximum, the test likelihood and the 'generated' likelihood, which quantifies the quality of the generated samples, have remarkably close values. The optimal value for the regularization strength is found to be approximately equal to the inverse sum of the squared couplings incoming on sites on the underlying network of interactions. Our results seem largely independent of the structure of the true underlying interactions that generated the data, of the regularization scheme considered, and are valid when small fluctuations of the posterior distribution around the MAP estimator are taken into account. Connections with empirical works on protein models learned from homologous sequences are discussed.
Borrowing from Similar Code: A Deep Learning NLP-Based Approach for Log Statement Automation
Authors: Sina Gholamian, Paul A. S. Ward
Date: 2021-12-02 00:00:00
Tasks: Clone Detection, Management, Prediction

Software developers embed logging statements inside the source code as an imperative duty in modern software development as log files are necessary for tracking down runtime system issues and troubleshooting system management tasks. However, the current logging process is mostly manual, and thus, proper placement and content of logging statements remain as challenges. To overcome these challenges, methods that aim to automate log placement and predict its content, i.e., 'where and what to log', are of high interest. Thus, we focus on predicting the location (i.e., where) and description (i.e., what) for log statements by utilizing source code clones and natural language processing (NLP), as these approaches provide additional context and advantage for log prediction. Specifically, we guide our research with three research questions (RQs): (RQ1) how similar code snippets, i.e., code clones, can be leveraged for log statements prediction? (RQ2) how the approach can be extended to automate log statements' descriptions? and (RQ3) how effective the proposed methods are for log location and description prediction? To pursue our RQs, we perform an experimental study on seven open-source Java projects. We introduce an updated and improved log-aware code-clone detection method to predict the location of logging statements (RQ1). Then, we incorporate natural language processing (NLP) and deep learning methods to automate the log statements' description prediction (RQ2). Our analysis shows that our hybrid NLP and code-clone detection approach (NLP CC'd) outperforms conventional clone detectors in finding log statement locations on average by 15.60% and achieves 40.86% higher performance on BLEU and ROUGE scores for predicting the description of logging statements when compared to prior research (RQ3).
Hierarchical Learning to Solve Partial Differential Equations Using Physics-Informed Neural Networks
Authors: Jihun Han, Yoonsang Lee
Date: 2021-12-02 00:00:00

The neural network-based approach to solving partial differential equations has attracted considerable attention due to its simplicity and flexibility in representing the solution of the partial differential equation. In training a neural network, the network learns global features corresponding to low-frequency components while high-frequency components are approximated at a much slower rate. For a class of equations in which the solution contains a wide range of scales, the network training process can suffer from slow convergence and low accuracy due to its inability to capture the high-frequency components. In this work, we propose a hierarchical approach to improve the convergence rate and accuracy of the neural network solution to partial differential equations. The proposed method comprises multi-training levels in which a newly introduced neural network is guided to learn the residual of the previous level approximation. By the nature of neural networks' training process, the high-level correction is inclined to capture the high-frequency components. We validate the efficiency and robustness of the proposed hierarchical approach through a suite of linear and nonlinear partial differential equations.
Youla-REN: Learning Nonlinear Feedback Policies with Robust Stability Guarantees
Authors: Ruigang Wang, Ian R. Manchester
Date: 2021-12-02 00:00:00

This paper presents a parameterization of nonlinear controllers for uncertain systems building on a recently developed neural network architecture, called the recurrent equilibrium network (REN), and a nonlinear version of the Youla parameterization. The proposed framework has "built-in" guarantees of stability, i.e., all policies in the search space result in a contracting (globally exponentially stable) closed-loop system. Thus, it requires very mild assumptions on the choice of cost function and the stability property can be generalized to unseen data. Another useful feature of this approach is that policies are parameterized directly without any constraints, which simplifies learning by a broad range of policy-learning methods based on unconstrained optimization (e.g. stochastic gradient descent). We illustrate the proposed approach with a variety of simulation examples.
Local Citation Recommendation with Hierarchical-Attention Text Encoder and SciBERT-based Reranking
Authors: Nianlong Gu, Yingqiang Gao, Richard H. R. Hahnloser
Date: 2021-12-02 00:00:00
Tasks: Citation Recommendation, Reranking

The goal of local citation recommendation is to recommend a missing reference from the local citation context and optionally also from the global context. To balance the tradeoff between speed and accuracy of citation recommendation in the context of a large-scale paper database, a viable approach is to first prefetch a limited number of relevant documents using efficient ranking methods and then to perform a fine-grained reranking using more sophisticated models. In that vein, BM25 has been found to be a tough-to-beat approach to prefetching, which is why recent work has focused mainly on the reranking step. Even so, we explore prefetching with nearest neighbor search among text embeddings constructed by a hierarchical attention network. When coupled with a SciBERT reranker fine-tuned on local citation recommendation tasks, our hierarchical Attention encoder (HAtten) achieves high prefetch recall for a given number of candidates to be reranked. Consequently, our reranker requires fewer prefetch candidates to rerank, yet still achieves state-of-the-art performance on various local citation recommendation datasets such as ACL-200, FullTextPeerRead, RefSeer, and arXiv.
Multi-task Self-distillation for Graph-based Semi-Supervised Learning
Authors: Yating Ren, Junzhong Ji, Lingfeng Niu, Minglong Lei
Date: 2021-12-02 00:00:00
Tasks: Node Classification

Graph convolutional networks have made great progress in graph-based semi-supervised learning. Existing methods mainly assume that nodes connected by graph edges are prone to have similar attributes and labels, so that the features smoothed by local graph structures can reveal the class similarities. However, there often exist mismatches between graph structures and labels in many real-world scenarios, where the structures may propagate misleading features or labels that eventually affect the model performance. In this paper, we propose a multi-task self-distillation framework that injects self-supervised learning and self-distillation into graph convolutional networks to separately address the mismatch problem from the structure side and the label side. First, we formulate a self-supervision pipeline based on pre-text tasks to capture different levels of similarities in graphs. The feature extraction process is encouraged to capture more complex proximity by jointly optimizing the pre-text task and the target task. Consequently, the local feature aggregations are improved from the structure side. Second, self-distillation uses soft labels of the model itself as additional supervision, which has similar effects as label smoothing. The knowledge from the classification pipeline and the self-supervision pipeline is collectively distilled to improve the generalization ability of the model from the label side. Experiment results show that the proposed method obtains remarkable performance gains under several classic graph convolutional architectures.
Forex Trading Volatility Prediction using Neural Network Models
Authors: Shujian Liao, Jian Chen, Hao Ni
Date: 2021-12-02 00:00:00
Tasks: Deep Learning, Prediction

In this paper, we investigate the problem of predicting the future volatility of Forex currency pairs using the deep learning techniques. We show step-by-step how to construct the deep-learning network by the guidance of the empirical patterns of the intra-day volatility. The numerical results show that the multiscale Long Short-Term Memory (LSTM) model with the input of multi-currency pairs consistently achieves the state-of-the-art accuracy compared with both the conventional baselines, i.e. autoregressive and GARCH model, and the other deep learning models.
FNR: A Similarity and Transformer-Based Approachto Detect Multi-Modal FakeNews in Social Media
Authors: Faeze Ghorbanpour, Maryam Ramezani, Mohammad A. Fazli, Hamid R. Rabiee
Date: 2021-12-02 00:00:00

The availability and interactive nature of social media have made them the primary source of news around the globe. The popularity of social media tempts criminals to pursue their immoral intentions by producing and disseminating fake news using seductive text and misleading images. Therefore, verifying social media news and spotting fakes is crucial. This work aims to analyze multi-modal features from texts and images in social media for detecting fake news. We propose a Fake News Revealer (FNR) method that utilizes transform learning to extract contextual and semantic features and contrastive loss to determine the similarity between image and text. We applied FNR on two real social media datasets. The results show the proposed method achieves higher accuracies in detecting fake news compared to the previous works.
Near-Optimal Lower Bounds For Convex Optimization For All Orders of Smoothness
Authors: Ankit Garg, Robin Kothari, Praneeth Netrapalli, Suhail Sherif
Date: 2021-12-02 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: All

We study the complexity of optimizing highly smooth convex functions. For a positive integer $p$, we want to find an $\epsilon$-approximate minimum of a convex function $f$, given oracle access to the function and its first $p$ derivatives, assuming that the $p$th derivative of $f$ is Lipschitz. Recently, three independent research groups (Jiang et al., PLMR 2019; Gasnikov et al., PLMR 2019; Bubeck et al., PLMR 2019) developed a new algorithm that solves this problem with $\tilde{O}(1/\epsilon^{\frac{2}{3p+1}})$ oracle calls for constant $p$. This is known to be optimal (up to log factors) for deterministic algorithms, but known lower bounds for randomized algorithms do not match this bound. We prove a new lower bound that matches this bound (up to log factors), and holds not only for randomized algorithms, but also for quantum algorithms.
Contrastive Adaptive Propagation Graph Neural Networks for Efficient Graph Learning
Authors: Jun Hu, Shengsheng Qian, Quan Fang, Changsheng Xu
Date: 2021-12-02 00:00:00
Tasks: Graph Learning, Self-Supervised Learning

Graph Neural Networks (GNNs) have achieved great success in processing graph data by extracting and propagating structure-aware features. Existing GNN research designs various propagation schemes to guide the aggregation of neighbor information. Recently the field has advanced from local propagation schemes that focus on local neighbors towards extended propagation schemes that can directly deal with extended neighbors consisting of both local and high-order neighbors. Despite the impressive performance, existing approaches are still insufficient to build an efficient and learnable extended propagation scheme that can adaptively adjust the influence of local and high-order neighbors. This paper proposes an efficient yet effective end-to-end framework, namely Contrastive Adaptive Propagation Graph Neural Networks (CAPGNN), to address these issues by combining Personalized PageRank and attention techniques. CAPGNN models the learnable extended propagation scheme with a polynomial of a sparse local affinity matrix, where the polynomial relies on Personalized PageRank to provide superior initial coefficients. In order to adaptively adjust the influence of both local and high-order neighbors, a coefficient-attention model is introduced to learn to adjust the coefficients of the polynomial. In addition, we leverage self-supervised learning techniques and design a negative-free entropy-aware contrastive loss to explicitly take advantage of unlabeled data for training. We implement CAPGNN as two different versions named CAPGCN and CAPGAT, which use static and dynamic sparse local affinity matrices, respectively. Experiments on graph benchmark datasets suggest that CAPGNN can consistently outperform or match state-of-the-art baselines. The source code is publicly available at https://github.com/hujunxianligong/CAPGNN.
Who will dropout from university? Academic risk prediction based on interpretable machine learning
Authors: Shudong Yang
Date: 2021-12-02 00:00:00
Tasks: BIG-bench Machine Learning, Binary Classification, Interpretable Machine Learning, Prediction

In the institutional research mode, in order to explore which characteristics are the best indicators for predicting academic risk from the student behavior data sets that have high-dimensional, unbalanced classified small sample, it transforms the academic risk prediction of college students into a binary classification task. It predicts academic risk based on the LightGBM model and the interpretable machine learning method of Shapley value. The simulation results show that from the global perspective of the prediction model, characteristics such as the quality of academic partners, the seating position in classroom, the dormitory study atmosphere, the English scores of the college entrance examination, the quantity of academic partners, the addiction level of video games, the mobility of academic partners, and the degree of truancy are the best 8 predictors for academic risk. It is contrary to intuition that characteristics such as living in campus or not, work-study, lipstick addiction, student leader or not, lover amount, and smoking have little correlation with university academic risk in this experiment. From the local perspective of the sample, the factors affecting academic risk vary from person to person. It can perform personalized interpretable analysis through Shapley values, which cannot be done by traditional mathematical statistical prediction models. The academic contributions of this research are mainly in two aspects: First, the learning interaction networks is proposed for the first time, so that social behavior can be used to compensate for the one-sided individual behavior and improve the performance of academic risk prediction. Second, the introduction of Shapley value calculation makes machine learning that lacks a clear reasoning process visualized, and provides intuitive decision support for education managers.
Data-Driven Interaction Analysis of Line Failure Cascading in Power Grid Networks
Authors: Abdorasoul Ghasemi, Holger Kantz
Date: 2021-12-02 00:00:00
Tasks: Time Series, Time Series Analysis

We use machine learning tools to model the line interaction of failure cascading in power grid networks. We first collect data sets of simulated trajectories of possible consecutive line failure following an initial random failure and considering actual constraints in a model power network until the system settles at a steady state. We use weighted $l_1$-regularized logistic regression-based models to find static and dynamic models that capture pairwise and latent higher-order lines' failure interactions using pairwise statistical data. The static model captures the failures' interactions near the steady states of the network, and the dynamic model captures the failure unfolding in a time series of consecutive network states. We test models over independent trajectories of failure unfolding in the network to evaluate their failure predictive power. We observe asymmetric, strongly positive, and negative interactions between different lines' states in the network. We use the static interaction model to estimate the distribution of cascade size and identify groups of lines that tend to fail together, and compare against the data. The dynamic interaction model successfully predicts the network state for long-lasting failure propagation trajectories after an initial failure.
ShuttleNet: Position-aware Fusion of Rally Progress and Player Styles for Stroke Forecasting in Badminton
Authors: Wei-Yao Wang, Hong-Han Shuai, Kai-Shiang Chang, Wen-Chih Peng
Date: 2021-12-02 00:00:00
Tasks: Decoder, Position

The increasing demand for analyzing the insights in sports has stimulated a line of productive studies from a variety of perspectives, e.g., health state monitoring, outcome prediction. In this paper, we focus on objectively judging what and where to return strokes, which is still unexplored in turn-based sports. By formulating stroke forecasting as a sequence prediction task, existing works can tackle the problem but fail to model information based on the characteristics of badminton. To address these limitations, we propose a novel Position-aware Fusion of Rally Progress and Player Styles framework (ShuttleNet) that incorporates rally progress and information of the players by two modified encoder-decoder extractors. Moreover, we design a fusion network to integrate rally contexts and contexts of the players by conditioning on information dependency and different positions. Extensive experiments on the badminton dataset demonstrate that ShuttleNet significantly outperforms the state-of-the-art methods and also empirically validates the feasibility of each component in ShuttleNet. On top of that, we provide an analysis scenario for the stroke forecasting problem.
Graph4Rec: A Universal Toolkit with Graph Neural Networks for Recommender Systems
Authors: Weibin Li, Mingkai He, Zhengjie Huang, Xianming Wang, Shikun Feng, Weiyue Su, Yu Sun
Date: 2021-12-02 00:00:00
Tasks: graph construction, Graph Neural Network, Graph Representation Learning, Recommendation Systems, Representation Learning

In recent years, owing to the outstanding performance in graph representation learning, graph neural network (GNN) techniques have gained considerable interests in many real-world scenarios, such as recommender systems and social networks. In recommender systems, the main challenge is to learn the effective user/item representations from their interactions. However, many recent publications using GNNs for recommender systems cannot be directly compared, due to their difference on datasets and evaluation metrics. Furthermore, many of them only provide a demo to conduct experiments on small datasets, which is far away to be applied in real-world recommender systems. To address this problem, we introduce Graph4Rec, a universal toolkit that unifies the paradigm to train GNN models into the following parts: graphs input, random walk generation, ego graphs generation, pairs generation and GNNs selection. From this training pipeline, one can easily establish his own GNN model with a few configurations. Besides, we develop a large-scale graph engine and a parameter server to support distributed GNN training. We conduct a systematic and comprehensive experiment to compare the performance of different GNN models on several scenarios in different scale. Extensive experiments are demonstrated to identify the key components of GNNs. We also try to figure out how the sparse and dense parameters affect the performance of GNNs. Finally, we investigate methods including negative sampling, ego graph construction order, and warm start strategy to find a more effective and efficient GNNs practice on recommender systems. Our toolkit is based on PGL https://github.com/PaddlePaddle/PGL and the code is opened source in https://github.com/PaddlePaddle/PGL/tree/main/apps/Graph4Rec.
Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation
Authors: Yeonsung Jung, Hajin Shim, June Yong Yang, Eunho Yang
Date: 2021-12-02 00:00:00
Tasks: Contrastive Learning, Translation

Deep neural networks (DNNs), despite their impressive ability to generalize over-capacity networks, often rely heavily on malignant bias as shortcuts instead of task-related information for discriminative tasks. To address this problem, recent studies utilize auxiliary information related to the bias, which is rarely obtainable in practice, or sift through a handful of bias-free samples for debiasing. However, the success of these methods is not always guaranteed due to the unfulfilled presumptions. In this paper, we propose a novel method, Contrastive Debiasing via Generative Bias-transformation (CDvG), which works without explicit bias labels or bias-free samples. Motivated by our observation that not only discriminative models but also image translation models tend to focus on the malignant bias, CDvG employs an image translation model to transform one bias mode into another while preserving the task-relevant information. Additionally, the bias-transformed views are set against each other through contrastive learning to learn bias-invariant representations. Our method demonstrates superior performance compared to prior approaches, especially when bias-free samples are scarce or absent. Furthermore, CDvG can be integrated with the methods that focus on bias-free samples in a plug-and-play manner for additional enhancements, as demonstrated by diverse experimental results.