pwc-archive/papers-with-abstracts huggingface dataset:
Language Model Evaluation in Open-ended Text Generation
Authors: An Nguyen
Date: 2021-08-08 00:00:00
Tasks: Attribute, Diversity, Language Model Evaluation, Language Modeling, Language Modelling, model, Text Generation

Although current state-of-the-art language models have achieved impressive results in numerous natural language processing tasks, still they could not solve the problem of producing repetitive, dull and sometimes inconsistent text in open-ended text generation. Studies often attribute this problem to the maximum likelihood training objective, and propose alternative approaches by using stochastic decoding methods or altering the training objective. However, there is still a lack of consistent evaluation metrics to directly compare the efficacy of these solutions. In this work, we study different evaluation metrics that have been proposed to evaluate quality, diversity and consistency of machine-generated text. From there, we propose a practical pipeline to evaluate language models in open-ended generation task, and research on how to improve the model's performance in all dimensions by leveraging different auxiliary training objectives.
Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations
Authors: Georg A. Gottwald, Sebastian Reich
Date: 2021-08-08 00:00:00

We present a supervised learning method to learn the propagator map of a dynamical system from partial and noisy observations. In our computationally cheap and easy-to-implement framework a neural network consisting of random feature maps is trained sequentially by incoming observations within a data assimilation procedure. By employing Takens' embedding theorem, the network is trained on delay coordinates. We show that the combination of random feature maps and data assimilation, called RAFDA, outperforms standard random feature maps for which the dynamics is learned using batch data.
Recurrent Graph Neural Networks for Rumor Detection in Online Forums
Authors: Di Huang, Jacob Bartel, John Palowitch
Date: 2021-08-08 00:00:00
Tasks: Articles, Graph Neural Network

The widespread adoption of online social networks in daily life has created a pressing need for effectively classifying user-generated content. This work presents techniques for classifying linked content spread on forum websites -- specifically, links to news articles or blogs -- using user interaction signals alone. Importantly, online forums such as Reddit do not have a user-generated social graph, which is assumed in social network behavioral-based classification settings. Using Reddit as a case-study, we show how to obtain a derived social graph, and use this graph, Reddit post sequences, and comment trees as inputs to a Recurrent Graph Neural Network (R-GNN) encoder. We train the R-GNN on news link categorization and rumor detection, showing superior results to recent baselines. Our code is made publicly available at https://github.com/google-research/social_cascades.
The Effect of Training Parameters and Mechanisms on Decentralized Federated Learning based on MNIST Dataset
Authors: Zhuofan Zhang, Mi Zhou, Kaicheng Niu, Chaouki Abdallah
Date: 2021-08-07 00:00:00
Tasks: Federated Learning

Federated Learning is an algorithm suited for training models on decentralized data, but the requirement of a central "server" node is a bottleneck. In this document, we first introduce the notion of Decentralized Federated Learning (DFL). We then perform various experiments on different setups, such as changing model aggregation frequency, switching from independent and identically distributed (IID) dataset partitioning to non-IID partitioning with partial global sharing, using different optimization methods across clients, and breaking models into segments with partial sharing. All experiments are run on the MNIST handwritten digits dataset. We observe that those altered training procedures are generally robust, albeit non-optimal. We also observe failures in training when the variance between model weights is too large. The open-source experiment code is accessible through GitHub\footnote{Code was uploaded at \url{https://github.com/zhzhang2018/DecentralizedFL}}.
Kinematics clustering enables head impact subtyping for better traumatic brain injury prediction
Authors: Xianghao Zhan, Yiheng Li, Yuzhe Liu, Nicholas J. Cecchi, Olivier Gevaert, Michael M. Zeineh, Gerald A. Grant, David B. Camarillo
Date: 2021-08-07 00:00:00
Tasks: Car Racing, Clustering, Injury Prediction, regression

Traumatic brain injury can be caused by various types of head impacts. However, due to different kinematic characteristics, many brain injury risk estimation models are not generalizable across the variety of impacts that humans may sustain. The current definitions of head impact subtypes are based on impact sources (e.g., football, traffic accident), which may not reflect the intrinsic kinematic similarities of impacts across the impact sources. To investigate the potential new definitions of impact subtypes based on kinematics, 3,161 head impacts from various sources including simulation, college football, mixed martial arts, and car racing were collected. We applied the K-means clustering to cluster the impacts on 16 standardized temporal features from head rotation kinematics. Then, we developed subtype-specific ridge regression models for cumulative strain damage (using the threshold of 15%), which significantly improved the estimation accuracy compared with the baseline method which mixed impacts from different sources and developed one model (R^2 from 0.7 to 0.9). To investigate the effect of kinematic features, we presented the top three critical features (maximum resultant angular acceleration, maximum angular acceleration along the z-axis, maximum linear acceleration along the y-axis) based on regression accuracy and used logistic regression to find the critical points for each feature that partitioned the subtypes. This study enables researchers to define head impact subtypes in a data-driven manner, which leads to more generalizable brain injury risk estimation.
Approximate Last Iterate Convergence in Overparameterized GANs
Authors: Elbert Du
Date: 2021-08-07 00:00:00

In this work, we showed that the Implicit Update and Predictive Methods dynamics introduced in prior work satisfy last iterate convergence to a neighborhood around the optimum in overparameterized GANs, where the size of the neighborhood shrinks with the width of the neural network. This is in contrast to prior results, which only guaranteed average iterate convergence.
Clustering Algorithms to Analyze the Road Traffic Crashes
Authors: Mahnaz Rafia Islam, Israt Jahan Jenny, Moniruzzaman Nayon, Md. Rajibul Islam, Md Amiruzzaman, M. Abdullah-Al-Wadud
Date: 2021-08-07 00:00:00
Tasks: Clustering

Selecting an appropriate clustering method as well as an optimal number of clusters in road accident data is at times confusing and difficult. This paper analyzes shortcomings of different existing techniques applied to cluster accident-prone areas and recommends using Density-Based Spatial Clustering of Applications with Noise (DBSCAN) and Ordering Points To Identify the Clustering Structure (OPTICS) to overcome them. Comparative performance analysis based on real-life data on the recorded cases of road accidents in North Carolina also show more effectiveness and efficiency achieved by these algorithms.
An empirical assessment of deep learning approaches to task-oriented dialog management
Authors: Lukáš Matějů, David Griol, Zoraida Callejas, José Manuel Molina, Araceli Sanchis
Date: 2021-08-07 00:00:00
Tasks: Management, speech-recognition, Speech Recognition

Deep learning is providing very positive results in areas related to conversational interfaces, such as speech recognition, but its potential benefit for dialog management has still not been fully studied. In this paper, we perform an assessment of different configurations for deep-learned dialog management with three dialog corpora from different application domains and varying in size, dimensionality and possible system responses. Our results have allowed us to identify several aspects that can have an impact on accuracy, including the approaches used for feature extraction, input representation, context consideration and the hyper-parameters of the deep neural networks employed.
A k-mer Based Approach for SARS-CoV-2 Variant Identification
Authors: Sarwan Ali, Bikram Sahoo, Naimat Ullah, Alexander Zelikovskiy, Murray Patterson, Imdadullah Khan
Date: 2021-08-07 00:00:00

With the rapid spread of the novel coronavirus (COVID-19) across the globe and its continuous mutation, it is of pivotal importance to design a system to identify different known (and unknown) variants of SARS-CoV-2. Identifying particular variants helps to understand and model their spread patterns, design effective mitigation strategies, and prevent future outbreaks. It also plays a crucial role in studying the efficacy of known vaccines against each variant and modeling the likelihood of breakthrough infections. It is well known that the spike protein contains most of the information/variation pertaining to coronavirus variants. In this paper, we use spike sequences to classify different variants of the coronavirus in humans. We show that preserving the order of the amino acids helps the underlying classifiers to achieve better performance. We also show that we can train our model to outperform the baseline algorithms using only a small number of training samples ($1\%$ of the data). Finally, we show the importance of the different amino acids which play a key role in identifying variants and how they coincide with those reported by the USA's Centers for Disease Control and Prevention (CDC).
Self-learning sparse PCA for multimode process monitoring
Authors: Jingxin Zhang, Donghua Zhou, Maoyin Chen
Date: 2021-08-07 00:00:00
Tasks: Self-Learning

This paper proposes a novel sparse principal component analysis algorithm with self-learning ability for successive modes, where synaptic intelligence is employed to measure the importance of variables and a regularization term is added to preserve the learned knowledge of previous modes. Different from traditional multimode monitoring methods, the monitoring model is updated based on the current model and new data when a new mode arrives, thus delivering prominent performance for sequential modes. Besides, the computation and storage resources are saved in the long run, because it is not necessary to retrain the model from scratch frequently and store data from previous modes. More importantly, the model furnishes excellent interpretability owing to the sparsity of parameters. Finally, a numerical case and a practical pulverizing system are adopted to illustrate the effectiveness of the proposed algorithm.
A Machine Learning Tool to Determine State of Mind and Emotion
Authors: Rodrigo S. Jamisola Jr
Date: 2021-08-07 00:00:00
Tasks: BIG-bench Machine Learning

This paper investigates the possibility of creating a machine learning tool that automatically determines the state of mind and emotion of an individual through a questionnaire, without the aid of a human expert. The state of mind and emotion is defined in this work as pertaining to preference, feelings, or opinion that is not based on logic or reason. It is the case when a person gives out an answer to start by saying, "I feel...". The tool is designed to mimic the expertise of a psychologist and is built without any formal knowledge of psychology. The idea is to build the expertise by purely computational methods through thousands of questions collected from users. It is aimed towards possibly diagnosing substance addiction, alcoholism, sexual attraction, HIV status, degree of commitment, activity inclination, etc. First, the paper presents the related literature and classifies them according to data gathering methods. Another classification is created according to preference, emotion, grouping, and rules to achieve a deeper interpretation and better understanding of the state of mind and emotion. Second, the proposed tool is developed using an online addiction questionnaire with 10 questions and 292 respondents. In addition, an initial investigation on the dimension of addiction is presented through the built machine learning model. Machine learning methods, namely, artificial neural network (ANN) and support vector machine (SVM), are used to determine a true or false or degree of state of a respondent.
Secure Neuroimaging Analysis using Federated Learning with Homomorphic Encryption
Authors: Dimitris Stripelis, Hamza Saleem, Tanmay Ghai, Nikhil Dhinagar, Umang Gupta, Chrysovalantis Anastasiou, Greg Ver Steeg, Srivatsan Ravi, Muhammad Naveed, Paul M. Thompson, Jose Luis Ambite
Date: 2021-08-07 00:00:00
Tasks: Benchmarking, Federated Learning

Federated learning (FL) enables distributed computation of machine learning models over various disparate, remote data sources, without requiring to transfer any individual data to a centralized location. This results in an improved generalizability of models and efficient scaling of computation as more sources and larger datasets are added to the federation. Nevertheless, recent membership attacks show that private or sensitive personal data can sometimes be leaked or inferred when model parameters or summary statistics are shared with a central site, requiring improved security solutions. In this work, we propose a framework for secure FL using fully-homomorphic encryption (FHE). Specifically, we use the CKKS construction, an approximate, floating point compatible scheme that benefits from ciphertext packing and rescaling. In our evaluation on large-scale brain MRI datasets, we use our proposed secure FL framework to train a deep learning model to predict a person's age from distributed MRI scans, a common benchmarking task, and demonstrate that there is no degradation in the learning performance between the encrypted and non-encrypted federated models.
Unsupervised learning of anomalous diffusion data
Authors: Gorka Muñoz-Gil, Guillem Guigó i Corominas, Maciej Lewenstein
Date: 2021-08-07 00:00:00

The characterization of diffusion processes is a keystone in our understanding of a variety of physical phenomena. Many of these deviate from Brownian motion, giving rise to anomalous diffusion. Various theoretical models exists nowadays to describe such processes, but their application to experimental setups is often challenging, due to the stochastic nature of the phenomena and the difficulty to harness reliable data. The latter often consists on short and noisy trajectories, which are hard to characterize with usual statistical approaches. In recent years, we have witnessed an impressive effort to bridge theory and experiments by means of supervised machine learning techniques, with astonishing results. In this work, we explore the use of unsupervised methods in anomalous diffusion data. We show that the main diffusion characteristics can be learnt without the need of any labelling of the data. We use such method to discriminate between anomalous diffusion models and extract their physical parameters. Moreover, we explore the feasibility of finding novel types of diffusion, in this case represented by compositions of existing diffusion models. At last, we showcase the use of the method in experimental data and demonstrate its advantages for cases where supervised learning is not applicable.
Missing Data Estimation in Temporal Multilayer Position-aware Graph Neural Network (TMP-GNN)
Authors: Bahareh Najafi, Saeedeh Parsaeefard, Alberto Leon-Garcia
Date: 2021-08-07 00:00:00
Tasks: Attribute, Graph Neural Network, Node Classification, Position, Representation Learning

GNNs have been proven to perform highly effective in various node-level, edge-level, and graph-level prediction tasks in several domains. Existing approaches mainly focus on static graphs. However, many graphs change over time with their edge may disappear, or node/edge attribute may alter from one time to the other. It is essential to consider such evolution in representation learning of nodes in time varying graphs. In this paper, we propose a Temporal Multi-layered Position-aware Graph Neural Network (TMP-GNN), a node embedding approach for dynamic graph that incorporates the interdependence of temporal relations into embedding computation. We evaluate the performance of TMP-GNN on two different representations of temporal multilayered graphs. The performance is assessed against the most popular GNNs on node-level prediction tasks. Then, we incorporate TMP-GNN into a deep learning framework to estimate missing data and compare the performance with their corresponding competent GNNs from our former experiment, and a baseline method. Experimental results on four real-world datasets yield up to 58% of lower ROC AUC for pairwise node classification task, and 96% of lower MAE in missing feature estimation, particularly for graphs with a relatively high number of nodes and lower mean degree of connectivity.
Global Self-Attention as a Replacement for Graph Convolution
Authors: Md Shamim Hussain, Mohammed J. Zaki, Dharmashankar Subramanian
Date: 2021-08-07 00:00:00
Tasks: Edge Classification, Graph Classification, Graph Learning, Graph Property Prediction, Graph Regression, Inductive Bias, Link Prediction, Node Classification, Transfer Learning

We propose an extension to the transformer neural network architecture for general-purpose graph learning by adding a dedicated pathway for pairwise structural information, called edge channels. The resultant framework - which we call Edge-augmented Graph Transformer (EGT) - can directly accept, process and output structural information of arbitrary form, which is important for effective learning on graph-structured data. Our model exclusively uses global self-attention as an aggregation mechanism rather than static localized convolutional aggregation. This allows for unconstrained long-range dynamic interactions between nodes. Moreover, the edge channels allow the structural information to evolve from layer to layer, and prediction tasks on edges/links can be performed directly from the output embeddings of these channels. We verify the performance of EGT in a wide range of graph-learning experiments on benchmark datasets, in which it outperforms Convolutional/Message-Passing Graph Neural Networks. EGT sets a new state-of-the-art for the quantum-chemical regression task on the OGB-LSC PCQM4Mv2 dataset containing 3.8 million molecular graphs. Our findings indicate that global self-attention based aggregation can serve as a flexible, adaptive and effective replacement of graph convolution for general-purpose graph learning. Therefore, convolutional local neighborhood aggregation is not an essential inductive bias.
Efficient Representation for Electric Vehicle Charging Station Operations using Reinforcement Learning
Authors: Kyung-bin Kwon, Hao Zhu
Date: 2021-08-07 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning (RL)

Effectively operating electrical vehicle charging station (EVCS) is crucial for enabling the rapid transition of electrified transportation. To solve this problem using reinforcement learning (RL), the dimension of state/action spaces scales with the number of EVs and is thus very large and time-varying. This dimensionality issue affects the efficiency and convergence properties of generic RL algorithms. We develop aggregation schemes that are based on the emergency of EV charging, namely the laxity value. A least-laxity first (LLF) rule is adopted to consider only the total charging power of the EVCS which ensures the feasibility of individual EV schedules. In addition, we propose an equivalent state aggregation that can guarantee to attain the same optimal policy. Based on the proposed representation, policy gradient method is used to find the best parameters for the linear Gaussian policy . Numerical results have validated the performance improvement of the proposed representation approaches in attaining higher rewards and more effective policies as compared to existing approximation based approach.
Estimating Graph Dimension with Cross-validated Eigenvalues
Authors: Fan Chen, Sebastien Roch, Karl Rohe, Shuqi Yu
Date: 2021-08-06 00:00:00
Tasks: Diagnostic

In applied multivariate statistics, estimating the number of latent dimensions or the number of clusters is a fundamental and recurring problem. One common diagnostic is the scree plot, which shows the largest eigenvalues of the data matrix; the user searches for a "gap" or "elbow" in the decreasing eigenvalues; unfortunately, these patterns can hide beneath the bias of the sample eigenvalues. This methodological problem is conceptually difficult because, in many situations, there is only enough signal to detect a subset of the $k$ population dimensions/eigenvectors. In this situation, one could argue that the correct choice of $k$ is the number of detectable dimensions. We alleviate these problems with cross-validated eigenvalues. Under a large class of random graph models, without any parametric assumptions, we provide a p-value for each sample eigenvector. It tests the null hypothesis that this sample eigenvector is orthogonal to (i.e., uncorrelated with) the true latent dimensions. This approach naturally adapts to problems where some dimensions are not statistically detectable. In scenarios where all $k$ dimensions can be estimated, we prove that our procedure consistently estimates $k$. In simulations and a data example, the proposed estimator compares favorably to alternative approaches in both computational and statistical performance.
A Deep Neural Network Approach for Crop Selection and Yield Prediction in Bangladesh
Authors: Tanhim Islam, Tanjir Alam Chisty, Amitabha Chakrabarty
Date: 2021-08-06 00:00:00
Tasks: Prediction

Agriculture is the essential ingredients to mankind which is a major source of livelihood. Agriculture work in Bangladesh is mostly done in old ways which directly affects our economy. In addition, institutions of agriculture are working with manual data which cannot provide a proper solution for crop selection and yield prediction. This paper shows the best way of crop selection and yield prediction in minimum cost and effort. Artificial Neural Network is considered robust tools for modeling and prediction. This algorithm aims to get better output and prediction, as well as, support vector machine, Logistic Regression, and random forest algorithm is also considered in this study for comparing the accuracy and error rate. Moreover, all of these algorithms used here are just to see how well they performed for a dataset which is over 0.3 million. We have collected 46 parameters such as maximum and minimum temperature, average rainfall, humidity, climate, weather, and types of land, types of chemical fertilizer, types of soil, soil structure, soil composition, soil moisture, soil consistency, soil reaction and soil texture for applying into this prediction process. In this paper, we have suggested using the deep neural network for agricultural crop selection and yield prediction.
Smooth Symbolic Regression: Transformation of Symbolic Regression into a Real-valued Optimization Problem
Authors: Erik Pitzer, Gabriel Kronberger
Date: 2021-08-06 00:00:00
Tasks: regression, Symbolic Regression

The typical methods for symbolic regression produce rather abrupt changes in solution candidates. In this work, we have tried to transform symbolic regression from an optimization problem, with a landscape that is so rugged that typical analysis methods do not produce meaningful results, to one that can be compared to typical and very smooth real-valued problems. While the ruggedness might not interfere with the performance of optimization, it restricts the possibilities of analysis. Here, we have explored different aspects of a transformation and propose a simple procedure to create real-valued optimization problems from symbolic regression problems.