pwc-archive/papers-with-abstracts huggingface dataset:
Machine learning based disease diagnosis: A comprehensive review
Authors: Md Manjurul Ahsan, Zahed Siddique
Date: 2021-12-31 00:00:00
Tasks: Articles, BIG-bench Machine Learning

Globally, there is a substantial unmet need to diagnose various diseases effectively. The complexity of the different disease mechanisms and underlying symptoms of the patient population presents massive challenges to developing the early diagnosis tool and effective treatment. Machine Learning (ML), an area of Artificial Intelligence (AI), enables researchers, physicians, and patients to solve some of these issues. Based on relevant research, this review explains how Machine Learning (ML) and Deep Learning (DL) are being used to help in the early identification of numerous diseases. To begin, a bibliometric study of the publication is given using data from the Scopus and Web of Science (WOS) databases. The bibliometric study of 1216 publications was undertaken to determine the most prolific authors, nations, organizations, and most cited articles. The review then summarizes the most recent trends and approaches in Machine Learning-based Disease Diagnosis (MLBDD), considering the following factors: algorithm, disease types, data type, application, and evaluation metrics. Finally, the paper highlights key results and provides insight into future trends and opportunities in the MLBDD area.
Machine Learning Trivializing Maps: A First Step Towards Understanding How Flow-Based Samplers Scale Up
Authors: Luigi Del Debbio, Joe Marsh Rossney, Michael Wilson
Date: 2021-12-31 00:00:00

A trivializing map is a field transformation whose Jacobian determinant exactly cancels the interaction terms in the action, providing a representation of the theory in terms of a deterministic transformation of a distribution from which sampling is trivial. Recently, a proof-of-principle study by Albergo, Kanwar and Shanahan [arXiv:1904.12072] demonstrated that approximations of trivializing maps can be `machine-learned' by a class of invertible, differentiable neural models called \textit{normalizing flows}. By ensuring that the Jacobian determinant can be computed efficiently, asymptotically exact sampling from the theory of interest can be performed by drawing samples from a simple distribution and passing them through the network. From a theoretical perspective, this approach has the potential to become more efficient than traditional Markov Chain Monte Carlo sampling techniques, where autocorrelations severely diminish the sampling efficiency as one approaches the continuum limit. A major caveat is that it is not yet understood how the size of models and the cost of training them is expected to scale. As a first step, we have conducted an exploratory scaling study using two-dimensional $\phi^4$ with up to $20^2$ lattice sites. Although the scope of our study is limited to a particular model architecture and training algorithm, initial results paint an interesting picture in which training costs grow very quickly indeed. We describe a candidate explanation for the poor scaling, and outline our intentions to clarify the situation in future work.
Scalable Deep Graph Clustering with Random-walk based Self-supervised Learning
Authors: Xiang Li, Dong Li, Ruoming Jin, Gagan Agrawal, Rajiv Ramnath
Date: 2021-12-31 00:00:00
Tasks: Clustering, Deep Clustering, GPU, Graph Clustering, Node Clustering, Self-Supervised Learning

Web-based interactions can be frequently represented by an attributed graph, and node clustering in such graphs has received much attention lately. Multiple efforts have successfully applied Graph Convolutional Networks (GCN), though with some limits on accuracy as GCNs have been shown to suffer from over-smoothing issues. Though other methods (particularly those based on Laplacian Smoothing) have reported better accuracy, a fundamental limitation of all the work is a lack of scalability. This paper addresses this open problem by relating the Laplacian smoothing to the Generalized PageRank and applying a random-walk based algorithm as a scalable graph filter. This forms the basis for our scalable deep clustering algorithm, RwSL, where through a self-supervised mini-batch training mechanism, we simultaneously optimize a deep neural network for sample-cluster assignment distribution and an autoencoder for a clustering-oriented embedding. Using 6 real-world datasets and 6 clustering metrics, we show that RwSL achieved improved results over several recent baselines. Most notably, we show that RwSL, unlike all other deep clustering frameworks, can continue to scale beyond graphs with more than one million nodes, i.e., handle web-scale. We also demonstrate how RwSL could perform node clustering on a graph with 1.8 billion edges using only a single GPU.
Inferring perceptual decision making parameters from behavior in production and reproduction tasks
Authors: Nils Neupärtl, Constantin A. Rothkopf
Date: 2021-12-31 00:00:00
Tasks: Bayesian Inference, Decision Making

Bayesian models of behavior have provided computational level explanations in a range of psychophysical tasks. One fundamental experimental paradigm is the production or reproduction task, in which subjects are instructed to generate an action that either reproduces a previously sensed stimulus magnitude or achieves a target response. This type of task therefore distinguishes itself from other psychophysical tasks in that the responses are on a continuum and effort plays an important role with increasing response magnitude. Based on Bayesian decision theory we present an inference method to recover perceptual uncertainty, response variability, and the cost function underlying human responses. Crucially, the cost function is parameterized such that effort is explicitly included. We present a hybrid inference method employing MCMC sampling utilizing appropriate proposal distributions and an inner loop utilizing amortized inference with a neural network that approximates the mode of the optimal response distribution. We show how this model can be utilized to avoid unidentifiability of experimental designs and that parameters can be recovered through validation on synthetic and application to experimental data. Our approach will enable behavioral scientists to perform Bayesian inference of decision making parameters in production and reproduction tasks.
Representation Learning via Consistent Assignment of Views to Clusters
Authors: Thalles Silva, Adín Ramírez Rivera
Date: 2021-12-31 00:00:00
Tasks: Clustering, Contrastive Learning, Deep Clustering, Linear evaluation, Representation Learning, Transfer Learning

We introduce Consistent Assignment for Representation Learning (CARL), an unsupervised learning method to learn visual representations by combining ideas from self-supervised contrastive learning and deep clustering. By viewing contrastive learning from a clustering perspective, CARL learns unsupervised representations by learning a set of general prototypes that serve as energy anchors to enforce different views of a given image to be assigned to the same prototype. Unlike contemporary work on contrastive learning with deep clustering, CARL proposes to learn the set of general prototypes in an online fashion, using gradient descent without the necessity of using non-differentiable algorithms or K-Means to solve the cluster assignment problem. CARL surpasses its competitors in many representations learning benchmarks, including linear evaluation, semi-supervised learning, and transfer learning.
Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs
Authors: Inbar Seroussi, Gadi Naveh, Zohar Ringel
Date: 2021-12-31 00:00:00

Deep neural networks (DNNs) are powerful tools for compressing and distilling information. Their scale and complexity, often involving billions of inter-dependent parameters, render direct microscopic analysis difficult. Under such circumstances, a common strategy is to identify slow variables that average the erratic behavior of the fast microscopic variables. Here, we identify a similar separation of scales occurring in fully trained finitely over-parameterized deep convolutional neural networks (CNNs) and fully connected networks (FCNs). Specifically, we show that DNN layers couple only through the second moment (kernels) of their activations and pre-activations. Moreover, the latter fluctuates in a nearly Gaussian manner. For infinite width DNNs, these kernels are inert, while for finite ones they adapt to the data and yield a tractable data-aware Gaussian Process. The resulting thermodynamic theory of deep learning yields accurate predictions in various settings. In addition, it provides new ways of analyzing and understanding DNNs in general.
Processing Images from Multiple IACTs in the TAIGA Experiment with Convolutional Neural Networks
Authors: Stanislav Polyakov, Andrey Demichev, Alexander Kryukov, Evgeny Postnikov
Date: 2021-12-31 00:00:00

Extensive air showers created by high-energy particles interacting with the Earth atmosphere can be detected using imaging atmospheric Cherenkov telescopes (IACTs). The IACT images can be analyzed to distinguish between the events caused by gamma rays and by hadrons and to infer the parameters of the event such as the energy of the primary particle. We use convolutional neural networks (CNNs) to analyze Monte Carlo-simulated images from the telescopes of the TAIGA experiment. The analysis includes selection of the images corresponding to the showers caused by gamma rays and estimating the energy of the gamma rays. We compare performance of the CNNs using images from a single telescope and the CNNs using images from two telescopes as inputs.
Effects of Plasticity Functions on Neural Assemblies
Authors: Christodoulos Constantinides, Kareem Nassar
Date: 2021-12-29 00:00:00

We explore the effects of various plasticity functions on assemblies of neurons. To bridge the gap between experimental and computational theories we make use of a conceptual framework, the Assembly Calculus, which is a formal system for the description of brain function based on assemblies of neurons. The Assembly Calculus includes operations for projecting, associating, and merging assemblies of neurons. Our research is focused on simulating different plasticity functions with Assembly Calculus. Our main contribution is the modification and evaluation of the projection operation. We experiment with Oja's and Spike Time-Dependent Plasticity (STDP) rules and test the effect of various hyper-parameters.
Private Private Information
Authors: Kevin He, Fedor Sandomirskiy, Omer Tamuz
Date: 2021-12-29 00:00:00
Tasks: Causal Inference, Fairness, Informativeness, Recommendation Systems

Private signals model noisy information about an unknown state. Although these signals are called "private," they may still carry information about each other. Our paper introduces the concept of private private signals, which contain information about the state but not about other signals. To achieve privacy, signal quality may need to be sacrificed. We study the informativeness of private private signals and characterize those that are optimal in the sense that they cannot be made more informative without violating privacy. We discuss implications for privacy in recommendation systems, information design, causal inference, and mechanism design.
Learning in Repeated Interactions on Networks
Authors: Wanying Huang, Philipp Strack, Omer Tamuz
Date: 2021-12-28 00:00:00

We study how long-lived, rational agents learn in a social network. In every period, after observing the past actions of his neighbors, each agent receives a private signal, and chooses an action whose payoff depends only on the state. Since equilibrium actions depend on higher order beliefs, it is difficult to characterize behavior. Nevertheless, we show that regardless of the size and shape of the network, the utility function, and the patience of the agents, the speed of learning in any equilibrium is bounded from above by a constant that only depends on the private signal distribution.
Training Recurrent Neural Networks by Sequential Least Squares and the Alternating Direction Method of Multipliers
Authors: Alberto Bemporad
Date: 2021-12-31 00:00:00

This paper proposes a novel algorithm for training recurrent neural network models of nonlinear dynamical systems from an input/output training dataset. Arbitrary convex and twice-differentiable loss functions and regularization terms are handled by sequential least squares and either a line-search (LS) or a trust-region method of Levenberg-Marquardt (LM) type for ensuring convergence. In addition, to handle non-smooth regularization terms such as $\ell_1$, $\ell_0$, and group-Lasso regularizers, as well as to impose possibly non-convex constraints such as integer and mixed-integer constraints, we combine sequential least squares with the alternating direction method of multipliers (ADMM). We call the resulting algorithm NAILS (nonconvex ADMM iterations and least squares) in the case line search (LS) is used, or NAILM if a trust-region method (LM) is employed instead. The training method, which is also applicable to feedforward neural networks as a special case, is tested in three nonlinear system identification problems.
Sufficient-Statistic Memory AMP
Authors: Lei Liu, Shunqi Huang, Yuzhi Yang, Zhaoyang Zhang, Brian M. Kurkoski
Date: 2021-12-31 00:00:00

Approximate message passing (AMP) type algorithms have been widely used in the signal reconstruction of certain large random linear systems. A key feature of the AMP-type algorithms is that their dynamics can be correctly described by state evolution. While state evolution is a useful analytic tool, its convergence is not guaranteed. To solve the convergence problem of the state evolution of AMP-type algorithms in principle, this paper proposes a sufficient-statistic memory AMP (SS-MAMP) algorithm framework under the conditions of right-unitarily invariant sensing matrices, Lipschitz-continuous local processors and the sufficient-statistic constraint (i.e., the current message of each local processor is a sufficient statistic of the signal vector given the current and all preceding messages). We show that the covariance matrices of SS-MAMP are L-banded and convergent, which is an optimal framework (from the local MMSE/LMMSE perspective) for AMP-type algorithms given the Lipschitz-continuous local processors. Given an arbitrary MAMP, we can construct an SS-MAMP by damping, which not only ensures the convergence of the state evolution, but also preserves the orthogonality, i.e., its dynamics can be correctly described by state evolution. As a byproduct, we prove that the Bayes-optimal orthogonal/vector AMP (BO-OAMP/VAMP) is an SS-MAMP. As an example, we construct a sufficient-statistic Bayes-optimal MAMP (SS-BO-MAMP) whose state evolution converges to the minimum (i.e., Bayes-optimal) mean square error (MSE) predicted by replica methods when it has a unique fixed point. In addition, the MSE of SS-BO-MAMP is not worse than the original BO-MAMP. Finally, simulations are provided to support the theoretical results.
Financial Vision Based Differential Privacy Applications
Authors: Jun-Hao Chen, Yi-Jen Wang, Yun-Cheng Tsai, Samuel Yen-Chi Chen
Date: 2021-12-28 00:00:00
Tasks: Deep Learning

The importance of deep learning data privacy has gained significant attention in recent years. It is probably to suffer data breaches when applying deep learning to cryptocurrency that lacks supervision of financial regulatory agencies. However, there is little relative research in the financial area to our best knowledge. We apply two representative deep learning privacy-privacy frameworks proposed by Google to financial trading data. We designed the experiments with several different parameters suggested from the original studies. In addition, we refer the degree of privacy to Google and Apple companies to estimate the results more reasonably. The results show that DP-SGD performs better than the PATE framework in financial trading data. The tradeoff between privacy and accuracy is low in DP-SGD. The degree of privacy also is in line with the actual case. Therefore, we can obtain a strong privacy guarantee with precision to avoid potential financial loss.
Learning in Random Utility Models Via Online Decision Problems
Authors: Emerson Melo
Date: 2021-12-21 00:00:00

This paper studies the Random Utility Model (RUM) in a repeated stochastic choice situation, in which the decision maker is imperfectly informed about the payoffs of each available alternative. We develop a gradient-based learning algorithm by embedding the RUM into an online decision problem. We show that a large class of RUMs are Hannan consistent (\citet{Hahn1957}); that is, the average difference between the expected payoffs generated by a RUM and that of the best-fixed policy in hindsight goes to zero as the number of periods increase. In addition, we show that our gradient-based algorithm is equivalent to the Follow the Regularized Leader (FTRL) algorithm, which is widely used in the machine learning literature to model learning in repeated stochastic choice problems. Thus, we provide an economically grounded optimization framework to the FTRL algorithm. Finally, we apply our framework to study recency bias, no-regret learning in normal form games, and prediction markets.
Balancing Adaptability and Non-exploitability in Repeated Games
Authors: Anthony DiGiovanni, Ambuj Tewari
Date: 2021-12-20 00:00:00

We study the problem of guaranteeing low regret in repeated games against an opponent with unknown membership in one of several classes. We add the constraint that our algorithm is non-exploitable, in that the opponent lacks an incentive to use an algorithm against which we cannot achieve rewards exceeding some "fair" value. Our solution is an expert algorithm (LAFF) that searches within a set of sub-algorithms that are optimal for each opponent class and uses a punishment policy upon detecting evidence of exploitation by the opponent. With benchmarks that depend on the opponent class, we show that LAFF has sublinear regret uniformly over the possible opponents, except exploitative ones, for which we guarantee that the opponent has linear regret. To our knowledge, this work is the first to provide guarantees for both regret and non-exploitability in multi-agent learning.
A Critical Review of Inductive Logic Programming Techniques for Explainable AI
Authors: Zheng Zhang, Liangliang Xu, Levent Yilmaz, Bo Liu
Date: 2021-12-31 00:00:00
Tasks: BIG-bench Machine Learning, Explainable artificial intelligence, Inductive logic programming, Relational Reasoning

Despite recent advances in modern machine learning algorithms, the opaqueness of their underlying mechanisms continues to be an obstacle in adoption. To instill confidence and trust in artificial intelligence systems, Explainable Artificial Intelligence has emerged as a response to improving modern machine learning algorithms' explainability. Inductive Logic Programming (ILP), a subfield of symbolic artificial intelligence, plays a promising role in generating interpretable explanations because of its intuitive logic-driven framework. ILP effectively leverages abductive reasoning to generate explainable first-order clausal theories from examples and background knowledge. However, several challenges in developing methods inspired by ILP need to be addressed for their successful application in practice. For example, existing ILP systems often have a vast solution space, and the induced solutions are very sensitive to noises and disturbances. This survey paper summarizes the recent advances in ILP and a discussion of statistical relational learning and neural-symbolic algorithms, which offer synergistic views to ILP. Following a critical review of the recent advances, we delineate observed challenges and highlight potential avenues of further ILP-motivated research toward developing self-explanatory artificial intelligence systems.
Bayesian Optimization of Function Networks
Authors: Raul Astudillo, Peter I. Frazier
Date: 2021-12-31 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Bayesian Optimization, Gaussian Processes

We consider Bayesian optimization of the output of a network of functions, where each function takes as input the output of its parent nodes, and where the network takes significant time to evaluate. Such problems arise, for example, in reinforcement learning, engineering design, and manufacturing. While the standard Bayesian optimization approach observes only the final output, our approach delivers greater query efficiency by leveraging information that the former ignores: intermediate output within the network. This is achieved by modeling the nodes of the network using Gaussian processes and choosing the points to evaluate using, as our acquisition function, the expected improvement computed with respect to the implied posterior on the objective. Although the non-Gaussian nature of this posterior prevents computing our acquisition function in closed form, we show that it can be efficiently maximized via sample average approximation. In addition, we prove that our method is asymptotically consistent, meaning that it finds a globally optimal solution as the number of evaluations grows to infinity, thus generalizing previously known convergence results for the expected improvement. Notably, this holds even though our method might not evaluate the domain densely, instead leveraging problem structure to leave regions unexplored. Finally, we show that our approach dramatically outperforms standard Bayesian optimization methods in several synthetic and real-world problems.
Convergence Rates of Two-Time-Scale Gradient Descent-Ascent Dynamics for Solving Nonconvex Min-Max Problems
Authors: Thinh T. Doan
Date: 2021-12-17 00:00:00
Tasks: Distributed Optimization

There are much recent interests in solving noncovnex min-max optimization problems due to its broad applications in many areas including machine learning, networked resource allocations, and distributed optimization. Perhaps, the most popular first-order method in solving min-max optimization is the so-called simultaneous (or single-loop) gradient descent-ascent algorithm due to its simplicity in implementation. However, theoretical guarantees on the convergence of this algorithm is very sparse since it can diverge even in a simple bilinear problem. In this paper, our focus is to characterize the finite-time performance (or convergence rates) of the continuous-time variant of simultaneous gradient descent-ascent algorithm. In particular, we derive the rates of convergence of this method under a number of different conditions on the underlying objective function, namely, two-sided Polyak-L ojasiewicz (PL), one-sided PL, nonconvex-strongly concave, and strongly convex-nonconcave conditions. Our convergence results improve the ones in prior works under the same conditions of objective functions. The key idea in our analysis is to use the classic singular perturbation theory and coupling Lyapunov functions to address the time-scale difference and interactions between the gradient descent and ascent dynamics. Our results on the behavior of continuous-time algorithm may be used to enhance the convergence properties of its discrete-time counterpart.