We consider generalized linear models where an unknown n-dimensional signal vector is observed through the successive application of a random matrix and a non-linear (possibly probabilistic) componentwise function. We consider the models in the high-dimensional limit, where the observation consists of m points, and m/n → α where α stays finite in the limit m, n → ∞. This situation is ubiquitous in applications ranging from supervised machine learning to signal processing. A substantial amount of work suggests that both the inference and learning tasks in these problems have sharp intrinsic limitations when the available data become too scarce or too noisy. Here, we provide rigorous asymptotic predictions for these thresholds through the proof of a simple expression for the mutual information between the observations and the signal. Thanks to this expression we also obtain as a consequence the optimal value of the generalization error in many statistical learning models of interest, such as the teacher-student binary perceptron, and introduce several new models with remarquable properties. We compute these thresholds (or "phase transitions") using ideas from statistical physics that are turned into rigorous methods thanks to a new powerful smart-path interpolation technique called the stochastic interpolation method, which has recently been introduced by two of the authors. Moreover we show that a polynomial-time algorithm refered to as generalized approximate message-passing reaches the optimal generalization performance for a large set of parameters in these problems. Our results clarify the difficulties and challenges one has to face when solving complex high-dimensional statistical problems. Authors contributions: J. Barbier, N. Macris and L. Miolane designed the proofs of the information theoretic results, namely of the replica formula and of the optimal errors. F. Krzakala and L. Zdeborová obtained the algorithmic and optimal phase transitions. All authors participated in the redaction of the paper.
The theoretical part of this paper is presented for a generalization of the perceptron model, specifically the for the random teacher-student Generalized Linear Models (GLM), see e.g. REF .
Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
Proceedings of the National Academy of Sciences 116. 12 (2019): 5451-5460
Computer Science Medicine Physics Mathematics