  Statistical preconditioning can be used to design fast methods for
distributed large-scale empirical risk minimization problems, for strongly
convex and smooth loss functions, allowing fewer communication rounds. Multiple
worker nodes compute gradients in parallel, which are then used by the central
node to update the parameter by solving an auxiliary (preconditioned)
smaller-scale optimization problem. However, previous works require an exact
solution of an auxiliary optimization problem by the central node at every
iteration, which may be impractical. This paper proposes a method that allows
the inexact solution of the auxiliary problem, reducing the total computation
time. Moreover, for loss functions with high-order smoothness, we exploit the
structure of the auxiliary problem and propose a hyperfast second-order method
with complexity $\tilde{O}(\kappa^{1/5})$, where $\kappa$ is the local
condition number. Combining these two building blocks (inexactness and
hyperfast methods), we show complexity estimates for the proposed algorithm,
which is provably better than classical variance reduction methods and has the
same convergence rate as statistical preconditioning with exact solutions.
Finally, we illustrate the proposed method's practical efficiency by performing
large-scale numerical experiments on logistic regression models.
