  When applying a stochastic/incremental algorithm, one must choose the order
to draw samples. Among the most popular approaches are cyclic sampling and
random reshuffling, which are empirically faster and more cache-friendly than
uniform-iid-sampling. Cyclic sampling draws the samples in a fixed, cyclic
order, which is less robust than reshuffling the samples periodically. Indeed,
existing works have established worst case convergence rates for cyclic
sampling, which are generally worse than that of random reshuffling. In this
paper, however, we found a certain cyclic order can be much faster than
reshuffling and one can discover it at a low cost!
  Studying and comparing different sampling orders typically require new
analytic techniques. In this paper, we introduce a norm, which is defined based
on the sampling order, to measure the distance to solution. Applying this
technique on proximal Finito/MISO algorithm allows us to identify the optimal
fixed ordering, which can beat random reshuffling by a factor up to log(n)/n in
terms of the best-known upper bounds. We also propose a strategy to discover
the optimal fixed ordering numerically. The established rates are
state-of-the-art compared to previous works.
