  Parameter servers (PSs) facilitate the implementation of distributed training
for large machine learning tasks. A key challenge for PS performance is that
parameter access is non-uniform in many real-world machine learning tasks,
i.e., different parameters exhibit drastically different access patterns. We
identify skew and nondeterminism as two major sources for non-uniformity.
Existing PSs are ill-suited for managing such non-uniform access because they
uniformly apply the same parameter management technique to all parameters. As
consequence, the performance of existing PSs is negatively affected and may
even fall behind that of single node baselines. In this paper, we explore how
PSs can manage non-uniform access efficiently. We find that it is key for PSs
to support multiple management techniques and to leverage a well-suited
management technique for each parameter. We present Lapse2, a PS that
replicates hot spot parameters, relocates less frequently accessed parameters,
and employs specialized techniques to manage nondeterminism that arises from
random sampling. In our experimental study, Lapse2 outperformed existing,
single-technique PSs by up to one order of magnitude and provided near-linear
scalability across multiple machine learning tasks.
