  Federated Distillation (FD) is a popular novel algorithmic paradigm for
Federated Learning, which achieves training performance competitive to prior
parameter averaging based methods, while additionally allowing the clients to
train different model architectures, by distilling the client predictions on an
unlabeled auxiliary set of data into a student model. In this work we propose
FedAUX, an extension to FD, which, under the same set of assumptions,
drastically improves performance by deriving maximum utility from the unlabeled
auxiliary data. FedAUX modifies the FD training procedure in two ways: First,
unsupervised pre-training on the auxiliary data is performed to find a model
initialization for the distributed training. Second, $(\varepsilon,
\delta)$-differentially private certainty scoring is used to weight the
ensemble predictions on the auxiliary data according to the certainty of each
client model. Experiments on large-scale convolutional neural networks and
transformer models demonstrate, that the training performance of FedAUX exceeds
SOTA FL baseline methods by a substantial margin in both the iid and non-iid
regime, further closing the gap to centralized training performance. Code is
available at github.com/fedl-repo/fedaux.
