  Classical federated learning approaches incur significant performance
degradation in the presence of non-IID client data. A possible direction to
address this issue is forming clusters of clients with roughly IID data. Most
solutions following this direction are iterative and relatively slow, also
prone to convergence issues in discovering underlying cluster formations. We
introduce federated learning with taskonomy (FLT) that generalizes this
direction by learning the task-relatedness between clients for more efficient
federated aggregation of heterogeneous data. In a one-off process, the server
provides the clients with a pretrained (and fine-tunable) encoder to compress
their data into a latent representation, and transmit the signature of their
data back to the server. The server then learns the task-relatedness among
clients via manifold learning, and performs a generalization of federated
averaging. FLT can flexibly handle a generic client relatedness graph, when
there are no explicit clusters of clients, as well as efficiently decompose it
into (disjoint) clusters for clustered federated learning. We demonstrate that
FLT not only outperforms the existing state-of-the-art baselines in non-IID
scenarios but also offers improved fairness across clients.
