Information Theoretic Meta Learning with Gaussian Processes
Authors: Michalis Titsias, Sotirios Nikoloutsopoulos, Alexandre Galashov
Keywords: Meta Learning, Information Bottleneck, Gaussian Processes, Few-shot learning, Variational Inference
Abstract: We formulate meta learning using information theoretic concepts such as mutual information and the information bottleneck. The idea is to learn a stochastic representation or encoding of the task description, given by a training or support set, that is highly informative about predicting the validation set. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning. We  particularly develop new memory-based meta learning algorithms based on Gaussian processes and derive extensions that combine memory and gradient-based meta learning. We demonstrate our method on few-shot regression and classification by using standard benchmarks such as Omniglot, mini-Imagenet and Augmented Omniglot.