  Research in machine learning is at a turning point. While supervised deep
learning has conquered the field at a breathtaking pace and demonstrated the
ability to solve inference problems with unprecedented accuracy, it still does
not quite live up to its name if we think of learning as the process of
acquiring knowledge about a subject or problem. Major weaknesses of present-day
deep learning models are, for instance, their lack of adaptability to changes
of environment or their incapability to perform other kinds of tasks than the
one they were trained for. While it is still unclear how to overcome these
limitations, one can observe a paradigm shift within the machine learning
community, with research interests shifting away from increasing the
performance of highly parameterized models to exceedingly specific tasks, and
towards employing machine learning algorithms in highly diverse domains. This
research question can be approached from different angles. For instance, the
field of Informed AI investigates the problem of infusing domain knowledge into
a machine learning model, by using techniques such as regularization, data
augmentation or post-processing.
  On the other hand, a remarkable number of works in the recent years has
focused on developing models that by themselves guarantee a certain degree of
versatility and invariance with respect to the domain or problem at hand. Thus,
rather than investigating how to provide domain-specific knowledge to machine
learning models, these works explore methods that equip the models with the
capability of acquiring the knowledge by themselves. This white paper provides
an introduction and discussion of this emerging field in machine learning
research. To this end, it reviews the role of knowledge in machine learning,
and discusses its relation to the concept of invariance, before providing a
literature review of the field.
