  An essential aspect for adequate predictions of chemical properties by
machine learning models is the database used for training them. However,
studies that analyze how the content and structure of the databases used for
training impact the prediction quality are scarce. In this work, we analyze and
quantify the relationships learned by a machine learning model (Neural Network)
trained on five different reference databases (QM9, PC9, ANI-1E, ANI-1 and
ANI-1x) to predict tautomerization energies from molecules in Tautobase. For
this, characteristics such as the number of heavy atoms in a molecule, number
of atoms of a given element, bond composition, or initial geometry on the
quality of the predictions are considered. The results indicate that training
on a chemically diverse database is crucial for obtaining good results but also
that conformational sampling can partly compensate for limited coverage of
chemical diversity. We explicitly demonstrate that when certain types of bonds
need to be covered in the target database (Tautobase) but are undersampled in
the reference databases the resulting predictions are poor. A quantitative
measure for these deficiencies is the Kullback-Leibler divergence between
reference and target distributions. Analysis of the results with a TreeMAP
algorithm provides deeper understanding of specific deficiencies in the
reference data sets. Capitalizing on this information can be used to either
improve existing databases or to generate new databases of sufficient diversity
for a range of ML applications in chemistry.
