  Cognitive Psychology and related disciplines have identified several critical
mechanisms that enable intelligent biological agents to learn to solve complex
problems. There exists pressing evidence that the cognitive mechanisms that
enable problem-solving skills in these species build on hierarchical mental
representations. Among the most promising computational approaches to provide
comparable learning-based problem-solving abilities for artificial agents and
robots is hierarchical reinforcement learning. However, so far the existing
computational approaches have not been able to equip artificial agents with
problem-solving abilities that are comparable to intelligent animals, including
human and non-human primates, crows, or octopuses. Here, we first survey the
literature in Cognitive Psychology, and related disciplines, and find that many
important mental mechanisms involve compositional abstraction, curiosity, and
forward models. We then relate these insights with contemporary hierarchical
reinforcement learning methods, and identify the key machine intelligence
approaches that realise these mechanisms. As our main result, we show that all
important cognitive mechanisms have been implemented independently in isolated
computational architectures, and there is simply a lack of approaches that
integrate them appropriately. We expect our results to guide the development of
more sophisticated cognitively inspired hierarchical methods, so that future
artificial agents achieve a problem-solving performance on the level of
intelligent animals.
