We propose a model of a learning agent whose interaction with the environment is governed by a simulation-based projection, which allows the agent to project itself into future situations before it takes real action. Projective simulation is based on a random walk through a network of clips, which are elementary patches of episodic memory. The network of clips changes dynamically, both due to new perceptual input and due to certain compositional principles of the simulation process. During simulation, the clips are screened for specific features which trigger factual action of the agent. The scheme is different from other, computational, notions of simulation, and it provides a new element in an embodied cognitive science approach to intelligent action and learning. Our model provides a natural route for generalization to quantum-mechanical operation and connects the fields of reinforcement learning and quantum computation. C omputers of various sorts play a role in many processes of modern society. A prominent example is the personal computer which has a specific user interface, waiting for human input and delivering output in a prescribed format. Computers also feature in automated processes, for example in the production lines of a modern factory. Here the input/output interface is usually with other machinery, such as a robot environment in a car factory. An increasingly important role is played by so-called intelligent agents that operate autonomously in more complex and changing environments. Examples of such environments are traffic, remote space, but also the internet. The design of intelligent agents, specifically for tasks such as learning 1 , has become a unifying agenda of various branches of artificial intelligence 2 . Intelligence is hereby defined as the capability of the agent to perceive and act on its environment in a way that maximizes its chances of success. In recent years, the field of embodied cognitive sciences 3 has provided a new conceptual and empirical framework for the study of intelligence, both in biological and in artificial entities. A particular manifestation of intelligence is creativity and it is therefore natural to ask: To what extent can agents or robots show creative behavior? Creativity is hereby understood as a distinguished capability of dealing with unprecedented situations and of relating a given situation with other conceivable situations. A similar question may arise in behavioral studies with animals, and it is related, on a more fundamental level, to the problem of free will 4 . In this paper, we introduce a scheme of information processing for intelligent agents which allows for an element of creative behavior in the above sense. Its central feature is a projection simulator (PS) which allows the agent, based on previous experience -and variations thereof-to project itself into potential future situations. The PS uses a specific memory system, which we call episodic & compositional memory (ECM) and which provides the platform for simulating future action before real action is taken. The ECM can be described as a stochastic network of so-called clips, which constitute the elementary excitations of episodic memory. Projective simulation consists of a replay of clips representing previous experience, together with the creation of new clips under certain variational and compositional principles. The simulation requires a platform which is detached from direct motor action and on which fictitious action is continuously ''tested''. Learning takes place by a continuous modification of the network of clips, which occurs in three distinct ways: (1) adaptive changes of transition probabilities between existing clips (bayesian updating); (2) creation of new clips in the network via new perceptual input (new clips from new percepts); (3) creation of new clips from existing ones under certain compositional principles (new clips through composition). In modern physics, the notion of simulation and the ultimate power of physical systems to simulate other systems has become one of the central topics in the field of quantum information and computation 5 . A timely example is the universal quantum simulator, which is capable of mimicking the time evolution of any other
Projective Simulation The projective simulation (PS) model for artificial intelligence REF is a model for agency which employs a specific form of an episodic and compositional memory to make decisions.
Projective simulation for artificial intelligence
Scientific Reports
Scientific Reports
Computer Science Physics Medicine Mathematics