  Working memory is a cognitive function involving the storage and manipulation
of latent information over brief intervals of time, thus making it crucial for
context-dependent computation. Here, we use a top-down modeling approach to
examine network-level mechanisms of working memory, an enigmatic issue and
central topic of study in neuroscience and machine intelligence. We train
thousands of recurrent neural networks on a working memory task and then
perform dynamical systems analysis on the ensuing optimized networks, wherein
we find that four distinct dynamical mechanisms can emerge. In particular, we
show the prevalence of a mechanism in which memories are encoded along slow
stable manifolds in the network state space, leading to a phasic neuronal
activation profile during memory periods. In contrast to mechanisms in which
memories are directly encoded at stable attractors, these networks naturally
forget stimuli over time. Despite this seeming functional disadvantage, they
are more efficient in terms of how they leverage their attractor landscape and
paradoxically, are considerably more robust to noise. Our results provide new
dynamical hypotheses regarding how working memory function is encoded in both
natural and artificial neural networks.
