We examine the problem of optimal sparse encoding of sensory stimuli by latent variables in stochastic models. Analyzing restricted Boltzmann machines with a communications theory approach, we search for the minimal model size that correctly conveys the correlations in stimulus patterns in an information-theoretic sense. We show that the Fisher information Matrix (FIM) reveals the optimal model size. In larger models the FIM reveals that irrelevant parameters are associated with individual latent variables, displaying a surprising amount of order . For well-fit models, we observe the emergence of statistical criticality as diverging generalized susceptibility of the model. In this case, an encoding strategy is adopted where highly informative , but rare stimuli selectively suppress variability in the encoding units. The information content of the encoded stimuli acts as an unobserved variable leading to criticality. Together, these results can explain the stimulus-dependent variability suppression observed in sensory systems, and suggest a simple, correlation-based measure to reduce the size of artificial neural networks .
In this work we explore emergent encoding strategies learned by statistical models of sensory coding in noisy spiking networks. Early stages of sensory communication in neural systems can be viewed as encoding channels in the information-theoretic sense. However, spiking neural populations face constraints not commonly considered in communications theory. Using Restricted Boltzmann machines, we find that networks with sufficient capacity learn to balance precision and noise-robustness in order to adaptively communicate stimuli with varying information content. Mirroring variability suppression observed in sensory systems, highly informative stimuli are encoded with high precision, at the cost of highly variable responses to frequent, hence less informative stimuli. We find that this coding strategy corresponds to statistical criticality in the neural population code, and emerges at model sizes where the input statistics are well captured. These phenomena have well-defined thermodynamic interpretations, and we discuss their connection to prevailing theories of coding and statistical criticality in neural populations .
R
We examine the problem of optimal sparse encoding of sensory stimuli by latent variables in stochastic models. Analyzing restricted Boltzmann machines with a communications theory approach, we search for the minimal model size that correctly conveys the correlations in stimulus patterns in an
In this work we explore emergent encoding strategies learned by statistical models of sensory coding in noisy spiking networks. Early stages of sensory communication in neural systems can be viewed as encoding channels in the
R
We show that the Fisher information Matrix (FIM) reveals the optimal model size. In larger models the FIM reveals that irrelevant parameters are associated with individual latent variables, displaying a surprising amount of order . For well-fit models, we observe the emergence of statistical criticality as diverging generalized susceptibility of the model. In this case, an encoding strategy is adopted where highly informative , but rare stimuli selectively suppress variability in the encoding units. The information content of the encoded stimuli acts as an unobserved variable leading to criticality. Together, these results can explain the stimulus-dependent variability suppression observed in sensory systems, and suggest a simple, correlation-based measure to reduce the size of artificial neural networks
However, spiking neural populations face constraints not commonly considered in communications theory. Using Restricted Boltzmann machines, we find that networks with sufficient capacity learn to balance precision and noise-robustness in order to adaptively communicate stimuli with varying information content. Mirroring variability suppression observed in sensory systems, highly informative stimuli are encoded with high precision, at the cost of highly variable responses to frequent, hence less informative stimuli. We find that this coding strategy corresponds to statistical criticality in the neural population code, and emerges at model sizes where the input statistics are well captured. These phenomena have well-defined thermodynamic interpretations, and we discuss their connection to prevailing theories of coding and statistical criticality in neural populations