While eminently successful for the transmission of data, Shannon's theory of information does not address semantic and subjective dimensions of data, such as relevance and surprise. We propose an observer-dependent computational theory of surprise where surprise is de ned by the relative e n tropy b e t ween the prior and the posterior distributions of an observer. Surprise requires integration over the space of models in contrast with Shannon's entropy, which requires integration over the space of data. We show how surprise can be computed exactly in a number of discrete and continuous cases using distributions from the exponential family with conjugate priors. We s h o w that during sequential Bayesian learning, surprise decreases like 1 =N and study how surprise di ers and complements Shannon's de nition of information.
Baldi introduced a computational theory of surprise where surprise is defined by the relative entropy between the prior and the posterior distribution of an observer REF .
A Computational Theory of Surprise
Information, Coding and Mathematics
Information, Coding and Mathematics
Computer Science