Observations on the past provide some hints about what will happen in the future, and this can be quantified using information theory. The "predictive information" defined in this way has connections to measures of complexity that have been proposed both in the study of dynamical systems and in mathematical statistics. In particular, the predictive information diverges when the observed data stream allows us to learn an increasingly precise model for the dynamics that generate the data, and the structure of this divergence measures the complexity of the model. We argue that divergent contributions to the predictive information provide the only measure of complexity or richness that is consistent with certain plausible requirements. There is obvious interest in having practical algorithms for predicting the future, and there is a correspondingly large literature on the problem of time series extrapolation [1]. But prediction is more (and less) than extrapolation-we might be able to predict, for example, the chance of rain in the coming week even if we cannot extrapolate the trajectory of temperature fluctuations. In the spirit of its thermodynamic origins, information theory [2] characterizes the potentialities and limitations of all possible prediction algorithms, as well as unifying the analysis of extrapolation with the more general notion of predictability. Specifically, we define a quantitythe predictive information-that measures how much our observations of the past can tell us about the future. The predictive information characterizes the world we are observing, and we shall see that this characterization is close to our intuition about the complexity of the underlying dynamics. Imagine that we observe a stream of data over the period from t = −T to t = 0; this constitutes our past. For simplicity, let the future extend forward also for a time T . Let us call the data we observe during the past X past , and the data we will observe in the future X future . The usual problem is to guess the values of X future from knowledge of X past , but this is too specific; only certain features of the data stream are predictable, and even these features may be predictable only in a statistical sense. Different kinds of prediction are often treated as different problems, and when we assess the quality of these predictions we seem forced to use different metrics in the different cases. Information theory allows us to treat the different notions of prediction on the same footing. Even before we look at the data, we already know that certain futures are more likely then others, and we can summarize this knowledge by a 'prior' probability distribution for the future, P (X future ). Our observations on the past lead us to a new, more tightly concentrated distribution, the distribution of futures conditional on the past data, P (X future |X past ). Different kinds of predictions can be seen as different slices through or averages over this conditional distribution. The greater concentration of the conditional distribution can be quantified directly by the fact that it has a smaller entropy than the prior distribution, and this reduction in entropy is Shannon's definition of the information that the past provides about the future [2] . We can write the average of this predictive information as Each of the terms in Eq. (2) is an entropy. If we have invariance under time translations, then the entropy of the past data depends only on the duration of our observations, so we can write − log 2 P (X past ) = S(T ), and by the same argument − log 2 P (X future ) = S(T ). Finally, the entropy of the past and future taken together is the entropy of observations on a window of duration 2T , so that − log 2 P (X future , X past ) = S(2T ). Putting these equations together we obtain the basic relation between predictability and the time dependence of the entropy, The entropy is an extensive quantity, so that lim T →∞ S(T )/T = S. In the same way that the entropy of a gas at fixed density is proportional to the volume, the entropy of a time series is (asymptotically) proportional to its duration. This entropy is also the minimum number of bits required to give a complete description of the past data. But from Eq. (3) any extensive component of the entropy cancels in the computation of the predictive information: predictability is associated with deviation of the entropy from extensivity. The cancellation of extensive components means that the predictive information must be subextensive, lim T →∞ I pred (T )/T = 0. As a result, of the total information we have taken in by 1
A number of recent works have suggested predictive information, the amount of information shared between the past and the future of a random variable, as a measure for the amount of non-trivial information obtained from embodied interaction REF .
Predictive Information
Physics Biology