Abstract Modern society depends on the flow of information over online social networks, and popular social platforms now generate significant behavioral data. Yet it remains unclear what fundamental limits may exist when using these data to predict the activities and interests of individuals. Here we apply tools from information theory to estimate the predictive information content of the writings of Twitter users and to what extent that information flows between users. Distinct temporal and social effects are visible in the information flow, and these estimates provide a fundamental bound on the predictive accuracy achievable with these data. Due to the social flow of information, we estimate that approximately 95% of the potential predictive accuracy attainable for an individual is available within the social ties of that individual only, without requiring the individual's data. The flow of information in online social platforms is now a significant factor in protest movements, national elections, and rumor and misinformation campaigns [1, 2, 3]. Data collected from these platforms are a boon for researchers [4] but also a source of concern for privacy, as the social flow of predictive information can reveal details on both users and non-users of the platform [5]. Information flow on social media has primarily been studied structurally (for example, by tracking the movements of keywords [2, 6, 7, 8] or adoptions of behaviors [9, 10, 11]) or temporally, by applying tools from information theory to quantify the information contained in the timings of user activity, as temporal relationships between user activity reflect underlying coordination patterns [12, 13]. Yet neither approach considers the full extent of information available, both the complete language data provided by individuals and their temporal activity patterns. Here we unify these two primary approaches, by applying information-theoretic estimators to a collection of Twitter user activities that fully incorporate language data while also accounting for the temporal ordering of user activities. We gathered a dataset of n = 927 users of the Twitter social media platform. Users were selected who wrote in English, were active for at least one year, and had comparably sized social networks. We applied both computational tools and human raters to help avoid bots and non-personal accounts. For each user, we retrieved all of their public posts excluding retweets (up to the 3200 most recent public posts, as allowed by Twitter). Examining these texts, we determined each user's 15 most frequent Twitter contacts and gathered the texts of those users as well, providing us ego-alter pairs. See Supporting Material (SM) for full details on data collection, filtering, and processing. The ability to accurately profile and predict individuals is reflected in the predictability of their written text. The predictive information contained within a user's text can be characterized by three related quantities, the entropy rate h, the perplexity 2 h , and the predictability Π. The entropy rate quantifies the average uncertainty one has about future words given the text one has already observed (Fig. 1A ). Higher entropies correspond to less predictable text and reflect individuals whose interests are more difficult to predict. In the context of language modeling, it is also common to consider the perplexity. While the entropy rate specifies how many bits h are needed on average to express subsequent words given the preceding text, the perplexity tells us that our remaining uncertainty about those unseen words is equivalent to that of choosing uniformly at random from among 2 h possibilities. For example, if h = 6 bits (typical of individuals in our dataset), the perplexity is 64 words, a significant reduction from choosing randomly over the entire vocabulary (social media users have ≈5000-word vocabularies on average; see SM). Lastly, the predictability Π, given via Fano's inequality [14] , is the probability that an ideal predictive algorithm will correctly predict the subsequent word given the preceding text. Repeated, accurate predictions of future words indicate that the available information can be used to build profiles and predictive models of a user and estimating Π allows us to fundamentally bound the usefulness of the information present in a user's writing without depending on the results of specific predictive algorithms. Information theory has a long history of estimating the information content of text [15, 16, 17, 18] . Crucially, information is available not just in the words of the text but in their order of appearance. We applied a nonparametric entropy estimator that incorporates the full sequence structure of the text [18] . This estimator has been proved to converge asymptotically to the true entropy rate for stationary processes and has been applied to human mobility data [19] . See SM for details. Predictive information may be available in the pasts of multiple alters, so we computed the cumulative cross-entropy as we included more alters in order of most to least frequently contacted. (D) As the past activities of more alters are used to predict the ego, more information is available and the entropy drops and predictability rises. Including the ego's past with the alters shows that the alters provided non-redundant predictive information. (E) Extrapolating beyond our data window estimates the prediction limit Π ∞ of online activity. The text streams of the egos were relatively well clustered around h ≈ 6.6 bits, with most falling between 5.5-8 bits (Fig. 1B) . Equivalently, this corresponds to a perplexity range of ≈45-256 words, far smaller than the typical user's ≈5000-word vocabulary, and a mean predictability of ≈53%, quite high for predicting a given word out of ≈5000 possible words on average. We found this typical value of information comparable to other sources of written text, but social media texts were more broadly distributed-individuals were more likely to be either highly predictable or highly unpredictable compared with formally written text (see SM). what the ego will write in the future, we ask how much information is present on average in what the alter has previously written regarding what the ego will write in the future (Fig. 1A) . If there is consistent, predictive 3 information in the alter's past about the ego's future, especially beyond the information available in the ego's own past, then there is evidence of information flow. Replacing the ego's past writing with the alter's past converts the entropy to the cross-entropy (see SM). The cross-entropy is always greater than the entropy when the alter provides less information about the ego than the ego, and so an increase in cross-entropy tells us how much information we lose by only having access to the alter's information instead of the ego's. Indeed, estimating the cross-entropy between each ego and their most frequently contacted alter (Fig. 1B) , we saw higher cross-entropies spanning from 6-12 bits (equivalently, perplexities from 64-4096 words or predictabilities spread from 0-60%). While less frequently contacted alters provided less predictive information than alters in close contact (see SM), even for the closest alters there was a broader range of cross-entropies than the entropies of the egos themselves. This implies a diversity of social relationships: sometimes the ego is well informed by the alter, leading to a cross-entropy closer to the ego's entropy, while other times the ego and alter exhibit little information flow. Thus far we have examined the information flow between the ego and individual alters, but actionable information regarding the future of the ego may be embedded in the combined pasts of multiple alters (Fig. 1C) . To address this, we generalized the cross-entropy estimator to multiple text streams (see SM). We then computed the cross-entropies and predictabilities as we successively accumulated alters in order of decreasing contact volume (Fig. 1D) . As more alters were considered, cross-entropy decreased and predictability increased, which is sensible as more potential information is available. Interestingly, with 8-9 alters, we observed a predictability of the ego given the alters at or above the original predictability of the ego alone. As more alters were added, up to our data limit of 15 alters, this increase continued. Paradoxically, this indicated that there is potentially more information about the ego within the total set of alters than within the ego itself. To understand this apparent paradox, we need to address a limitation with the above analysis: it does not incorporate the ego's own past information. It may be that the information provided by the alters is simply redundant when compared with that of the ego. Therefore, we simply included the ego's past alongside the alters, generalizing the estimator to an entropy akin to a transfer entropy [20, 21] , a common approach to studying information flow. This entropy is computed in the "Alters and ego" curves in Fig. 1D . A single alter provided a small amount of extra information beyond that of the ego, ≈1.9% more predictability. This 4 provided us a quantitative measure of the extent of information flow between individual users of social media Beyond the most frequently contacted alter, as more alters were added this extra predictability grew: at 15 alters and the ego there was ≈6.9% more predictability than via the ego alone. Furthermore, the information provided by the alters without the ego is strictly less than the information provided by the ego and alters together, resolving the apparent paradox. However, this extra predictability also appeared to saturate, and eventually adding more alters will not provide extra information. This observation is compatible with Dunbar's number which uses cognitive limits to argue for an upper bound on the number of meaningful ties an ego can maintain (≈150 alters) [22] . The question now becomes, given enough ties what is the upper bound for the predictability? To extrapolate beyond our data window, we fitted a nonlinear saturating function to the curves in Fig. 1D , (see SM for details and validation of our extrapolation procedure). From fits to the raw data, we found a limiting predictability given the alters of Π ∞ = 60.8% ± 0.691% (Fig. 1E ). Of course, egos will not have an infinite number of alters, so a more plausible extrapolation point may be to Dunbar's number: Π 150 ≈ 60.3%, within the margin of error for Π ∞ , indicating that saturation of predictive information has been reached. Similarly, extrapolating the predictability including the ego's past gives Π ∞ = 64.0% ± 1.54% (Π 150 = 63.5%). These extrapolations showed that significant predictive information was available in the combined social ties of individual users of social media. In fact, there is so much social information that an entity with access to all social media data will have only slightly more potential predictive accuracy (≈64% in our case) than an entity which has access to the activities of an ego's alters but not to that ego (≈61%). This may have distinct implications for privacy: if an individual forgoes using a social media platform or deletes her account, yet her social ties remain, then that platform owner potentially possesses 95.1% ± 3.36% of the achievable predictive accuracy of the future activities of that individual. Two issues can affect the cross-entropy as a measure of information flow. The first is that the predictive information may be due simply to the structure of English: commonly repeated words and phrases will represent a portion of the information flow. The second is that of a common cause: egos and alters may be independently discussing the same concepts. This is particularly important on social media with its emphasis on current events [23] . To study these issues, we constructed two types of controls. The first randomly pairs users together by shuffling alters between egos. The second constructed pseudo-alters by assembling, for each real alter, a random set of posts made at approximately the same times as the real alter's posts, thus controlling for temporal confounds. Both controls used real posted text and only varied the sources of the text. As shown in Fig. 1D , the real alters provided more social information than either control. There was a decrease in entropy as more control alters were added, but the control cross-entropy remained above the real cross-entropy. We also observed that for a single alter the temporal control had a lower cross-entropy than the social control and therefore temporal effects provide more information than social effects (underscoring the role of social media as a news platform [23]), although both controls eventually converge to a limiting predictability of Given the importance of temporal information in online activity, to what extent is this reflected in the information flow? Do recent activities contain most of the predictive information or are there long-term sources of information? To estimate recency effects, we applied a censoring filter to the ego's text stream, removing at each time period the text written in the previous ∆T hours and measuring how much the mean predictability decreased compared with the mean predictability including the recent text. Increasing ∆T decreased Π, especially evident when removing the first 3-4 hours ( Fig. 2A) : we found an average decrease in predictability of ≈1.4% at 4 hours. This loss in predictability relative to the uncensored baseline is comparable to the gain from the rank-1 alter we observed in Fig. 1D . In other words, close alters tended to contain a quantity of information about the ego comparable to the information within just a few hours of the ego's own recent past. Beyond 24 hours the predictability loss continued approximately linearly (SM). We then applied this censoring procedure to the alters alone and the alters combined with the ego, excluding their recent text and measuring how the cross predictability changed on average from their respective baselines. We found a similar drop in predictability during the first few hours, but then a more level trend than when censoring the ego alone. This leveling off showed that less long-term information was present in the alters' pasts than within the ego's. Next we studied recency by the activity frequencies of alters and egos. Individuals who post frequently to social media, keeping up on current events, may provide more predictive information about either themselves or their social ties than other, infrequent posters. We found that the self-predictability of users was actually independent of activity frequency (SM) but there were strong associations between activity frequency and social information flow: egos who posted 8 times per day on average were ≈17% more predictable given their alters than egos who posted once per day on average (Fig. 2B) . Interestingly, this trend reversed itself when considering the activity frequencies of the alters: alters who posted 8 times per day on average were ≈23% less predictive of their egos than alters who posted once per day on average. Highly active alters tended to inhibit information flow, perhaps due to covering too many topics of low relevance to the ego. Information flow reflects the social network and social interaction patterns (Fig. 3) . We measured information flow for egos with more popular alters compared with egos with less popular alters. Alters with more social ties provided less predictive information about their egos than alters with fewer ties (Fig. 3A) . The decrease in predictability of the ego was especially strong up to alters with ≈400 ties, where the bulk of our data lies, but the trend continued beyond this as well. This decreasing trend belies the power of hubs in many ways: while hubs strongly connect a social network topologically [24] , limited time and divided attention across their social ties bounds the alter's ability to participate in information dynamics mediated by the social network and this is reflected in the predictability. Reciprocated contact is an important indicator of social relationships [25] , especially in online social activity where so much communication is potentially one-sided [23] . In Fig. 3B we investigated how directionality in contact volume, how often the ego mentions the alter and vice versa, related to information flow. We found that the ego was more predictable given the alter for those dyads where the alter more frequently contacted the ego, but there was little change across dyads when the ego mentioned the alter more or less frequently (Fig. 3B) . We also observed a similar trend for information flow but in reverse, when predicting the alter given the ego (SM). These trends captured the reciprocity of information flow: an alter frequently contacting an ego will tend to give predictive information about the ego, but the converse is not 8 true: an ego can frequently contact her alter but that does not necessarily mean that the alter will be any more predictive, as evidenced by the flat trend in Fig. 3B . In summary, the ability to repeatedly and accurately predict the text of individuals provides considerable value to the providers of social media, allowing them to develop profiles to identify and track individuals [26, 27] and even manipulate information exposure [28] . That information is so strongly embedded socially The time-ordered cross-entropy (Fig. 1A) applied here to online social activity is a natural, principled information-theoretic measure that incorporates all the available textual and temporal information. While weaker than full causal entailment, by incorporating time ordering we identify social information flow as the presence of useful, predictive information in the past of one's social tie beyond that of the information in one's own past. Doing so closely connects this measure with Granger causality and other strong approaches to information flow [29, 20] . Acknowledgments We gratefully acknowledge the resources provided by the Vermont Advanced Computing
In it, Bagrow et al. REF estimated how much knowing the contents of the tweets of a person's friends helps in predicting the contents of this person's tweets.
Information flow reveals prediction limits in online social activity
Nature Human Behaviour 3 (2019) 122-128
Computer Science Medicine Physics