Paper: Fractal Patterns May Unravel the Intelligence in Next-Token Prediction (ICLR-2024)

Reviewer comment (Evaluation):
With regards to the supposed value of Figure 1 — a single (tripartite) anecdote showing self-similarity would not nearly be enough to show a trend, but this graph doesn’t even show unambiguous similarity — the mere presence of ‘burstiness’ is not sufficient evidence. Moreover, the Hǒlder exponent of 0.638 is claimed as sufficient to confirm self-similarity but no context is provided to contextualize this single statistic.

Author response:
Addressing your concern about the value of Figure 1, we would like to emphasize that the purpose of this figure is to visually demonstrate what a self-similar process looks like. We never used a figure to prove self-similarity. 
The correct and well-established approach in the (extensive) literature is to show that the auto-correlecation function decays algebraically (i.e. with a power law), giving rise to the Hǒlder exponent. This is precisely what we establish! The fact that a power law holds is shown in Figure 3 (top).

We elaborate on this in detail in Section 2, where we discuss the nature of self-similarity in language​​. The burstiness observed in Figure 1 is an intuitive and useful representation of self-similarity. Additionally, we provide extensive discussion on the significance and context of the Hǒlder exponent, particularly in Section 2.2, where its calculation and relevance to language are explained in detail​​.

Reviewer comment (Theory):
The relationship between the predictability of the form that would emerge from fractal structures and those that have been approached by Zipf, Shannon, and their respective camps should be explored. I.e., despite the brief mentions of Zipfian distributions and entropy, they are not contrasted at all with the theory behind the current work; it is not clear if the authors mean to present their line of work as an alternative to well-established methods, but any deeper comparison is advised.

Author response:
Your suggestion to explore the connection between fractal structures in language and the works of Zipf and Shannon is discussed in Section 4. As we mention there, Zipf’s law and Shannon’s method of estimating the entropy of English only use first order statistics so they cannot be used to answer questions about long-range dependence or self-similarity. If there is anything missing that needs to be clarified, please let us know and we would be happy to elaborate.

Reviewer comment (Evaluation):
Attempting to draw links between language and, say, phenomena of large bodies of water is very tenuous when the only connection is that they each share a Hurst parameter of 0.75. The authors are recommended to establish more concrete outcomes in their empirical results and conclusions, relevant to either understanding the learning process, improving outcomes, or interpreting outcomes of LLMs.

Author response:
We only mention that the Hurst parameter of 0.75 is quite common in natural phenomena and the fact that it also holds for language is interesting. Regarding the relation to learning processes, we have explored this in Section 3 in which we show a connection between scaling law parameters and fractal dimensions. For example, we find that when the Hǒlder exponent and the Hurst parameter are large in a particular domain, language models would benefit from having longer contexts during inference. We also studied the connection to the context length at training time but with no conclusive results, as we discussed in Section 3 using the Big Bench benchmark. There are many other similar questions of how fractal dimensions relate to learning and we believe we are only scratching the surface here.

Reviewer comment (Theory):
By means of the $\tau$-increments, are you suggesting there are fixed, expected lengths for clauses, sentences, paragraphs (and then presumably…sections? Chapters? Articles/novels/scripts/…?).

Author response:
The lengths are random variables. Since the process is stationary, those lengths have definite expected values, but the value of course varies from clauses to sentences and sections. How the expected length changes will determine the level of self-similarity. These are standard definitions as we state in the paper and we provide references for them. We do not invent them.

Reviewer comment (Presentation):
Could you please check your references for completeness (e.g., Aref, 1998; Bubeck et al, 2023)?

Author response:
Thank you. We will fix the missing fields in those references.

Reviewer comment (Theory):
The identity of tokens is ignored in the analysis and the representation is just a sequence of negative logprobs. While this is still indicative of some macro features of language, this analysis is limited to an impoverished representation of language.

Author response:
Addressing your first concern regarding the representation of language as a sequence of negative log probabilities (i.e. bits), we acknowledge that this approach may simplify the rich and complex nature of language. As you mention, it is unclear how the approach can be extended to take the identities of tokens into account. However, the approach can be interpreted as a lower bound to the complexity of natural language. In particular, long-range dependence on the sequence of bits implies the existence of long-range dependence on the sequence of tokens (by the data processing inequality).

Reviewer comment (Novelty):
While this is a fresh perspective, the main finding that next-token prediction is a difficult task because language tends to show burstiness at varying granularities and exhibits long-range dependence is not surprising or novel. For example, the hierarchical and recursive nature of language is an actively studied area of research in cognitive sciences and linguistics.

Author response:
Regarding the concern that our findings might not be perceived as surprising or novel, we appreciate this viewpoint. While the hierarchical and recursive nature of language is a subject of study in cognitive sciences and linguistics, our work aims to bridge these concepts with the computational perspective of language models, focusing on how these inherent properties of language influence model performance. This interdisciplinary approach, we hope, adds value to the existing body of research by providing a fresh perspective to understand LLMs.

Reviewer comment (Theory):
“Self-similarity” and “fractalness” are not identical concepts. The findings in fact suggest that language is not necessarily fractal which makes predictability challenging. This is acknowledged in the paper but yet the two concepts seem to have been used interchangeably in the discussion.

Author response:
We thank you for emphasizing on the distinction between “self-similarity” and "fractalness." It is not our intention to use these terms interchangeably, and we have acknowledged that they are different concepts in the paper. We will take care to avoid this in the revised version.

Reviewer comment (Theory):
The connection between fractal structure and capabilities of LLMs is suggestive but not conclusively proven. The arguments linking self-similarity to modeling patterns across granularities seem reasonable but remain speculative without more analysis.

Author response:
Thank you for your thoughtful and constructive feedback on our manuscript. We are grateful for your recognition of the novelty and technical soundness of our approach and its potential impact in advancing the conceptual understanding of large language models (LLMs).

Addressing your concern about the connection between fractal structure and the capabilities of LLMs, we acknowledge that our current work is an initial step in this direction. As you suggested, extending the analysis to other modalities and models would offer interesting insights, and finding further connections between fractal dimensions and learning could uncover other results that are useful for practitioners. We plan to explore these directions in the future.

In our work, we propose a precise mathematical formalism (using fractals and self-similarity) to quantify properties of language that may have been known intuitively but not formally shown, such as long-range dependence, and connected those to properties of learning. We hope this work lays the foundation for further explorations, highlighting the potential of fractal analysis as a lens for understanding and improving LLMs. We believe that additional research in this direction could provide substantial value to the field and enhance the applicability of our findings.

If there are any other questions or concerns, we would appreciate it if you let us know so we can respond to them during the discussion and improve the manuscript. 

Thank you again for your constructive critique.