Paper: Fusion Token: Enhancing Compression and Efficiency in Language Model Tokenization (ICLR-2024)

Reviewer comment (Reproducibility):
The details of the tokenization process are not entirely clear. For standard subword tokenization methods such as BPE or SentencePiece, tokens separated by whitespace are not merged. For example, in the sentence "I go to the park every day.", these methods would not merge "go to" or "every day" into a single token, even if these phrases occur frequently. It is unclear if the method proposed in this paper follows this same principle. If it does, there are serious doubts about the claim that adding 1k vocabulary can outperform 1M vocabulary, as this seems unlikely based on understanding of tokenization. There is a suspicion that the proposed method might merge tokens separated by spaces (e.g., "go_to" or "every_day"), which would make the comparison with the original BPE method unfair since they operate under different settings. While this approach may be more reasonable for code data, standard tokenization could also be easily applied in this setting. If this is the case, more details and evaluations are needed to demonstrate the superiority of the method.

Author response:
All whitespace in all tokenizers presented in this work follows the standards set in other tokenizers by encoding every byte and replacing whitespace with special characters. As such all whitespace is treated as if it were any other character for the purpose of learning token encodings. Therefore both BPE and FusionToken have tokens that include both regular and whitespace characters.

Reviewer comment (Experiments):
The method seems to be only applicable to code data, as code contains many very frequent patterns (e.g., "int cnt = 0;"). If tokens beyond spaces are merged, examples like "int_cnt_=_0;" could be treated as a single token, which makes sense for code. However, in the Natural Language (NL) domain, this could lead to a significant reduction in performance improvement and potential semantic confusion due to the tokenization. This raises questions about the universality of the method and whether experiments on natural language datasets have been conducted to validate it.

Author response:
With respect to NL, we acknowledge that code data, being fairly regular, can have great benefits from FusionToken. This however does not mean NL would be immune from benefits. Even with NL comprising a small percentage of the FusionToken dataset we still see a compression improvement of 2.57% (Table 1). With a greater percentage of text we would expect that number to be higher still.

Reviewer comment (Experiments):
The improvement of this tokenization method on the code dataset is not substantial, with only about a 10% increase. Recent works using neural methods for context compression have achieved compression ratios of 2–4 times. The authors should discuss and compare their method with these techniques to demonstrate the value of FuseToken.

Author response:
We are not aware of any such neural compression paper that can improve compression by 2-4x as to our knowledge the state of the art in tokenization (TokenMonster) is only ~25% more compressive than baseline BPE (but at the cost of not being easily integrated into Language Model training)

Reviewer comment (Experiments):
My main concern with this paper is the performance of token fusion on natural language (NL) corpora. The authors have only conducted experiments using a code corpus, which is significantly different from NL. Code language typically consists of similar patterns, better structure, and limited vocabulary size, making token fusion more likely to be effective. However, NL content is more complex, with greater variability in content. Therefore, it is essential for the authors to conduct experiments on an NL corpus to demonstrate the effectiveness of their method. In my opinion, token fusion should have a greater impact on code than on language.

Author response:
While we agree the addition of NL tests would strengthen the work, we chose to focus on code because of its relative challenge in tokenization compression (as per Table 1 code compression is uniformly worse than NL compression) and for evaluation in a more robust fashion. 
As a follow up to this work we intend to extend the work to NL and extend our experimentation further to answer these questions

Reviewer comment (Experiments):
The paper would benefit from conducting additional ablation studies in the experiments. I am particularly curious about the rationale behind selecting only 1K tokens in the paper. It would be interesting to explore the outcomes by varying the number of tokens, such as selecting 500 tokens, 2K tokens, or 4K tokens. This would provide valuable insights into how the performance changes with different token selections.

Author response:
Similarly, while the addition of ablations around fusion token length would assuredly strengthen the work, the primary objective of this work is to present the algorithm and compare it robustly against BPE. While information on how to best select the optimal fusion token size is important, it is less so and thus was left out of this submission.

Reviewer comment (Experiments):
How to decide the fusion token for different corpus? Why you choose 1K in your paper?

Author response:
We use the union over all corpora to produce our fusion tokens
We chose 1K after performing ablations over the number of added tokens. These ablations however were not particularly notable and thus were not included in the full text

Reviewer comment (Evaluation):
The aforementioned bias is also observed in Table 1 and Figures 2(a) and 2(c). Tokens introduced via Fusion Token show a remarkable performance improvement in code part relative to text. Since the tokens were derived from code data, their impact in the experiments is profound.

Author response:
As both the BPE tokens and FusionTokens are both trained on code data the impact of any such bias is non-existant and the improvements come only from the overall efficiency of FusionTokens not from any introduced bias

Reviewer comment (Experiments):
Assessing the efficacy of the 1K tokens added by the proposed method necessitates a comparison against a tokenization with 1K tokens appended solely through bi-gram merging (existing BPE approach).

Author response:
We do indeed compare the 1K FusionTokens against BPE, in fact extending the BPE vocabulary up to 1M (all trained on the same exact code dataset that the 1K fusion tokens are drawn from) and show a substantial improvement over a BPE model with far larger vocabulary size

Reviewer comment (Experiments):
Q1: Of what are the actually added 1K tokens comprised?

Author response:
The 1K tokens, like the BPE tokens, comprise ngrams (in BPE’s case bigrams) computed over the code dataset

Reviewer comment (Experiments):
Only the code generation downstream task is evaluated.

Author response:
It is for these reasons that our model, primarily trained on code, was only evaluated on code generation tasks.

Reviewer comment (Experiments):
It appears that the proposed method is not specifically designed for programming languages but rather for general language modeling. If that's the case, why was a programming language chosen for testing?

Author response:
Programming languages are chosen due to their lower compression in traditional BPE based approaches and for the robustness in evaluation.