Paper: Language Models as Semantic Indexers (ICLR-2024)

Reviewer comment (Evaluation):
We have not seen a full section to analyze the semantic IDs by demonstrating all sorts of key examples. As a rising domain, many readers may wonder what the semantic IDs exactly look like and how they benefit the downstream tasks.

Author response:
**What do semantic IDs look like?** We thank the reviewer for raising this comment. Accordingly, we add a semantic ID study section in Appendix A.7 to show what the IDs look like and what documents are assigned to those IDs. From the result, we can find that documents/products sharing similar IDs are semantically similar to each other. In addition, the first c^1 captures coarse-grained functions (e.g., Bath products) while the second c^2 will dig into more fine-grained semantics (e.g., Soaps, Shampoo).

Reviewer comment (Experiments):
In your experiments, could you provide some human annotator evaluation for the semantics of your generated IDs? This is a key experiment, though we have seen enough assessment.

Author response:
**Human evaluation.** We agree with the reviewer that a human evaluation can make the experiment on ID quality more comprehensive. We add a human evaluation by 1) Randomly selecting product pairs (20 pairs for each method) in the Amazon-sports dataset which share the first two IDs c^{<2}=c^1c^2. 2) Ask four trained annotators to evaluate if the two products in each pair are semantically related to each other. 3) Calculate the accuracy of each method. The results are shown below:

| Model              | Accuracy |
|--------------------|--------|
| rq-VAE indexer     |  0.7375 | 
| HC Indexer         |  0.5375 | 
| Ours               | 0.7750 |

From the result, our LMIndexer can outperform baseline methods by large margins. The human evaluation form can be found at https://docs.google.com/forms/d/e/1FAIpQLSfwtA4C6IY-YF4soWBWOqKMtJOso33p7DdUJic1_tE8utv5ww/viewform.

Reviewer comment (Experiments):
Baselines in section 4.2 are weak since they are using an off-the-shelf text encoder and hence have no knowledge about the task; a very simple baseline that could be tried here is to train a dual-encoder model on this corpus and then cluster the embeddings.

Author response:
1. **Simple baseline in 4.2.** We thank the reviewer for proposing this baseline. According to your comment, we add experiments on Amazon datasets by first training a dual-encoder on the target corpus with contrastive learning [1] and adopting hierarchical clustering or rq-VAE to obtain semantic IDs based on the learned embeddings, denoted as rq-vae-contrastive and HC-contrastive, respectively. The results for ID quantitative study are shown below (can also be found in Appendix A.11):

| Model              | Beauty | Sports | Toys   |
|--------------------|--------|--------|--------|
| rq-VAE-bert        | 0.2654 | 0.2774 | 0.3154 |
| HC-bert            | 0.2428 | 0.2387 | 0.2729 |
| rq-VAE-contrastive | 0.3100 | 0.2695 | 0.3126 |
| HC-contrastive     | 0.2771 | 0.2622 | 0.2968 |
| Ours               | 0.3563 | 0.4163 | 0.3536 |

From the result, we can find that the semantic IDs generated by embeddings obtained after in-domain contrastive learning are better than those generated by embeddings from off-the-shelf text encoders. However, LMIndexer can outperform both baselines, which demonstrates its effectiveness in learning semantic IDs with self-supervision.

Reviewer comment (Experiments):
The learned semantic ID lengths seem to be very small (1-3), is this because of training instability when scaling to larger ID lengths? Do the other generative baselines also use the same ID lengths?

Author response:
4. **Semantic ID length.**  We illustrate from two perspectives why we set T to be 3. 1) *Performance*. As shown in Figure 5,  as T increases, the performance increase speed becomes smaller. This means that the gain of making T larger is smaller. Since a larger T will cost more time in training and inference, we adopt T=3 in our experiments. 2) *ID duplication*. As shown in Figure 7 in the Appendix, when we set T=3, the duplication issues of IDs are largely alleviated. It means that T=3 can make the ID for different documents distinguishable.

Reviewer comment (Experiments):
How does the performance gets affected when using a more powerful reconstructor? Perhaps an ablation on the number of layers in the reconstructor might be helpful here.

Author response:
2. **Ablation study on reconstructor.** We agree with the reviewer that adding such a study can make the experiments more comprehensive. We try reconstructors with 2 layers and 3 layers in Amazon-beauty dataset and the results are shown below (can also be found in Appendix A.12):

| Model              | ID AMI |   Recommendation (R@5)  | Product Search (NDCG@5)   |
|--------------------|--------|--------|--------|
| Ours (recon-1)     | 0.3563 | 0.0415 | 0.3187 |
| recon-2            | 0.2390 | 0.0284 | 0.2528 |
| recon-3            | 0.1679 | 0.0281 | 0.2522 |

From the result, we can find that as the reconstructor layer increases (the reconstructor becomes more powerful), the quality of the semantic indexer and its generated semantic IDs decreases. This is because more knowledge is learned inside the reconstructor rather than in the semantic indexer during self-supervised learning, which results in ID quality degeneration.

Reviewer comment (Reproducibility):
Is the contrastive loss $\mathcal{L}_{\text{contrastive}}$ computed over all documents?

Author response:
6. **Contrastive loss.** The contrastive loss is designed to promote distinction on c^t between documents that previously shared the same c^{<t}, enabling the model to discern finer-grained hierarchical relationships between documents and alleviate ID duplication issues. The contrastive loss is computed over all documents. For each document, we serve other documents that share c^{<t} with it to be negative samples when learning c^t.

Reviewer comment (Theory):
The paper says that the proposed method "learns the document’s discrete semantic embeddings and its hierarchical structure simultaneously". But it is not clear what the authors mean by the hierarchical structure of a document, how the proposed method is guaranteed to learn such a structure, and whether the proposed method actually learns such a structure.

Author response:
1. **Hierarchical structure.** 1) *Meaning*: Learning the hierarchical structure of documents refers to understanding documents from different semantic granularity and reflecting them in their semantic IDs (The preliminary set of IDs should predominantly encapsulate coarse-grained semantics, with successive IDs delving deeper into nuanced specifics). A more detailed illustration is added in Appendix A.7. 2) *Techniques*: The LMIndexer is proposed to learn the semantics into the IDs with self-supervised sequential discrete auto-reconstruction and capture the different granularity of semantics into their IDs with progressive training and contrastive objective. 3) *Results*: We show the learned hierarchy in Figure 9 and Figure 10 in the Appendix, where we can find that the products sharing the same first ID have a similar coarse-grained function (e.g., Bath products), while products sharing the first and second ID have related fine-grained functions (e.g., Soaps, Shampoo).

Reviewer comment (Experiments):
The size of the semantic ID (T) is set to less than or equal to 3 in the experiments, which is surprisingly small. Figure 5 shows the performance increases with T. Why not trying bigger T values? Would an ID with length 3 be informative enough?

Author response:
2. **Size of the semantic ID (T).** We illustrate from two perspectives why we set T to be 3. 1) Performance. As shown in Figure 5, the performance increases speed as T increase becomes smaller. This means that the gain of making T larger is smaller. Since a larger T will cost more time in training and inference, we adopt T=3 in our experiments. 2) ID duplication. As shown in Figure 7 in the Appendix, when we set T=3, the duplication issues of IDs are largely alleviated. It means that T=3 can make the ID for different documents distinguishable.

Reviewer comment (Experiments):
How should the codebook size be determined? It seems that only three sizes are tried in the experiments. How does the size of the codebook affect the performance?

Author response:
3. **Codebook size.** The codebook size is set as a hyperparameter in our model design. We agree with the reviewer that showing the study of codebook size can make our research more comprehensive. The results are shown below (added in Appendix A.8):

| Model              |   Recommendation (R@5)  |   Product Search (NDCG@5)   |
|--------------------|--------|--------|
| 128            | 0.0291 | 0.3071 |
| 256            | 0.0355 | 0.3089 |
| Ours (512)     | 0.0415 | 0.3187 |

From the result, we can find that the downstream task performance generally increases as codebook size increases. It is intuitive since the larger the codebooks are (before overfitting), the more information they can represent.

Reviewer comment (Evaluation):
The performance metric used in ID quantitative study, AMI (in Table 1), is not defined or explained.

Author response:
4. **Definition of AMI.** The Adjusted Mutual Information (AMI) score is a measure used in statistics and information theory to quantify the agreement between two clusters (in our experiments, the two clusters refer to ground truth category clusters and Semantic ID clusters) while correcting for chance. It is an adjustment of the Mutual Information (MI) score that accounts for the fact that MI is generally higher for clusters with a larger number of clusters, thus providing a normalized score that is more comparable across different clusters. We have added the definition of AMI in Appendix A.9 according to the reviewer's suggestion.

Reviewer comment (Presentation):
Not clear what the word clouds picture (Figure 3) is trying to show? The text explaining it is confusing. What do you mean by “two semantic ID prefixes”? Are the two prefixes from the same generated ID or two different IDs?

Author response:
5. **Word clouds.** The word clouds are shown for the semantic ID qualitative study. “Two semantic ID prefixes” refers to two different c^{<=2}=c^1c^2 (e.g., ID (0,0,*) and ID (1,2,*)). We try to print out the word clouds to summarize what documents are sharing IDs (0,0,*) and what documents are sharing IDs (1,2,*) respectively. We can find that the two groups of documents are related to beach/sand set toys (corresponding to ID (0,0,*)), and star war toys (corresponding to ID (1,2,*)) respectively. This demonstrates that the IDs acquired through LMIndexer convey a higher degree of semantic relevance (enabling similar documents to have similar semantic IDs). We have added a more comprehensive ID qualitative study section in Appendix A.7.

Reviewer comment (Evaluation):
It would be better if authors provided other performance metrics such as latency for the retrieval in comparison with baselines.

Author response:
6. **Latency analysis.** We conduct latency analysis on baseline methods and LMIndex on Amazon-beauty dataset. We measure the total latency of product search on the whole Amazon-beatuy test set. The results are shown below (details can be found in Appendix A.10):

| Model              | Beauty |
|--------------------|--------|
| rq-VAE indexer     |  13.66s | 
| HC indexer         |  12.85s | 
| SEAL               |  21min |
| Ours               | 12.21s |

From the result, the inference latency of our method is comparable with the rq-VAE indexer and HC indexer and is much smaller than SEAL.

Reviewer comment (Novelty):
While this paper presents an approach termed LMINDEXER, it's important to note that the novelty of the method is somewhat limited. Additionally, the paper lacks a comprehensive discussion of related work, including notable prior efforts that have explored the use of encoders for text encoding and decoders for reconstruction in the context of information retrieval. Several works, such as [1], [2], and [3], have examined similar techniques and deserve acknowledgment for their contributions to the field.

[1] Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder (EMNLP 2021)  
[2] A Contrastive Pre-training Approach to Learn Discriminative Autoencoder for Dense Retrieval (CIKM 2022)  
[3] RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder (EMNLP 2022)

Author response:
1. **More related works.** We thank the reviewer for pointing out the related papers and we are happy to add them to the related work section (added to the “Self-supervised Learning with Language Models” section). We would like to emphasize the novelty of LMIndexer compared with these works: 1) *Focus*: LMIndex mainly focuses on learning semantic IDs via self-supervised learning, which is different from existing works’ focus on pretraining a text encoder for dense retrieval. 2) *Techniques*: Since semantic IDs lie in a discrete and sequential fashion which is different from a soft embedding for dense retrieval, we propose a new sequential discrete auto-reconstruction pipeline and introduce solutions for challenges including Reconstructor collapse, Posterior collapse, and ID diversity. 3) *Task*: We explore LMIndex on product recommendation, product retrieval, and document retrieval and existing works mainly focus on document retrieval.

Reviewer comment (Experiments):
In the context of document retrieval, it would be beneficial to broaden the comparison to include more generative retrieval baselines. For instance, evaluating how the LMINDEXER approach compares to FM-index-based SEAL or other recent generative retrieval methods would provide a more comprehensive understanding of its performance. Focusing solely on comparisons with DSI, which may be considered a relatively weaker baseline, might not offer a complete picture of the method's capabilities.

Author response:
2. **Comparison with SEAL.** We thank the reviewer for pointing out the baseline. We add their results on Amazon product search datasets and NQ320k (code to reproduce is uploaded to https://anonymous.4open.science/r/ICLR24-submit-B2E7/):

| Model              | Beauty | Sports | Toys   |  NQ320k |  
|--------------------|--------|--------|--------|--------|
| rq-VAE indexer        |  0.2710 |  0.2606 | 0.2511 |  0.6480  |
| HC indexer     |  0.2172 | 0.1979 |  0.2379 |   0.6439 |
| SEAL               | 0.1271 | 0.2011 | 0.1035 |   0.5698 |
| Ours               | 0.3187 |  0.2870 | 0.2865 |   0.6631|

We also add the experimental results in Appendix A.5 in our paper. From the result, we can find that our method outperforms SEAL significantly. The main reason is that after the self-supervised semantic ID learning, LMIndexer can generate higher quality semantic ID as identifiers for documents than Ngrams used in SEAL.

Reviewer comment (Experiments):
While the proposed self-supervised approach has the potential to address the "new document problem" by automatically acquiring semantic IDs, it would be valuable to see experimental results that explicitly address this issue. Incorporating experiments that involve adding new documents to the model and assessing its adaptability and performance in such scenarios would provide a more robust evaluation.

Author response:
3. **New document problem.** We conduct experiments to study this problem in Section 5.2 (Table 4). After the semantic indexer is finetuned on the downstream product search task, we add 100 new documents into the corpus. We use the semantic indexers before fine-tuning on the downstream task (including rq-VAE indexer, HC indexer, and LMIndexer) to give semantic IDs for new documents and test how the fine-tuned semantic indexers (on the downstream task) can generalize to the new documents. The results are shown in Table 4, where LMIndexer outperforms the baselines significantly. The reason is that LMIndexer is trained to achieve semantic IDs on a large corpus with self-supervision. This enables the LMIndexer to learn the projection between text semantics and semantic IDs, which can be generalized to new documents and applied to downstream tasks. We will make this part more clear in our revision.

Reviewer comment (Evaluation):
An important consideration when using automatically generated semantic IDs is the possibility of duplication. It is crucial to include a discussion or explanation of how the LMINDEXER approach handles or mitigates this potential issue. A detailed exploration of the approach's robustness in preventing or addressing duplication would enhance the paper's completeness and practicality.

Author response:
4. **ID Duplication.** This is a great point. We add a section to discuss this issue in Appendix A.6. We agree with the reviewer that the duplication issue is very important here. To alleviate this issue, we propose a contrastive objective in Section 3.2 to promote distinction between documents that previously shared the same learned ID and encourage them to obtain different IDs for the next position (alleviate duplication). The effectiveness of this design is shown in Figure 8 in Appendix. We can find that during self-supervised learning, if the contrastive objective is added, the difference ratio on the next ID position of documents sharing learned ID is bigger and the diversity (perplexity) of IDs on the next position is larger, which means that the duplication issue is alleviated. We also show in Figure 7 in the Appendix that the semantic IDs learned by LMIndexer are quite distinguishable. While it is nearly impossible to guarantee zero duplication since there can be documents that have very similar semantics, we simply add another final ID position to distinguish them.

Reviewer comment (Experiments):
The authors should consider expanding their related work section to include the most recent developments in generative Information Retrieval (IR) techniques. Additionally, they should include a comparative analysis of these recent works alongside the proposed LMINDEXER method in the experimental section. This would provide a more comprehensive overview of how LMINDEXER stacks up against the state-of-the-art in generative IR.

Author response:
2. **Comparison with SEAL.** We thank the reviewer for pointing out the baseline. We add their results on Amazon product search datasets and NQ320k (code to reproduce is uploaded to https://anonymous.4open.science/r/ICLR24-submit-B2E7/):

| Model              | Beauty | Sports | Toys   |  NQ320k |  
|--------------------|--------|--------|--------|--------|
| rq-VAE indexer        |  0.2710 |  0.2606 | 0.2511 |  0.6480  |
| HC indexer     |  0.2172 | 0.1979 |  0.2379 |   0.6439 |
| SEAL               | 0.1271 | 0.2011 | 0.1035 |   0.5698 |
| Ours               | 0.3187 |  0.2870 | 0.2865 |   0.6631|

We also add the experimental results in Appendix A.5 in our paper. From the result, we can find that our method outperforms SEAL significantly. The main reason is that after the self-supervised semantic ID learning, LMIndexer can generate higher quality semantic ID as identifiers for documents than Ngrams used in SEAL.

Reviewer comment (Experiments):
It is essential for the authors to address the issue of handling new documents within the LMIndexer framework. The paper should discuss how this framework manages the addition of new documents, what mechanisms or strategies are employed, and the performance of the LMIndexer approach in comparison to baseline methods when confronted with this "new document" scenario. This analysis would help assess the adaptability and robustness of the approach.

Author response:
1. **New Document.** 1) *Comparison with baselines*: We compare the performance of our method with baselines on the new document setting in Table 4, where LMIndexer outperforms baselines significantly. This demonstrates the advancement of LMIndexer for zero-shot retrieval compared with semantic ID baselines. 2) *Generalizing to new documents is not our focus*. We want to argue that zero-shot retrieval itself is a hard research problem [1]. We agree with the reviewer that it can be an interesting future direction for semantic IDs, but it is not the focus of this paper (this paper is mainly proposed for learning self-supervised semantic IDs).

Reviewer comment (Experiments):
To ensure that distinct semantic IDs are assigned to different documents, the paper should provide detailed explanations and discussions regarding the mechanisms and safeguards in place within the LMIndexer framework. Experimental results or case studies showcasing how the system maintains distinct semantic IDs for various documents would add substantial value to the paper, reinforcing its practicality and effectiveness in preventing semantic ID duplication.

Author response:
2. **ID Duplication.** 1) *Depending on corpus*. We would like to argue that the ID duplication problem is not the problem of our method, but originated from the nature of the corpus. In text corpus, there often exists documents that are nearly the same as each other, or just different in several words. For example, in the Amazon-toys dataset (which we show the duplication results for), there are items “Monster High Skull Shores Ghoulia”, “Monster High Skull Shores Abbey” and “​​Monster High Skull Shores Lagoona”. In these cases, it is extremely difficult to assign IDs only based on text semantics. 2) *Solution*. We solve this problem by simply adding a final ID position which is not learned based on semantics to distinguish the products sharing the same IDs. By doing this, the IDs will not only contain the text semantics of the original document/products but are also 100% distinguishable.