Paper: SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (ICLR-2024)

Reviewer comment (Experiments):
This paper primarily focuses on evaluating the zero-shot performance of SILO. Given that pretrained LLMs usually serve as base models for instruction-tuning or task fine-tuning, understanding the extent of SILO's influence on these processes is of significant importance. However, this is not explored in this paper.

Author response:
We agree instruction-tuning and task fine-tuning will be important to explore in future work. As described in our response to reviewer cDVV, we focus on LM perplexity and zero-shot text classification in the paper due to the following two reasons:
Prior work has shown that LM perplexity correlates well with downstream tasks across a large collection of tasks and all models [1, 2, 3].
Prior work has shown that adding a nonparametric datastore improves a range of tasks in both fine-tuning scenarios [4, 5] and instruction tuning [6] even when pretrained LMs are already trained on in-domain data. Gains are expected to be larger in SILO where pretrained LMs are not trained on in-domain data (this is the case in perplexity, as reported in Appendix D.2 & Figure 5).

* [1] Kaplan et al. "Scaling Laws for Neural Language Models". 2020. https://arxiv.org/abs/2001.08361 
* [2] Hernandez et al. Scaling Laws for Transfer. 2021. https://arxiv.org/abs/2102.01293 
* [3] Xia et al. “Training Trajectories of Language Models Across Scales”. ACL 2023. https://arxiv.org/abs/2212.09803 
* [4] Guu et al. 2020. “REALM: Retrieval-Augmented Language Model Pre-Training”. https://arxiv.org/abs/2002.08909  
* [5] Izcard et al. "Atlas: Few-shot Learning with Retrieval Augmented Language Models". 2022  https://arxiv.org/abs/2208.03299 
* [6] Lin et al. 2023. "RA-DIT: Retrieval-Augmented Dual Instruction Tuning" https://arxiv.org/abs/2310.01352

Reviewer comment (Evaluation):
In the evaluation of SILO, this paper primarily relies on PPL as a metric, although it does carry out experiments on text classification tasks. A more comprehensive assessment across various settings, a broader range of tasks, and in terms of multiple aspects (such as helpfulness and harmfulness) could offer a deeper and more insightful understanding of SILO's performance.

Author response:
We agree a larger-scale evaluation in downstream tasks will be valuable. Some evaluations the reviewer suggested necessitate instruction-tuning or RLHF (e.g., helpfulness and harmfulness), which we think is out of scope for this paper but can be a good future direction.

Reviewer comment (Reproducibility):
In the paper, the authors claim that "Pythia ... is trained with a similar amount of compute (data size and model parameters)." What are the exact computational costs when training SILO and Pythia?

Author response:
Based on the Pythia paper [1], both Pythia and SILO are trained with A100 GPUs, each with 40GB of memory. Pythia 1.4B (the model we used for comparison as SILO has 1.3B parameters) is trained for 7,120 GPU hours. The main SILO model (PDSW) is trained for 6,613 GPU hours. We will add this information in the next version of the paper.

[1] Biderman et al. "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling". 2023. https://arxiv.org/abs/2304.01373

Reviewer comment (Novelty):
Technological innovation is a slight drawback since the study primarily builds upon the kNN-LM framework by Khandelwal et al., 2020. However, this is somewhat mitigated by the strategic application of the technology to a pertinent issue, backed by thorough analysis and experimentation.

Author response:
As the reviewer mentioned, this work is among the first that provide technical mitigation of legal risk in LMs. The training recipe – or more broadly, the idea of separating the data for training and the data for a datastore in general – is novel and has not been explored in prior work, to the best of our knowledge. We believe the novelty of our paper lies in how the idea of separating the data can be used to mitigate legal risk–an area whose technical solutions have largely been underexplored, and empirical findings on how our model allows recovering performance close to the existing LMs. Future work may investigate new model architectures that further improve the performance and efficiency of our framework.

Reviewer comment (Evaluation):
As per Table 16, the runtime speed remains a notable challenge for kNN-LM. Future work should explore innovative technological solutions to address this limitation.

Author response:
We agree – generally, we think of SILO as a way to control tradeoffs between legal risk, runtime and performance.  We believe the connection between the legal risk of non-permissive training data and a nonparametric datastore, now drawn for the first time in our work, will motivate continued research on more efficient inference with such architectures.  We will include more discussion of runtime in the extra page in the final version of the paper, and move some relevant results from Appendix D.2 to the main paper.

Reviewer comment (Experiments):
When data is taken away during pretraining and is only added during inference, the first question is on which tasks this matters and has a larger impact, and on which tasks it does not matter. The suite of text-classification benchmarks is too simplistic to give any relevant insights into the effect this might have.

Author response:
We have evaluated our models on zero-shot classification tasks and perplexity. We agree that future work should evaluate our approach on a larger collection of tasks and see how consistent our findings are. Our focus on LM perplexity in the paper was based on the following two reasons:  
1. Prior work has shown that LM perplexity correlates well with downstream tasks (including zero-shot and few-shot in-context learning) across a large collection of tasks and all models [1, 2, 3].  So it’s a natural first step.  
2. Prior work has shown adding a nonparametric datastore improves performance on downstream tasks [4, 5, 6] even when pretrained LMs are already trained on in-domain data. Gains are expected to be larger in SILO where pretrained LMs are not trained on in-domain data (this is the case in perplexity, as reported in Appendix D.2 & Figure 5).  

Our text classification experiments confirm these findings.  

Furthermore, perplexity has been the primary evaluation criterion in many recent ICLR papers that innovated on language modeling [7, 8, 9, 10]. Nevertheless, we agree that conducting further downstream evaluations (especially with other classes  of models) is an exciting area for future work.  

* [1] Kaplan et al. "Scaling Laws for Neural Language Models". 2020. https://arxiv.org/abs/2001.08361   
* [2] Hernandez et al. Scaling Laws for Transfer. 2021. https://arxiv.org/abs/2102.01293   
* [3] Xia et al. “Training Trajectories of Language Models Across Scales”. ACL 2023. https://arxiv.org/abs/2212.09803   
* [4] Asai et al. “ACL 2023 Tutorial: Retrieval-based Language Models and Applications”. https://acl2023-retrieval-lm.github.io/   
* [5] Khandelwal et al. "Nearest Neighbor Machine Translation". ICLR 2021. https://arxiv.org/abs/2010.00710   
* [6] Jiang et al. "Active Retrieval Augmented Generation" EMNLP 2023. https://arxiv.org/abs/2305.06983   
* [7] https://openreview.net/forum?id=HklBjCEKvH ICLR 2020  
* [8] https://openreview.net/forum?id=nnU3IUMJmN ICLR 2022  
* [9] https://openreview.net/forum?id=R8sQPpGCv0 ICLR 2022  
* [10] https://openreview.net/forum?id=BS49l-B5Bql ICLR 2022

Reviewer comment (Evaluation):
The perplexity analysis is not too insightful, as we won't know how that will influence any of the benchmarks that might be relevant to a particular use-case. An increase in perplexity is expected under domain-shift and also adding the inference augmentation will reduce it, so it is not clear what the take-away for the reader should be. It is good to have it in the appendix, but it is already known that knn-LM and RIC work.

Author response:
We have evaluated our models on zero-shot classification tasks and perplexity. We agree that future work should evaluate our approach on a larger collection of tasks and see how consistent our findings are. Our focus on LM perplexity in the paper was based on the following two reasons:  
1. Prior work has shown that LM perplexity correlates well with downstream tasks (including zero-shot and few-shot in-context learning) across a large collection of tasks and all models [1, 2, 3].  So it’s a natural first step.  
2. Prior work has shown adding a nonparametric datastore improves performance on downstream tasks [4, 5, 6] even when pretrained LMs are already trained on in-domain data. Gains are expected to be larger in SILO where pretrained LMs are not trained on in-domain data (this is the case in perplexity, as reported in Appendix D.2 & Figure 5).  

Our text classification experiments confirm these findings.  

Furthermore, perplexity has been the primary evaluation criterion in many recent ICLR papers that innovated on language modeling [7, 8, 9, 10]. Nevertheless, we agree that conducting further downstream evaluations (especially with other classes  of models) is an exciting area for future work.  

* [1] Kaplan et al. "Scaling Laws for Neural Language Models". 2020. https://arxiv.org/abs/2001.08361   
* [2] Hernandez et al. Scaling Laws for Transfer. 2021. https://arxiv.org/abs/2102.01293   
* [3] Xia et al. “Training Trajectories of Language Models Across Scales”. ACL 2023. https://arxiv.org/abs/2212.09803   
* [4] Asai et al. “ACL 2023 Tutorial: Retrieval-based Language Models and Applications”. https://acl2023-retrieval-lm.github.io/   
* [5] Khandelwal et al. "Nearest Neighbor Machine Translation". ICLR 2021. https://arxiv.org/abs/2010.00710   
* [6] Jiang et al. "Active Retrieval Augmented Generation" EMNLP 2023. https://arxiv.org/abs/2305.06983   
* [7] https://openreview.net/forum?id=HklBjCEKvH ICLR 2020  
* [8] https://openreview.net/forum?id=nnU3IUMJmN ICLR 2022  
* [9] https://openreview.net/forum?id=R8sQPpGCv0 ICLR 2022  
* [10] https://openreview.net/forum?id=BS49l-B5Bql ICLR 2022

Reviewer comment (Presentation):
As a strong paper, I would have liked to see the potential advantages contrasted more with the current challenges. The paper leaves the run-time and various adjustments to make the approach feasible in the appendix and focuses the main paper on selling the approach as a potential solution, yet realistically it is far from that.

Author response:
Yes, in general, we think of SILO as a way to control tradeoffs between legal risk, runtime and performance.  We believe the connection between the legal risk of non-permissive training data and a nonparametric datastore, now drawn for the first time in our work, will motivate continued research on more efficient inference with such architectures.  We will include more discussion of runtime in the extra page in the final version of the paper, and move some relevant results from Appendix D.2 to the main paper.

Reviewer comment (Experiments):
It did not become clear why the paper focuses on the different licences PD, SW and BY, as it does not have any significance in the experiments.

Author response:
There are two reasons:  
1. We think OLC, the dataset we introduce, should inherently have varying levels of permissiveness, as what constitutes permissive or not is not a binary decision but rather a spectrum. We want to allow LM developers to draw their own boundaries and control the trade-off between the size of the training data, legal risk, and performance.  
2. We demonstrated that the general empirical findings hold across different possible boundaries (Appendix D.2 & Figure 3).  

We have revised the paper to make these justifications clearer.