Repository: EleutherAI/lm-evaluation-harness
Issue: Implement GPT-3 style contamination study
Root cause: Unimplemented Feature Gap

In https://github.com/EleutherAI/lm-evaluation-harness/tree/master/scripts/clean_training_data, @researcher2 has implemented the 13-grams algorithm from the paper "Language Models are Few Shot Learners." However, the code doesn't perform the same analysis using that 13-gram algorithm as the paper. "Language Models are Few Shot Learners" subtracts the contamination from the _test set_, and then looks at the %-change in performance between the entire test set and the uncontaminated test set. This type of analysis is growing in popularity, as is evidenced by it being done in [Finetuned Language Models are Few Shot Learners](https://arxiv.org/abs/2109.01652) and asked for by reviewers of [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://openreview.net/forum?id=9Vrb9D0WI4).

Do we just need to:
1) Generate 13-grams on the Pile (or retrieve them if others like @researcher2 have already done this)
2) For each task on the harness, measure performance of a model (which model?) before/after cleaning (1)'s 13-gram contaminants from the task.

> Do we just need to:
> 
> 1. Generate 13-grams on the Pile (or retrieve them if others like @researcher2 have already done this)
> 2. For each task on the harness, measure performance of a model (which model?) before/after cleaning (1)'s 13-gram contaminants from the task.

Yes, with the caveat that I would like to offer this as a function that anyone can use. I am interested in using this on some models I am currently training on the Pile, and NVIDIA has asked about our plans to implement this functionality as well. NVIDIA’s data is based on the Pile, but is modified and they cannot release the dataset.

I’m thinking that we would provide a function for computing n-grams (default to 13, but can be set by the user) and then adding an argument pointing to the n-grams to the main eval harness call. When called with the n-grams input, it does the eval harness as usual except it also computes the deduplicated metrics in addition to the metrics on the full dataset.

@leogao2 do you have any plans to implement metrics as a class? This is a good excuse to do so and I could see it being useful to someone with non-standard evaluation criteria.

Found this via Bigscience from Stella. Getting into this discussion.
How the 13-gram search is generated for a dataset? I believe pre-calculating n-gram search support for a dataset is a costly task (Both memory and CPU compute-wise).
Here's some of the stuff that I have already explored,
1. Initially, I thought this would be a straightforward implementation of trie-tree but later found that trie-tree is a token-based data structure. I believe trie tree can be used to do a token level analysis of the corpus. It could be also worth exploring too.
2. Actual fast pattern matching algorithm is "suffix array", Are you guys using it? 
3. Another way could be the "minhash" algorithm but I don't know about the details of this one. I would love to know about the details if you are using it.  


Here are some of the content I have collected from Teven.
1. Suffix array implementation from google: https://github.com/google-research/deduplicate-text-datasets
2. Additional scripts for checking data contamination. https://github.com/TevenLeScao/library_of_babel/blob/main/prompt_deduplication/mass_count_occurences_per_word.py

> I’m thinking that we would provide a function for computing n-grams (default to 13, but can be set by the user) and then adding an argument pointing to the n-grams to the main eval harness call. When called with the n-grams input, it does the eval harness as usual except it also computes the deduplicated metrics in addition to the metrics on the full dataset.

This is actually a great point. Basically, we want to mimic the functionality of [GPT-3 Appendix Table C1 Page 45](https://arxiv.org/pdf/2005.14165.pdf)

> > I’m thinking that we would provide a function for computing n-grams (default to 13, but can be set by the user) and then adding an argument pointing to the n-grams to the main eval harness call. When called with the n-grams input, it does the eval harness as usual except it also computes the deduplicated metrics in addition to the metrics on the full dataset.

> This is actually a great point. Basically, we want to mimic the functionality of GPT-3 Appendix Table C1 Page 45

Yup! I was looking at that table as I wrote that :)

### Summary Of Current Work
We have a [script](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/scripts/clean_training_data/generate_13_grams.py) for generating 13 grams.

This accepts a --working-directory argument, which yields documents from all files found at f"{working_directory}/\*.jsonl.zst\*". That extension is [lm_dataformat](https://github.com/leogao2/lm_dataformat). It also accepts --n_value (defaults to 13-grams), and --buckets which specifies how many output buckets the ngrams are broken into. They are bucketed simply by hashing the ngram modulo num_buckets.

After generating and bucketing, the ngrams in each bucket are [sorted](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/scripts/clean_training_data/sort_13_gram_buckets.py).

Once we have sorted buckets it's very quick to count up to exclude n-grams that occur less than 10 times, the remainder is used as an exclude list when running Janitor.

Based on my understanding of **Initial training set filtering** in "Language Models are Few Shot Learners.", the process would go:

1. Generate 13grams/sort/deduplicate/save >10occurences for training set.
2. Run [Janitor](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/scripts/clean_training_data/janitor.py), registering the test contaminates that aren't in the list from step 1, than run the training set(s) through using Janitor.clean to return filtered training data.

### Discussion

With some minor changes the 13-gram script could be adapted to any required input file type, or just import it and override the get_pile function.

We have some artifacts of this whole process saved somewhere, @StellaAthena or @leogao2 might know? I remember copying stuff over to a hdd0 or something like that.

In terms of the paper, I don't believe we continued onto the following section **Overlap methodology**, @leogao2?

### Suggested Improvements

Stella has suggested we add the ability to provide your 13 grams from the training set which triggers a filtering of the test set to exclude any overlap with the provided 13 grams. The evaluation will produce results for filtered and unfiltered, though it shouldn't be necessary to do separate runs if we account for filtering during eval.

I may still keep the generation script separate from the main eval harness?



Sorry, I don't know where the 13-grams are saved. Also, we never implemented the overlap methodology, but it shouldn't be too hard to implement given the 13-gram stuff.

I'll take a crack at this over the coming week

I've implemented the suggested improvements above, waiting on server config to get access to the pile 13 grams for testing. At this stage we still need to do the overlap study from 6(C) of Language Models are Few-Shot Learners.