Repository: open-compass/opencompass
Issue: [Bug] LLM Compression evaluation results do not match official results
Root cause: Algorithmic Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

Original author to provide environment info later

### Reproduces the problem - code/configuration sample

```bash
CUDA_VISIBLE_DEVICES=1,2,3,4,5,6 python run.py --datasets llm_compression --hf-path huggyllama/llama-7b --model-kwargs use_flash_attention_2=True cache_dir="/ssddata/yuzhen/model_hub" device_map='auto' --tokenizer-kwargs cache_dir="/ssddata/yuzhen/model_hub" --batch-size 20 --num-gpus 1
```

### Reproduces the problem - command or script

```bash
CUDA_VISIBLE_DEVICES=1,2,3,4,5,6 python run.py --datasets llm_compression --hf-path huggyllama/llama-7b --model-kwargs use_flash_attention_2=True cache_dir="/ssddata/yuzhen/model_hub" device_map='auto' --tokenizer-kwargs cache_dir="/ssddata/yuzhen/model_hub" --batch-size 20 --num-gpus 1
```

### Reproduces the problem - error message

No error messages

### Other information

Comment from original author: https://github.com/hkust-nlp/llm-compression-intelligence/issues/7#issuecomment-2086242889

Main points:
- Dataset not found in the provided zip file
- OpenCompass eval results differ from official results
- Suggestions for improving LLM compression evaluation efficiency

Please try to run the evaluation with the provided demo config file like the following code:

```bash
conda activate opencompass

export HF_EVALUATE_OFFLINE=1
export HF_DATASETS_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export TRANSFORMERS_CACHE=/ssddata/yuzhen/model_hub

python run.py ./configs/eval_llm_compression.py --reuse latest
```

A few things to note:
1. We release dataset updates on a regular basis. Some newer datasets like this one do not make it in time to a release. In the meantime, please use the script provided in the [dataset README](https://github.com/open-compass/opencompass/blob/main/configs/datasets/llm_compression/README.md) to download the corpora for testing.
2. Due to the sliding window approach used to calculate the BPC, dataset samples must be read sequentially. This means that we cannot use the `SizePartitioner` to partition tasks by samples, as doing so would produce unexpected results. Currently, only the `NaivePartitioner` (which partitions tasks by model-dataset combination) is supported for llm_compression. In theory, we could partition by sliding window instead of by original samples, which would enable the use of `SizePartitioner`, but that would require a more complicated implementation. 
3. We haven't done a thorough benchmark on the evaluation efficiency, but even with `NaivePartitioner`, increasing `batch_size` and `num_gpus` should improve the overall runtime (you need more gpu ram for bigger batch sizes). It is not clear to me the relationship between `batch_size` and the precision of your final BPCs, but the final results of our testing with `batch_size=8` are fairly close to those on your [main page](https://github.com/hkust-nlp/llm-compression-intelligence), see [dataset README](https://github.com/open-compass/opencompass/blob/main/configs/datasets/llm_compression/README.md)
4. You can modify the model list, batch_size, number of gpus and more in the config file [eval_llm_compression.py](https://github.com/open-compass/opencompass/blob/main/configs/eval_llm_compression.py) to perform LLM compression evaluation to your liking.
5. Please don't hesitate to let us know if you have any other questions.

### With [OpenCompass](https://github.com/open-compass/opencompass/blob/main/configs/datasets/llm_compression/README.md):
```
   metric version            model commoncraw  python arxiv_math  average
0     bpc  af04af   qwen1.5-32b-hf     0.5910  0.2584     0.4080   0.4191
1     bpc  af04af   qwen1.5-14b-hf     0.6459  0.2766     0.4310   0.4512
2     bpc  af04af      qwen-14b-hf     0.6197  0.2849     0.4498   0.4515
3     bpc  af04af     llama-30b-hf     0.5773  0.3212     0.4562   0.4516
4     bpc  af04af   llama-2-13b-hf     0.5807  0.3336     0.4752   0.4632
5     bpc  af04af    qwen1.5-7b-hf     0.6658  0.2935     0.4500   0.4698
6     bpc  af04af       qwen-7b-hf     0.6453  0.3088     0.4830   0.4790
7     bpc  af04af     llama-13b-hf     0.6083  0.3555     0.4865   0.4834
8     bpc  af04af    llama-2-7b-hf     0.6117  0.3536     0.4995   0.4883
9     bpc  af04af      llama-7b-hf     0.6285  0.3794     0.5096   0.5058
```

### [llm-compression-intelligence](https://github.com/hkust-nlp/llm-compression-intelligence) official results:
![image](https://github.com/hkust-nlp/llm-compression-intelligence/assets/21337801/301195c8-1dd9-4f15-9dea-0d2057c5d345)

Great works, please update the documentation with adding a new page, `Scaling Law Calculation`

> Great works, please update the documentation with adding a new page, `Scaling Law Calculation`

No problem, I will add it to `docs/advanced_guides/`