Repository: EleutherAI/lm-evaluation-harness
Issue: Logarithm-likelihood results are identical between fine-tuned and original LLMs
Root cause: Validation Gap

Hello! First, I just wanted to thank the great team who has made this powerful tool available for community use. However, I'm running into an issue is using it on fine-tuned models.

I'm trying to run LLM evaluation on a custom dataset (which I've called `nbc`). For this evaluation, I've made a custom test file which essentially measures the log-likelihood that the labeled response is generated by the model. Relevant code (based heavily off of the files used to evaluate other datasets) is below in case that is the issue:

```python3
def construct_requests(self, doc, ctx):
    lls = []
    lls.append(rf.loglikelihood("", ctx + " " + doc["answer"])[0])
    return lls

def process_results(self, doc, results):
    return {"raw": results[0]}

def aggregation(self):
    return {"raw": mean}

def higher_is_better(self):
    return {"raw": True}
```

I'm finding that running the program on fine-tuned and non-fine-tuned models produces the exact same results (the exact same average log likelihood). The fine-tuned models have been fine-tuned using QLoRA via HuggingFace. For example, these commands produce the exact same output value of `-508.6010886`:

```
python3 main.py --model hf-causal-experimental --model_args pretrained=tiiuae/falcon-7b,trust_remote_code=True--tasks nbc --num_fewshot 0
```
```
python3 main.py --model hf-causal-experimental --model_args pretrained=tiiuae/falcon-7b,trust_remote_code=True,peft="../final/falcon-7b-finetuned/" --tasks nbc --num_fewshot 0
```

Am I using the program wrong? Have I defined the evaluation code incorrectly? Does the log-likelihood metric even work on LoRA-fine-tuned LLMs?

Any help would be appreciated!

Hi, thanks for raising this issue! This does not seem like expected behavior.

Could you possibly try the following, so we can narrow this down?
- upload the LoRA to the HF hub (this can be done privately on your account) and load it from the hub rather than locally, and see if things change
- print `self.model` here to check that the PEFT was supposedly applied to your model : https://github.com/EleutherAI/lm-evaluation-harness/blob/3326c547a733d598b4377e54be96e194861b964c/lm_eval/models/huggingface.py#L256

Thanks for the suggestions! I've added that `print` statement and it seems to confirm that the PEFT was correctly applied:

```python3
PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): RWForCausalLM(
      (transformer): RWModel(
        (word_embeddings): Embedding(65024, 4544)
        (h): ModuleList(
          (0-31): 32 x DecoderLayer(
            (input_layernorm): LayerNorm((4544,), eps=1e-05, elementwise_affine=True)
            (self_attention): Attention(
              (maybe_rotary): RotaryEmbedding()
              (query_key_value): Linear(
                in_features=4544, out_features=4672, bias=False
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4544, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=4672, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
              )
              (dense): Linear(in_features=4544, out_features=4544, bias=False)
              (attention_dropout): Dropout(p=0.0, inplace=False)
            )
            (mlp): MLP(
              (dense_h_to_4h): Linear(in_features=4544, out_features=18176, bias=False)
              (act): GELU(approximate='none')
              (dense_4h_to_h): Linear(in_features=18176, out_features=4544, bias=False)
            )
          )
        )
        (ln_f): LayerNorm((4544,), eps=1e-05, elementwise_affine=True)
      )
      (lm_head): Linear(in_features=4544, out_features=65024, bias=False)
    )
  )
)
```

However, now testing on a different multiple choice task the results are still identical:

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/90410161/91c1a601-c74b-4f4a-b07e-93b61261481f)
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/90410161/63df6c68-f509-4fcc-af8e-874ba34abad2)

I guess this just means the fine-tuning has had no effect. I'll play around some more to try to see if I can get some different results. I can also try to load it from the hub, but I don't think that's the issue (but please let me know if that's still a possibility).

Thank you again!

What happens if you load this LoRA and your model, but outside of the lm-eval library, and then generate from it? do you get different results out? If so, then you likely should not be experiencing this behavior in lm-eval.

I think loading from the hub might be still worth trying! Another thing you could try is to merge your LoRA into the model weights, and load only that pretrained model for evaluation.

@ijoffe bumping this!

I am having the same issue when I load pretrained Flacon with peft as an model_args. Although the model loads the LoRA weights, but the results equal to the pretrained model.

@sasaadi Is there a public lora on the HF hub I can use to replicate this issue? If you point me to one, then I can test this issue.

Unable to replicate, on `big-refactor` branch, using 

`pretrained=tiiuae/falcon-7b,load_in_4bit=True,peft=ck46/falcon-7b-math-algebra-lora`

either when loading lora from hub or from a locally saved directory 

```
hf (pretrained=tiiuae/falcon-7b,load_in_4bit=True,peft=../falcon-7b-math-algebra-lora), limit: 10.0, num_fewshot: None, batch_size: 1
| Tasks  |Version|Filter| Metric |Value|   |Stderr|
|--------|-------|------|--------|----:|---|-----:|
|arc_easy|Yaml   |none  |acc     |  0.8|±  |0.1333|
|        |       |none  |acc_norm|  0.6|±  |0.1633|
```

```
hf (pretrained=tiiuae/falcon-7b,load_in_4bit=True), limit: 10.0, num_fewshot: None, batch_size: 1
| Tasks  |Version|Filter| Metric |Value|   |Stderr|
|--------|-------|------|--------|----:|---|-----:|
|arc_easy|Yaml   |none  |acc     |  0.7|±  |0.1528|
|        |       |none  |acc_norm|  0.3|±  |0.1528|
```

(also observing different perplexities with `lambada_openai` task, `--limit 10`)

~~I also tested on piqa, sciqa, and hellaswag and found no change.~~

@StellaAthena what branch was that testing on, and by "no change" do you mean that the scores were the same? on the whole dataset?

The actual loglikelihoods do appear to change, as demonstrated by `lambada_openai`, which wouldn't be incompatible with multiple choice performance staying the same (if loglikelihoods of answer choices change, but not their relative ranking). But if this was observed on multiple full datasets that the results were precisely the same, I can investigate further.

> @StellaAthena what branch was that testing on, and by "no change" do you mean that the scores were the same? on the whole dataset?
> 
> The actual loglikelihoods do appear to change, as demonstrated by `lambada_openai`, which wouldn't be incompatible with multiple choice performance staying the same (if loglikelihoods of answer choices change, but not their relative ranking). But if this was observed on multiple full datasets that the results were precisely the same, I can investigate further.

No, I commented on the wrong issue. I thought this was the one about batch size.