Repository: EleutherAI/lm-evaluation-harness
Issue: Benchmarking custom model gives off-putting results 
Root cause: Validation Gap

Hi!

My group is trying to benchmark a custom model using `lm-eval`. We have followed the instructions on how to register a model in `docs/model_guide.md`. We get the following results.

|    Tasks     |Version|Filter|n-shot|  Metric  |   Value   |   |  Stderr  |
|--------------|-------|------|-----:|----------|----------:|---|---------:|
|lambada_openai|Yaml   |none  |     0|perplexity|147532.4047|±  |20493.2460|
|              |       |none  |     0|acc       |     0.0600|±  |    0.0033|
|piqa          |Yaml   |none  |     0|acc       |     0.5316|±  |    0.0116|
|              |       |none  |     0|acc_norm  |     0.5076|±  |    0.0117|
|winogrande    |Yaml   |none  |     0|acc       |     0.5170|±  |    0.0140|
|wsc           |Yaml   |none  |     0|acc       |     0.3750|±  |    0.0477|
|arc_easy      |Yaml   |none  |     0|acc       |     0.3375|±  |    0.0097|
|              |       |none  |     0|acc_norm  |     0.3161|±  |    0.0095|
|arc_challenge |Yaml   |none  |     0|acc       |     0.1817|±  |    0.0113|
|              |       |none  |     0|acc_norm  |     0.2261|±  |    0.0122|
|sciq          |Yaml   |none  |     0|acc       |     0.4560|±  |    0.0158|
|              |       |none  |     0|acc_norm  |     0.3830|±  |    0.0154|
|logiqa        |Yaml   |none  |     0|acc       |     0.2074|±  |    0.0159|
|              |       |none  |     0|acc_norm  |     0.2350|±  |    0.0166|

Observing the perplexity on `lambada_openai` we ran a script to compute the perplexity scores ourselves using the dataset from `EleutherAI/lambada_openai`. However, on doing so we get a perplexity score of around 60. The following is our class that inherits `LM`.  Note the following two points:
- The loglikelihood function is mostly based on the implementation in `lm_eval/models/huggingface.py` for `_loglikelihood_tokens` and `loglikelihood`. 
- We also noticed that the functions `loglikelihood_rolling` and `generate_until` were not called when evaluating the model using the above datasets with `n-shot=0`. Therefore, in our class we don't implement them since we only want to evaluate our model on these datasets.

```
class HarnessModel(LM):

    _DEFAULT_MAX_LENGTH = 2048

    def __init__(self, model, tokenizer, max_length=None, type_=None) -> None:
        super().__init__()

        self.model = model.to(torch.float32)
        self.model.eval()
        self.tokenizer = tokenizer
        self.device = 'cuda' if torch.cuda.is_available() else 'cpu'
        self.tok2id = self.tokenizer.get_vocab()
        self.max_batch_size = 64
        self._max_length = max_length
        # self.eot_token_id = ...
        # self.rank = ...
        # self.world_size = ...
        self.batch_size = 8
        self.truncation = False
        self.batch_sizes = {}
        # self.max_gen_toks = ...
        self.batch_schedule = 1
        self.accelerator = Accelerator()
        self.model.to(self.device)
        self.type_ = type_

    @property
    def max_length(self):
        if self._max_length:  # if max length manually set, return it
            return self._max_length
        seqlen_config_attrs = ("n_positions", "max_position_embeddings", "n_ctx")
        for attr in seqlen_config_attrs:
            if hasattr(self.model.config, attr):
                return getattr(self.model.config, attr)
        if hasattr(self.tokenizer, "model_max_length"):
            if self.tokenizer.model_max_length == 1000000000000000019884624838656:
                return self._DEFAULT_MAX_LENGTH
            return self.tokenizer.model_max_length
        return self._DEFAULT_MAX_LENGTH
    
    @property
    def eot_token_id(self):
        # we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*
        return self.tokenizer.eos_token_id
    
    @property
    def max_gen_toks(self) -> int:
        return 256
    
    @property
    def rank(self):
        return self._rank

    @property
    def world_size(self):
        return self._world_size

    def loglikelihood(self, requests: list[Instance]) -> list[tuple[float, bool]]:
        print("The loglikelihood function is run.")
        
        new_reqs = []
        
        for context, continuation in [req.args for req in requests]:
            if context == "":
                # end of text as context
                context_enc, continuation_enc = [self.eot_token_id], self.tokenizer.encode(continuation)
            else:
                context_enc, continuation_enc = self.tokenizer.encode(context), self.tokenizer.encode(continuation)

            new_reqs.append(((context, continuation), context_enc, continuation_enc))

        results = [] # (logprobs, is_greedily_generated)

        pbar = tqdm(total=len(new_reqs), desc="loglikelihood")
        with torch.no_grad():
            for (context, continuation), context_enc, continuation_enc in new_reqs:

                """
                V: Vocab size
                C: Context length
                L: Continuation length
                """

                input_ids = (torch.tensor(context_enc + continuation_enc)
                            [:-1].unsqueeze(0).to(self.device)) # [1, C+L-1]
                mode_output = self.model(input_ids) # [1, C+L-1, V]
                
                logits = mode_output[:,len(context_enc)-1:, :].squeeze(0) # only get the logits for the continuation [L, V]
                labels = torch.tensor(continuation_enc).to(self.device) # [L]
                is_match = (logits.argmax(-1) == labels).all().item()
                # logprobs = F.cross_entropy(logits, labels) # [1]
                log_probs_per_target_token = F.log_softmax(logits, dim=-1) # [L, V]
                loglikelihood_val = torch.gather(log_probs_per_target_token, -1, labels.unsqueeze(0)).sum() # [1]
                

                results.append((loglikelihood_val.detach().cpu().item(), is_match))
                pbar.update(1)

        pbar.close()

        return results
    
    def generate_until():
        print("The generate until function is run.")
        pass

    def loglikelihood_rolling():
        print("The loglikelihood rolling function is run")
        pass
```

We evaluate by:

```
    lm_obj = HarnessModel(
        model=custom_model,
        tokenizer=tokenizer,
    )
    lm_eval.tasks.initialize_tasks()    
    results = lm_eval.simple_evaluate(
        model=lm_obj,
        tasks=["lambada_openai", "piqa", "winogrande", "wsc",
               "arc_easy", "arc_challenge", "sciq", "logiqa"],
        num_fewshot=0,
    )
```

Any help would be much appreciated.

I believe `labels.unsqueeze(0)` ([L] -> [1, L]) should be `labels.unsqueeze(-1)` -> ([L] -> [L, 1]) to appropriately gather along the vocab dimension with input [L, V]. You can also consider just subclassing `HFLM` and overriding the `_model_call` and/or `_model_generate` methods as `mamba` does it. Might be simpler :).

Hi! Thanks for the simplifying guideline!

I subclassed `HFLM` and overrode the `_model_call` method. For `lambada_openai` we still get a very high perplexity score. However, for the other ones, we do seem to get a reasonable score. **I was wondering if there is something obvious that we are missing in terms of the reasons for this discrepancy.** **Also, does incorrect results for one dataset mean the results are also incorrect?** 

The reason we say the other scores are reasonable is because our custom model is essentially the `pythia-70m` with the attention blocks being switched out for a custom block. The scores we get for just `pythia-70m` is also given below. We do expect the scores to be slightly worse as is the case for all the datasets except for `lambada_openai`.

OUR CUSTOM MODEL BENCHMARK RESULTS
|    Tasks     |Version|Filter|n-shot|  Metric  | Value  |   |Stderr |
|--------------|-------|------|-----:|----------|-------:|---|------:|
|arc_challenge |Yaml   |none  |     0|acc       |  0.1775|±  | 0.0112|
|              |       |none  |     0|acc_norm  |  0.2184|±  | 0.0121|
|arc_easy      |Yaml   |none  |     0|acc       |  0.3998|±  | 0.0101|
|              |       |none  |     0|acc_norm  |  0.3708|±  | 0.0099|
|lambada_openai|Yaml   |none  |     0|perplexity|923.3513|±  |41.1848|
|              |       |none  |     0|acc       |  0.0600|±  | 0.0033|
|logiqa        |Yaml   |none  |     0|acc       |  0.1966|±  | 0.0156|
|              |       |none  |     0|acc_norm  |  0.2212|±  | 0.0163|
|piqa          |Yaml   |none  |     0|acc       |  0.5832|±  | 0.0115|
|              |       |none  |     0|acc_norm  |  0.5838|±  | 0.0115|
|sciq          |Yaml   |none  |     0|acc       |  0.5910|±  | 0.0156|
|              |       |none  |     0|acc_norm  |  0.5370|±  | 0.0158|
|winogrande    |Yaml   |none  |     0|acc       |  0.5004|±  | 0.0141|
|wsc           |Yaml   |none  |     0|acc       |  0.3750|±  | 0.0477|

PYTHIA-70M BENCHMARK RESULTS
|    Tasks     |Version|Filter|n-shot|  Metric  | Value  |   |Stderr|
|--------------|-------|------|-----:|----------|-------:|---|-----:|
|arc_challenge |Yaml   |none  |     0|acc       |  0.1749|±  |0.0111|
|              |       |none  |     0|acc_norm  |  0.2184|±  |0.0121|
|arc_easy      |Yaml   |none  |     0|acc       |  0.3754|±  |0.0099|
|              |       |none  |     0|acc_norm  |  0.3523|±  |0.0098|
|lambada_openai|Yaml   |none  |     0|perplexity|130.9636|±  |5.5012|
|              |       |none  |     0|acc       |  0.2271|±  |0.0058|
|logiqa        |Yaml   |none  |     0|acc       |  0.2104|±  |0.0160|
|              |       |none  |     0|acc_norm  |  0.2780|±  |0.0176|
|piqa          |Yaml   |none  |     0|acc       |  0.5985|±  |0.0114|
|              |       |none  |     0|acc_norm  |  0.5919|±  |0.0115|
|sciq          |Yaml   |none  |     0|acc       |  0.6400|±  |0.0152|
|              |       |none  |     0|acc_norm  |  0.5640|±  |0.0157|
|winogrande    |Yaml   |none  |     0|acc       |  0.5296|±  |0.0140|
|wsc           |Yaml   |none  |     0|acc       |  0.3654|±  |0.0474|

> Also, does incorrect results for one dataset mean the results are also incorrect?

Definitely not, that just means the model doesn't do well for that specific task (as long as your implementation is correct!). The benchmarks are all pretty [imperfect](https://arxiv.org/abs/2401.00595) anyway, so it's best to take the results with a grain of salt.

> The reason we say the other scores are reasonable is because our custom model is essentially the pythia-70m with the attention blocks being switched out for a custom block.

The high perplexity could just be due to this? All your other tasks score the relative log probability, effectively ranking the likelihood of multiple completion options given a prompt and choosing the most probable one.  High perplexity suggests the actual loglikelihoods have changed for given token sequences but this doesn't necessarily have to compromise its comparative ranking ability in multiple-choice tasks, where relative, not absolute, logliklihood matters.

I guess as a sanity check, you could compare the generations of the vanilla `pythia` with your modified model to see if there is a noticeable decline in the quality of the generations (should be obvious if the perplexity decreased so much). You could also debug by comparing your original code with the subclassed one, if you haven't yet already, just to confirm this didn't introduce a new bug. Didn't you get a perplexity of 60 when you did the scoring manually? You can check if you're seeing a difference there as well with the unmodified pythia. Also should confirm if you are using fp32 as in #1269.

As @baberabb mentioned, these other tasks are multiple-choice loglikelihood based, so if *all the MCQA answer choices are higher perplexity but still correct in relation to each other* then you might expect this result. I'd definitely investigate your model's generation quality or other perplexity results further, as it may be that these multiple-choice scores hide a significant performance gap when measuring actual PPL or generation quality.

Hi @Sami6720 , checking in on this--are you still experiencing any problems ?

Closing for tracking purposes--if you experience more problems please feel free to reopen this or comment again here!