Repository: EleutherAI/lm-evaluation-harness
Issue: AssertionError: len(continuation_enc) > 0
Root cause: Validation Gap

Hello

I ran

lm_eval --model hf     --model_args pretrained=EleutherAI/pythia-160m     --tasks /home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/tasks/ru_metrics     --device cuda:0     --batch_size 8

And I got error on example on image

<img width="1146" height="233" alt="Image" src="https://github.com/user-attachments/assets/595419b1-242c-4764-b0c0-11fe6ec543ef" />

Some tasks run ok on this model, but some like this do not


I read issues https://github.com/EleutherAI/lm-evaluation-harness/issues/1297  about this before and it was written that errors with tokenization might caise it
Unfortunately, advice from issue did not help me becuse repo changed


Running loglikelihood requests:   6%|██▉                                              | 3297/56168 [00Running loglikelihood requests:   6%|██▉                                              | 3361/56168 [00:07<01:24, 627.05it/s]Traceback (most recent call last):
  File "/home/user/conda/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/__main__.py", line 461, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/utils.py", line 456, in _wrapper
    return fn(*args, **kwargs)
  File "/home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/evaluator.py", line 357, in simple_evaluate
    results = evaluate(
  File "/home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/utils.py", line 456, in _wrapper
    return fn(*args, **kwargs)
  File "/home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/evaluator.py", line 585, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/api/model.py", line 391, in loglikelihood
    return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
  File "/home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/models/huggingface.py", line 1212, in _loglikelihood_tokens
    assert len(continuation_enc) > 0
AssertionError
Running loglikelihood requests:   6%|██▉                                              | 3392/56168 [00:07<01:54, 459.25it/s]


<img width="906" height="457" alt="Image" src="https://github.com/user-attachments/assets/7785190c-1de5-4b77-84f2-6863f6560537" />

Quick fix that worked for me - adding target_delimiter: "ANSWER" field. 

```
lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-2-7b-chat-hf,dtype="bfloat16" --apply_chat_template \
    --tasks arc_challenge,winogrande,hellaswag --batch_size 32 --seed 0 \
    --device cuda:0 --num_fewshot 0 --output_path ./log_dir/llama2-7b-magicoder-sft/lm_eval
```
I had the same issue with the above command. I tried adding `target_delimiter: "ANSWER"` to `lm_eval/tasks/winogrand/default.yaml`, `lm_eval/tasks/arc/arc_challenge.yaml`, and `lm_eval/tasks/hellaswag/hellaswag.yaml` and none of the edits to the yaml files resolved this issue. 

Interestingly, if I run the above command with `--model_args pretrained=meta-llama/Llama-3.2-1B-Instruct,dtype="bfloat16"` (note the different model name), there is no assertion error for `assert len(continuation_enc) > 0`.

Update: using suggestions by eco-bone and haileyschoelkopf in #1297 modifying the `else` block of  https://github.com/EleutherAI/lm-evaluation-harness/blob/d021bf846218f1bb3bdc0603864789329476d464/lm_eval/api/model.py#L366 seems to work.
```
        else:
            whole_enc = self.tok_encode(context + continuation)
            context_enc = self.tok_encode(context)
            if len(whole_enc) == len(context_enc):
                cont_enc = self.tok_encode(continuation, add_special_tokens=False)
                return context_enc, cont_enc

            context_enc_len = len(context_enc)
            continuation_enc = whole_enc[context_enc_len:]
```

The above solution follows directly from #1297.