Repository: EleutherAI/lm-evaluation-harness
Issue: IndexError: list index out of range when running benchmark on gguf model
Root cause: Validation Gap

I am running a llamacpp server on local host with phi-2_16.gguf model (as described here: https://github.com/EleutherAI/lm-evaluation-harness/issues/1254).
When running the lm-eval test suite (version 0.4.2) with the following commands: 

`lm_eval --model gguf --tasks winogrande --model_args base_url=http://localhost:8080 --output_path ./output/phi-2/winogrande --verbosity DEBUG --num_fewshot 5`

the execution breaks during the loglikelihood requests (for the 259th request) with the following exception:

```
File "<path_to_repository>\lm-evaluation-harness\lm_eval\__main__.py", line 341, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File <path_to_repository>\lm-evaluation-harness\lm_eval\utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "<path_to_repository>\lm-evaluation-harness\lm_eval\evaluator.py", line 251, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "<path_to_repository>\lm-evaluation-harness\lm_eval\utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "<path_to_repository>\lm-evaluation-harness\lm_eval\evaluator.py", line 390, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path_to_repository>\lm-evaluation-harness\lm_eval\models\gguf.py", line 89, in loglikelihood
    logprob, is_greedy = get_result(logprobs, len(context))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path_to_repository>\lm-evaluation-harness\lm_eval\models\gguf.py", line 22, in get_result
    while offsets[idx] < context_length:
          ~~~~~~~^^^^^
IndexError: list index out of range
```

I can't figure out what the problem is, am I using this incorrectly?

i meet the similar issue:
i use lm-eval (0.3.2) run agieval:
`
lm_eval --model vllm \
        --model_args pretrained=${model},trust_remote_code=True,tokenizer_mode="slow",tensor_parallel_size=8,dtype=auto,gpu_memory_utilization=0.8 \
        --tasks agieval \ 
        --batch_size auto \
        --output_path ${x} \
        --num_fewshot 5 \
        --device cuda 
    done
`
getting:
`Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())

  File "<path_to_repository>/lm_eval/__main__.py", line 342, in cli_evaluate
    results = evaluator.simple_evaluate(

  File "<path_to_repository>/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "<path_to_repository>/lm_eval/evaluator.py", line 234, in simple_evaluate
    results = evaluate(
  File "<path_to_repository>lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "<path_to_repository>/lm_eval/evaluator.py", line 325, in evaluate
    task.build_all_requests(
  File "<path_to_repository>/lm_eval/api/task.py", line 418, in build_all_requests
    fewshot_ctx = self.fewshot_context(
  File "<path_to_repository>/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "<path_to_repository>/lm_eval/api/task.py", line 950, in fewshot_context
    labeled_examples = description + self.sampler.get_context(doc, num_fewshot)
  File "<path_to_repository>/lm_eval/api/samplers.py", line 37, in get_context
    [
  File "<path_to_repository>/lm_eval/api/samplers.py", line 49, in <listcomp>
    str(self.doc_to_target(doc)[0])
IndexError: list index out of range
`
i don't know the reason and i change the --num_fewshot 0 it works. but i don't know why num_fewshot can't be 5.
and can someone teach me how to set up num_fewshot for different datasets?

I faced a similar issue. 
In my case, the problem was that **llama-cpp-python** assumes the prompt will always include a BOS token when preparing the `text_offset` for `logprobs`. 
This resolved my issue: https://github.com/abetlen/llama-cpp-python/pull/1471.