Repository: EleutherAI/lm-evaluation-harness
Issue: How to get the embeddings and initial prompt for all answer options in a dataset (e.g., HellaSwag)?
Root cause: Documentation Deficiency

I am evaluating a model with the following command:

`lm_eval \
          --model hf \
          --model_args "pretrained=${MODEL_DIR},add_bos_token=False,parallelize=False,dtype=bfloat16" \
          --tasks "hellaswag=0" \
          --num_fewshot "0" \
          --write_out \
          --show_config \
          --batch_size "${BATCH_SIZE}" \
          --output_path "${OUTPUT_DIR}/${MODEL_NAME}_${TASK}" \
          --trust_remote_code --log_samples \
          > "${LOG_FILE}" 2>&1`

I'd like to extract the last layer embeddings of this model (Llama-3.1 8B) for each question in the dataset. I can generally do that extraction with NNsight or similar, but I was wondering if eval-harness provides an interface for question-by-question embedding extraction. Furthermore, I'd like to know how the final prompt is formatted to the model. Is there a way I can see these two?

Thanks! 



I'm interested in this also.

For multiple-choice tasks the final prompt is formatted as `doc_to_choice` + `target_delimiter` (defaults to `" "`) + `choice`, for each of the choices and the logliklihood of of each choice is computed to determine the model's prediction. The choice with the highest loglikelihood score is selected as the model's answer. You can set it in the task [config](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/hellaswag/hellaswag.yaml#L11-L13). Also [see](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#multiple-choice-format).

The harness doesn't provide a direct way for extracting the hidden-states but maybe you can add some modifications [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/19ba1b16fef9fa6354a3e4ef3574bb1d03552922/lm_eval/models/huggingface.py#L1199)






Thank you! I have figured out how to extract hidden states by adding some code [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/19ba1b16fef9fa6354a3e4ef3574bb1d03552922/lm_eval/models/huggingface.py#L882). However, I am confused about the logit outputs. For the eval dataset I am using (PIQA), there are two possible options per question. When I print out the contents of the results file, I see the following:

```
pth = PATH_TO_RESULTS_FILE
import json
with open(pth, 'r') as f:
    data = [json.loads(line) for line in f]
for i in range(10):
    print(data[i]['resps'], data[i]['acc'])
```

The output I see is something like:

`[[['-100.0', 'False']], [['-100.5', 'False']]] 1.0
[[['-40.0', 'False']], [['-65.5', 'False']]] 0.0
[[['-35.25', 'False']], [['-39.25', 'False']]] 0.0
[[['-37.25', 'False']], [['-38.0', 'False']]] 0.0
[[['-37.75', 'False']], [['-44.0', 'False']]] 1.0`

I don't understand what the 'False' is here. It seems that these two indices correspond to the logits for different choices. However, I'm unclear on what 'False' means. The YAML file for the pythia eval is [here](https://github.com/EleutherAI/lm-evaluation-harness/tree/19ba1b16fef9fa6354a3e4ef3574bb1d03552922/lm_eval/tasks/piqa).

Hi! the boolean indicates if the target string is the most [likely](https://github.com/EleutherAI/lm-evaluation-harness/blob/19ba1b16fef9fa6354a3e4ef3574bb1d03552922/lm_eval/models/huggingface.py#L1237) N-token string to be output by the LM given the context (that is if we sample greedily from the context). It's only used by `lambada` from my understanding. Also [see](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md#interface)

Thanks! So it seems like if there are two options (e.g., ['A', 'B']), the eval harness code makes two inference calls to the model, one for each choice (i.e., basically two prompts). However, if I want to get a single embedding (like hidden_state[:,-1,:]) per question from the model, do you have a recommendation for how to achieve that?

> Thanks! So it seems like if there are two options (e.g., ['A', 'B']), the eval harness code makes two inference calls to the model, one for each choice (i.e., basically two prompts). However, if I want to get a single embedding (like hidden_state[:,-1,:]) per question from the model, do you have a recommendation for how to achieve that?

We do group samples for single-token continuations so there's only one forward pass per similar context. ([1](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/models/huggingface.py#L1056-L1062), [2](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/models/huggingface.py#L1223-L1237)). For tasks with multi-token continuations like PIQA, I guess the simplest way would be to keep a cache of [the](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/models/huggingface.py#L1111) `context_enc` (or the first index which is the is the (context, continuation) string tuple) and skip if theres a hit? 

Okay, so when the get_cache() method is run and has multiple hits, all of the logits are computed from a single forward pass. However, for PIQA, which has 1838 questions, I end up having 3661 forward passes to compute 1838*2=3676 output logits, so the get_cache() only computes a single forward pass for 15 of the questions and not all of the questions. Is there a way to force a new forward pass with each input?

Yeah just pass `logits_cache=False` to the model_args

Ok, awesome! I was also checking over the `context_enc` lengths inputwise, and I noticed that they varied (ex. lengths: 59, 62, 50, 55). How are the lengths of `context_enc` determined for each input?