zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Raise error/warning in case `max_gen_toks` is too small (e.g. for both thinking tokens + answer)
Root cause: Validation Gap

As per title, with hf backend.

At the moment, the accuracy may be 0 because the model is verbose for thinking tokens, consuming all the allocated `max_gen_toks`.

There should be an error (or at least a warning) in case a model is not given enough budget to actually generate an answer, e.g. when:

- the generated sequence hits `max_gen_toks` (=> but sometimes I guess it is OK not to finish due to BOS token)
- not enough `
`/`
`/etc tokens in the generated output (i.e., answer not detected)

Similarly, it may be good to warn users (or raise error) not to trust loglikelihood tasks for reasoning models (as gpt-oss).

related issue https://github.com/EleutherAI/lm-evaluation-harness/issues/3382
Repository: EleutherAI/lm-evaluation-harness
Issue: Duplicated groups/tasks (`--tasks mmlu_redux_generative,mmlu_pro_llama,mmlu_pro,mmlu_generative,mmlu_continuation`)

Hi, running:

```bash
CUDA_VISIBLE_DEVICES=1 lm_eval \
  --model hf \
  --model_args pretrained=${MODEL_ID},dtype=auto \
  --gen_kwargs max_gen_toks=2048 \
  --tasks mmlu_redux_generative,mmlu_pro_llama,mmlu_pro,mmlu_generative,mmlu_continuation \
  --apply_chat_template \
  --batch_size 36
```

One runs into:

```
ValueError: Found 1 or more tasks while trying to call get_task_dict() that were members of more than 1 called group: ['humanities', 'other', 'social sciences', 'stem']. Offending groups: ['mmlu_redux_generative', 'mmlu_generative', 'mmlu_continuation']. Please call groups which overlap their constituent tasks in separate evaluation runs.
```

Is it expected?

Thank you!
Repository: EleutherAI/lm-evaluation-harness
Issue: GPQA `strict-match` regex pattern does not match the fewshot response template
Root cause: Validation Gap

Hi,

As per title.

In https://github.com/EleutherAI/lm-evaluation-harness/blob/68b03658ace40fa93221518b30096485a2387c58/lm_eval/tasks/gpqa/cot_n_shot/_gpqa_cot_n_shot_yaml#L9-L17 and https://github.com/EleutherAI/lm-evaluation-harness/blob/68b03658ace40fa93221518b30096485a2387c58/lm_eval/tasks/gpqa/generative/_gpqa_generative_n_shot_yaml#L9-L17 we have the instruction `Answer the final question yourself, following the format of the previous questions exactly`, with the expected output `regex_pattern: "(?<=The answer is )(.*)(?=.)"`.

However, when using e.g. `--num_fewshot 5`, the answers in the prompt are formatted as follow: 

<img width="1178" height="1170" alt="Image" src="https://github.com/user-attachments/assets/d248ea56-57f4-4f45-8dcd-196fa5350b0e" />

Which is not the format suggested in the regex, as `The answer is` is missing. The ends up not incentivizing the model to use the regex format, and eventually the `strict-match` is 0.

Only `flexible-extract` is decent.

One can reproduce with:

```bash
CUDA_VISIBLE_DEVICES=0 nohup lm_eval \
  --model hf \
  --model_args '{"pretrained":"openai/gpt-oss-20b","dtype":"auto","chat_template_args":{"reasoning_effort":"low"},"enable_thinking": true}' \
  --device "cuda" \
  --gen_kwargs max_gen_toks=4048 \
  --tasks gpqa_diamond_generative_n_shot \
  --apply_chat_template \
  --fewshot_as_multiturn \
  --limit 1 \
  --num_fewshot 5 \
  --batch_size 1
```

Thank you!

Related issues: https://github.com/EleutherAI/lm-evaluation-harness/issues/3110 and https://github.com/EleutherAI/lm-evaluation-harness/issues/2707

Because gpqa-n-shot doesn't have fewshot directly set in the `.yaml` so it just takes the first n questions from the dataset and formats it according to the chat template

Thank you @jannalulu, what is surprising is that the template answer for n-shot is not formatted as `The answer is XXXX`. Am I missing an option to format the n-shot answers this way?

Oh I see what's wrong. This is not correct:

https://github.com/EleutherAI/lm-evaluation-harness/blob/68b03658ace40fa93221518b30096485a2387c58/lm_eval/tasks/gpqa/generative/_gpqa_generative_n_shot_yaml#L11

(or there needs to be a param to template n-shot answers, seem not to exist at the moment)
Repository: EleutherAI/lm-evaluation-harness
Issue: Qwen3-4b model strange behavior on MMLU
Root cause: Documentation Deficiency

Hi guys, 

I’m benchmarking Qwen3-4B on MMLU using lm-harness, and I’m seeing large performance differences depending on whether I use apply_chat_template and whether “thinking mode” is enabled.

**Results**

5-shot MMLU (lm-harness)

1. No apply_chat_template, 

- Score: 0.7014. This is close to Qwen’s reported 72.99 in the technical report.

2. With apply_chat_template, thinking disabled, 

- Score: 0.5721

With apply_chat_template, thinking enabled

- Score: 0.2281

**Why I’m Confused**

These results suggest that Qwen3-4B behaves like a base model, where using chat templates significantly hurts completion-style evaluation (e.g., MMLU). That's what their technical report implies. 

However, the official HuggingFace example for Qwen3-4B
👉 [https://huggingface.co/Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B?utm_source=chatgpt.com)

uses apply_chat_template, which is more typical for chat- or instruct-tuned models.

This raises a few questions:

Is Qwen3-4B actually a base model or an instruct/chat model?
1. The HF example implies chat usage, but the eval behavior aligns with a base LM.
2. if it is a base model, then why does enabling/disabling thinking mode have such a large impact? 
3. What is the intended evaluation setup for tasks like MMLU for Qwen3-4B?

**Below is the code i use "thinking disabled"**
```
lm_eval \
  --model hf \
  --tasks "mmlu" \
  --model_args pretrained=Qwen/Qwen3-4B,tokenizer=Qwen/Qwen3-4B,enable_thinking=False,dtype=float16,parallelize=True \
  --batch_size auto:4 \
  --num_fewshot 5 \
  --apply_chat_template \
  --trust_remote_code \
```

**enable thinking**
```
lm_eval \
  --model hf \
  --tasks "mmlu" \
  --model_args pretrained=Qwen/Qwen3-4B,tokenizer=Qwen/Qwen3-4B,dtype=float16,parallelize=True \
  --batch_size auto:4 \
  --num_fewshot 5 \
  --apply_chat_template \
  --trust_remote_code \
```

i am using 
```
Name: transformers
Version: 4.53.0

Name: lm_eval
Version: 0.4.9.1
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Support int `think_end_token` for vllm backend
Root cause: Unimplemented Feature Gap

Hi,

As per title.

Transformers backend supports `int` think_end_token: https://github.com/EleutherAI/lm-evaluation-harness/blob/29a0765a4d13ff88797743347aa57b230b27bd98/lm_eval/models/huggingface.py#L105

But vllm backend does not support it: https://github.com/EleutherAI/lm-evaluation-harness/blob/29a0765a4d13ff88797743347aa57b230b27bd98/lm_eval/models/vllm_causallms.py#L148

Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: Adding Instruction following benchmarks
Root cause: Unimplemented Feature Gap

It will useful to include instruction following benchmarks to explicitly evaluate the instruction following capability of LLMs. Some datasets like [Self-instruct](https://arxiv.org/abs/2212.10560), [SuperNaturalInstructions](https://arxiv.org/pdf/2204.07705), [Natural Instructions](Cross-task generalization via natural language crowdsourcing instructions)
Repository: EleutherAI/lm-evaluation-harness
Issue: When evaluating Qwen3-32B with enabling thinking mode, the responses will be wrongly extracted (only thinking contains)
Root cause: Validation Gap

As the title, I tried to add a QA task for Qwen3-32B and see the different results if I switch on the thinking mode or off. For the Qwen3 series, I supposed to see the response in this form (I enabled log_sample):
```json
"reps":[
["<think>\n blah blah blah(the thinking contains).\n</think>\n\n Answer: XXX. 
"]
],
"filtered_resps":["Answer: XXX."],
```
But now I could only see the things:
```json
"reps":[
["<think>\n blah blah blah(the thinking contains).\n</think>"]
],
"filtered_resps":["<think>\n blah blah blah(the thinking contains).\n</think>"],
```
The real response missing. In other words, the lm-eval extracts the beginning thinking token to the end thinking token as response, but drops other things till the end token.

Although I could modify my utils script for catching the answer from the thoughts, the incomplete response should be a bug, right?

Solved with set correct until token
Repository: EleutherAI/lm-evaluation-harness
Issue: TypeError: expected str, bytes or os.PathLike object, not NoneType
Root cause: Validation Gap

I got this error when testing with
```
python main.py \
	--model gpt2 \
	--model_args pretrained=EleutherAI/gpt-neo-125M \
	--device 0 \
	--tasks lambada,hellaswag
```
I think this occurs because of the default value of the subfolder is None.
```
tokenizer1 = AutoTokenizer.from_pretrained('EleutherAI/gpt-neo-125M')

tokenizer2 = AutoTokenizer.from_pretrained('EleutherAI/gpt-neo-125M', subfolder=None)
```
tokenizer1 have no issue, but tokenizer2 got same error.
Repository: EleutherAI/lm-evaluation-harness
Issue: Optional mode to skip samples exceeding max_length
Root cause: Unimplemented Feature Gap

Hi, thanks for lm-eval-harness.

Right now if an example exceeds max_length, it gets truncated and still evaluated. Would it be possible to add an optional mode to just skip/drop those over-length samples instead?

This is useful when truncation changes the question too much, or when comparing models with different context windows.
If enabled, I’d expect:
- over-length instances are not evaluated
- they’re excluded from metric denominators
- skipped count/% is logged per task

Something like --skip_overlength / skip_overlength: true would be great.
Happy to help with a PR if this fits the project direction.

Thanks!

Or maybe have both outputs by default: one including running into `max_new_toks`/`[invalid]` filters (current), and one excluding those.

That sounds good to me as well, having both metrics would make comparisons much clearer.
Repository: EleutherAI/lm-evaluation-harness
Issue: Requirements Are Unsatisfiable - Installing From Source
Root cause: External Dependency Breakage

Hi. I'm trying to install evaluation harness from source, since I want to use AIME benchmarks.

I run `uv add ./lm-evaluation-harness`

I get the following:
```
Added `lm-evaluation-harness` to workspace members
  × No solution found when resolving dependencies:
  ╰─▶ Because tarski==0.8.2 depends on antlr4-python3-runtime==4.7.2 and
      lm-eval[tasks] depends on antlr4-python3-runtime==4.11, we can conclude that
      lm-eval[tasks] and tarski[clingo]==0.8.2 are incompatible.
      And because lm-eval[tasks] depends on tarski[clingo]==0.8.2 and your
      workspace requires lm-eval[tasks], we can conclude that your workspace's
      requirements are unsatisfiable.
  help: If you want to add the package regardless of the failed resolution, provide
        the `--frozen` flag to skip locking and syncing.
```

Will the dependencies be fixed? What would you recommend? Thank you.

This seems to be due to `lm_eval[acp_bench]` running into `lm_eval[math]`. I think you can install it with `uv add "./lm-evaluation-harness[math]"` if you just want to run AIME
Repository: EleutherAI/lm-evaluation-harness
Issue: Cannot load custom task config file via its path
Root cause: Validation Gap

I'm trying to load a custom config file by passing its path as the `tasks` argument, according to the description in [New Task Guide](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#task-name--tags-registering-a-task), but lm_eval cannot find the task file even if the given path is correct.

```bash
$ ls -al                      
total 12
drwxrwxr-x  2 zbw zbw 4096 Nov 24 06:53 .
drwxr-x--- 23 zbw zbw 4096 Nov 24 07:27 ..
-rw-rw-r--  1 zbw zbw 1037 Nov 24 06:53 gsm8k.yaml
$ lm-eval --tasks ./gsm8k.yaml
2025-11-24:07:24:55 ERROR    [__main__:419] Tasks were not found: ./gsm8k.yaml
                                               Try `lm-eval --tasks list` for list of available tasks
Traceback (most recent call last):
  File "/data/zbw/miniconda3/envs/dllm/bin/lm-eval", line 7, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/data/zbw/miniconda3/envs/dllm/lib/python3.12/site-packages/lm_eval/__main__.py", line 423, in cli_evaluate
    raise ValueError(
ValueError: Tasks not found: ./gsm8k.yaml. Try `lm-eval --tasks {list_groups,list_subtasks,list_tags,list}` to list out all available names for task groupings; only (sub)tasks; tags; or all of the above, or pass '--verbosity DEBUG' to troubleshoot task registration issues.
```

However, if the path of its parent directory (in the above case it's `~/demo` or `.`) is passed, lm_eval can load the custom task.

```bash
$ lm-eval --tasks .           
2025-11-24:07:45:35 INFO     [__main__:446] Selected Tasks: [{'tag': ['math_word_problems'], 'task': 'gsm8k', 'dataset_path': '/data/zbw/szj/datasets/openai/gsm8k', 'dataset_name': 'main', 'output_type': 'generate_until', 'training_split': 'train', 'fewshot_split': 'train', 'test_split': 'test', 'doc_to_text': 'Question: {{question}}\nAnswer:', 'doc_to_target': '{{answer}}', 'metric_list': [{'metric': 'exact_match', 'aggregation': 'mean', 'higher_is_better': True, 'ignore_case': True, 'ignore_punctuation': False, 'regexes_to_ignore': [',', '\\$', '(?s).*#### ', '\\.$']}], 'generation_kwargs': {'until': ['Question:', '
', '
'], 'do_sample': False, 'temperature': 0.0}, 'repeats': 1, 'num_fewshot': 5, 'filter_list': [{'name': 'strict-match', 'filter': [{'function': 'regex', 'regex_pattern': '#### (\\-?[0-9\\.\\,]+)'}, {'function': 'take_first'}]}, {'name': 'flexible-extract', 'filter': [{'function': 'regex', 'group_select': -1, 'regex_pattern': '(-?[$0-9.,]{2,})|(-?[0-9]+)'}, {'function': 'take_first'}]}], 'metadata': {'version': 3.0}}]
# skip unrelevant following output
```

Following the error stack trace, the problem lies in the check for missing tasks, where lm_eval will try to match a `dict` with a `str` and eventually fail in the case of loading an external file.

https://github.com/EleutherAI/lm-evaluation-harness/blob/7ddb2b1e4bc819292ff56f334c572d1fc77dec28/lm_eval/__main__.py#L409-L415
Repository: EleutherAI/lm-evaluation-harness
Issue: Get model SHA is always failed when pass existing model to HFLM
Root cause: Validation Gap

When passing an existing model to HFLM, it always outputs the warning "WARNING Failed to get model SHA for <model_name> ..."

https://github.com/EleutherAI/lm-evaluation-harness/blob/7ddb2b1e4bc819292ff56f334c572d1fc77dec28/lm_eval/models/huggingface.py#L1555-L1570

Because `HfApi().model_info` only accepts strings, this causes this issues when `self.pretrained` is a `transformers.PreTrainedModel`.
It could be resolved by modifying the variable known as `self.pretrained` to be of the type PreTrainedModel, or by not fetching the SHA.
Repository: EleutherAI/lm-evaluation-harness
Issue: Custom tasks fail to load due to forced relative path resolution
Root cause: Validation Gap

I'm trying to load a custom task config YAML by passing it using the path using`--include-path` and task name using `--tasks`.
This previously worked, but after the recent changes in `get_task_dict` and addition of `pretty_print_task`, LM-Eval can no longer load YAMLs located outside the `lm_eval/tasks/` directory.

Even when the file exists and has a valid config, LM-Eval raises a ValueError during task loading.

### Error Traceback (redacted version): 

```
Traceback (most recent call last):
  File "/path/to/env/bin/lm-eval", line 7, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/path/to/lm_eval/__main__.py", line 474, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/lm_eval/utils.py", line 458, in  #_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/path/to/lm_eval/evaluator.py", line 283, in simple_evaluate
    task_dict = get_task_dict(
                ^^^^^^^^^^^^^^
  File "/path/to/lm_eval/tasks/__init__.py", line 716, in get_task_dict
    pretty_print_task(key, task_manager, indent=0)
  File "/path/to/lm_eval/tasks/__init__.py", line 683, in pretty_print_task
    relative_yaml_path = yaml_path.relative_to(lm_eval_tasks_path)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/python3.11/pathlib.py", line 730, in relative_to
    raise ValueError(f"{str(self)!r} is not in the subpath of {str(other)!r} "
ValueError: '/full/path/to/custom_configs/custom_task.yaml' is not in the subpath of '/full/path/to/lm_eval/tasks' OR one path is relative and the other is absolute.
```


### Potential Cause 

After checking the code, the breakage is caused by a recent addition of `pretty_print_task` inside `get_task_dict,` where LM-Eval now assumes that every task's YAML file is a child of the built-in tasks directory:

https://github.com/EleutherAI/lm-evaluation-harness/blob/ad3f4d0cad1cfcdb815f1e795f7947e49ed9f2e9/lm_eval/tasks/__init__.py#L683


### Problematic code for reference

https://github.com/EleutherAI/lm-evaluation-harness/blob/ad3f4d0cad1cfcdb815f1e795f7947e49ed9f2e9/lm_eval/tasks/__init__.py#L679-L686




Fixed in #3436
Repository: EleutherAI/lm-evaluation-harness
Issue: Question about evals - gsm8k Template & Strict extraction
Root cause: Configuration Error

Hi,

Am using the gsm8k eval and looking at the yaml, results are better when I do `Question: {{question}}\nEnd your answer with: #### <answer>\nAnswer: ` instead of `Question: {{question}}\nAnswer: `. Should this be changed? The original makes sense for n-shot but makes no sense for zero-shot.

Also, should strict be `group_select: -1` instead of leftmost? I have several cases where reasoning models (deepseek-ai/DeepSeek-R1-0528) recite the n-shot cases during their reasoning ramble.



@Oseltamivir same conclusion in https://github.com/EleutherAI/lm-evaluation-harness/pull/3411
Repository: EleutherAI/lm-evaluation-harness
Issue: [fix] Chat 모델 평가 시 --apply_chat_template 누락으로 인한 성능 저하
Root cause: Configuration Error

## 문제

KMMLU 평가 시 `--apply_chat_template` 플래그 누락으로 인해 성능이 낮게 나왔습니다.

## 원인

Qwen3-8B는 **Chat 모델**이므로 특정 포맷(`
user`...`
`)으로 훈련되었는데, 평가 시 raw text가 입력되어 성능 저하가 발생했습니다.

## 해결

`--apply_chat_template` 플래그 추가:

```bash
accelerate launch \
    -m lm_eval \
    --model hf \
    --model_args pretrained=/data/cache_dir/models/Qwen/Qwen3-8B,... \
    --tasks kmmlu \
    --apply_chat_template  # ✅ 추가
```

## 영향

- **Before**: Raw text 입력 → 모델 혼란
- **After**: Chat format 입력 → 정상 동작
- **예상 개선**: KMMLU +2~5% 정확도 향상

## 체크리스트

- [x] KMMLU launch.json 수정 완료
- [x] Interview eval 수정 완료

---
**Priority**: P1  
**Tags**: #performance #chat-template


잘못된 레포지토리에 생성되었습니다. ThakiCloud fork로 이동합니다.
Repository: EleutherAI/lm-evaluation-harness
Issue: [feat] 면접 평가 AI 성능 측정 태스크 (interview_eval) 구현
Root cause: Unimplemented Feature Gap

## 📋 개요
면접 평가 AI 모델의 성능을 측정하기 위한 lm-evaluation-harness 커스텀 태스크를 구현했습니다.

## 🎯 목적
- 면접 평가 AI가 생성한 결과물의 **구조적 정확성** 검증
- **등급 예측 정확도** (S/A/B/C/D) 측정
- **점수 오차** (MAE) 측정

## 📁 구현 파일
- `lm_eval/tasks/interview_eval/interview_eval.yaml` - 태스크 설정 파일
- `lm_eval/tasks/interview_eval/utils.py` - 메트릭 및 유틸리티 함수

## 📊 평가 메트릭

### 주요 메트릭 (3개)
| 메트릭 | 설명 | 범위 | 좋은 값 |
|--------|------|------|---------|
| `structure_overall` | 종합 구조 검증 점수 | 0-1 | 높을수록 좋음 |
| `grade_exact_match` | 등급 정확도 (S/A/B/C/D) | 0-1 | 높을수록 좋음 |
| `total_score_mae` | 총점 평균 절대 오차 | 0-20 | 낮을수록 좋음 |

### 구조 검증 항목 (structure_overall)
1. JSON 파싱 가능 여부
2. 필수 키 존재 여부 (interview_info, evaluation, overall_review)
3. candidate_name 정확도
4. 점수 범위 유효성 (1-5)
5. grade와 total_score 매칭 검증
6. final_decision 유효성 (PASS/HOLD/FAIL)

### 등급 기준
| 등급 | 점수 범위 | 의미 |
|------|----------|------|
| S | 19-20점 | 즉시 채용 강력 추천 |
| A | 15-18점 | 채용 추천 |
| B | 10-14점 | 조건부 채용/보류 |
| C | 5-9점 | 채용 불가 |
| D | 1-4점 | 부적격 |

### 선택적 디버그 메트릭
- `grade_proximity`: ±1 등급 허용 근접도
- `criteria_mae`: 개별 평가 항목별 MAE (experience, knowledge, personality, culture)

## 🔧 사용 방법
```bash
python -m lm_eval \
  --model hf \
  --model_args pretrained=<MODEL_PATH> \
  --tasks interview_eval \
  --device cuda:0 \
  --batch_size 16 \
  --apply_chat_template
```

## ✅ 체크리스트
- [x] interview_eval.yaml 태스크 설정
- [x] utils.py 메트릭 함수 구현
- [x] 구조 검증 메트릭 (structure_overall)
- [x] 등급 정확도 메트릭 (grade_exact_match)
- [x] 점수 MAE 메트릭 (total_score_mae)
- [x] 디버그 메트릭 (grade_proximity, criteria_mae)
Repository: EleutherAI/lm-evaluation-harness
Issue: Comma-separated `--tasks` does not work
Root cause: Validation Gap

The `--tasks` option no longer works for me with comma-separated lists of tasks, as it used to work. Now I can only get it to work with space-separated lists or a single task, but the docs say it should work with comma-separated lists as well:
```
model and tasks:
  --tasks [<task> ...], -t [<task> ...]
                        Space or Comma-separated list of task names or groupings. Use 'lm-eval list tasks' to see all available tasks.
```

Testing it in inside a fully fresh clone of `lm-evaluation-harness`, with no other version of `lm_eval` installed:

```
$ python -m lm_eval run --model hf --model_args "pretrained=$model_path" --tasks ifeval,hellaswag,cola

(...)
  File "lm-evaluation-harness/lm_eval/config/evaluate_config.py", line 405, in process_tasks
    raise ValueError(f"Tasks not found: {missing}")
ValueError: Tasks not found: ifeval,hellaswag,cola
```

But this works:
```
$ python -m lm_eval run --model hf --model_args "pretrained=$model_path" --tasks ifeval hellaswag cola

(...)
2025-12-03:14:44:19 INFO     [tasks:691] Task: cola (glue/cola/default.yaml)
2025-12-03:14:44:19 INFO     [tasks:691] Task: hellaswag (hellaswag/hellaswag.yaml)
2025-12-03:14:44:19 INFO     [tasks:691] Task: ifeval (ifeval/ifeval.yaml)
(...)
```

Either the ArgumentParser and the post-processing is bugged or the documentation of this argument needs to be updated.

+1, cc @baberabb 

other breaking change: https://github.com/EleutherAI/lm-evaluation-harness/pull/3440#discussion_r2588622920
Repository: EleutherAI/lm-evaluation-harness
Issue: Add Big-Bench Lite to eval-harness
Root cause: Unimplemented Feature Gap

We want to be able to evaluate our models on BIG-Bench Lite. The BIGBench code is not the most outsider-friendly so I'll try to add the BigBench lite tasks to the eval-harness, and test on GPT2 for equivalence to confirm the scores should transfer back to the original implementation.

So far: have looked through the BigBench HF code a bit.

I remember testing T0 on Bigbench with Eval Harness last year. This probably could save us time and effort.

Oh that would be awesome if so. was this in mesh-tensorflow / t5x or in Huggingface?

https://colab.research.google.com/github/google/BIG-bench/blob/main/bigbench/bbseqio/docs/t5x_eval.ipynb This seems to be the script to use?
Repository: EleutherAI/lm-evaluation-harness
Issue: [feat] Add vLLM support for interview_eval task
Root cause: Unimplemented Feature Gap

## 🎯 목표
interview_eval 태스크의 평가 속도를 개선하기 위해 vLLM 백엔드를 지원합니다.

## 📋 배경
- 현재 interview_eval 평가가 오래 걸리는 문제가 있음
- vLLM을 사용하면 약 3배 빠른 inference가 가능하다고 알려져 있음
- lm-evaluation-harness는 vLLM 백엔드를 지원함

## 📝 주요 작업 내용
- [ ] vLLM 백엔드로 interview_eval 실행 테스트
- [ ] 필요시 interview_eval.yaml 설정 조정
- [ ] vLLM 실행 스크립트 또는 문서 작성
- [ ] 성능 비교 (HuggingFace vs vLLM)

## ✅ 성공 기준
- vLLM 백엔드로 interview_eval 평가가 정상 동작함
- 기존 대비 inference 속도가 유의미하게 개선됨
- 평가 결과의 일관성이 유지됨

## 📚 참고
- lm-evaluation-harness vLLM 지원: `lm_eval --model vllm`
- 관련 문서: https://github.com/EleutherAI/lm-evaluation-harness#vllm