zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Standardize Task Templates
Root cause: Unimplemented Feature Gap

Creating new evaluation tasks currently requires understanding some low-level implementation detauls, even for common patterns like MMLU-style multiple choice or cloze tasks, and require repetitive boilerplate configs, making the framework less accessible than it could be.

We can standardize these common patterns by creating task templates that handle the typical evaluation formats: this would involve defining standard field mappings (question, choices, answer) for the templates. However, this standardization would require enforcing certain field usage (and type) patterns, so need to handle backward compatibility with care. 

For example a template could be set from the config like:
```
task_template: "mmlu"
required_fields:
  question: "question"        str
  choices: "choices"          list
  answer: "answer"            int
```
And internally they would be mapped to `doc_to_text`, `doc_to_choice` and `doc_to_target`, in the appropriate format.
Repository: EleutherAI/lm-evaluation-harness
Issue: Simplify ConfigurableTask.process_results()
Root cause: Architectural Constraint

The current process_results method has grown unwieldy and difficult to understand, with deep branching logic  for different output types (for example `multiple_input`, 'multiple_output`) and implicit type conversions. We can simplify this by keeping the method focused on the usual case with explicit types, while more complicated scenarios can be handled by overloading the method in custom tasks.

Hi! I've refactored process_results() into smaller helper functions (e.g., _process_results_loglikelihood, etc.) to address the complexity mentioned here.

The new structure keeps the standard logic in the base class and makes it easier to override specific output types for custom tasks. I’ve also isolated mutual info handling and gold normalization into separate helpers.

Would you be open to a PR for this? Happy to polish it further if needed.
Repository: EleutherAI/lm-evaluation-harness
Issue: Streamlining lm-eval Architecture

## Motivation

As the LM Evaluation Harness has grown and evolved, we've accumulated some complexity in our codebase. While this flexibility has been valuable for supporting a wide range of use cases, it has also created several challenges:

- **Steeper learning curve**: New contributors and users encounter a learning curve when getting familiar with the evaluation pipeline, task configurations, and filter mechanisms
- **Maintenance overhead**: Some abstractions could be streamlined or be made more explicit.
- **Code clarity**: The current codebase has grown organically, leading to some patterns that could be more intuitive and maintainable

I'm considering several potential modifications to streamline the harness architecture:

## 1. Filter/Metric Pipeline Restructuring
- [ ] Create more intuitive abstractions for common post-processing patterns (e.g. creating specific filters for common use cases)
- [ ] Make the pipeline more type-explicit (e.g. #3082)
- [ ] Address repeat handling limitations (#3080)
- [ ] Make metrics explicit rather than passthrough calculations in `process_results` (requires handling different argument types across task patterns)

### 2. Task Definition Ergonomics
- [ ] Create simplified interfaces for conventional task formats (MMLU-style multiple choice, cloze hybrid tasks, etc.) through templating systems (tracked in #3081) and converting from multiple-choice to generation

### 3. Handle CI Pain Points:
- [ ] #2893 (tracked in #3083)

### 3. Documentation and Discoverability
- [ ] Make the tasks more discoverable through better organization and indexing (maybe through some hierarchical grouping)
- [ ] Provide clearer documentation for how the pieces fit together
- [ ] Improve examples and onboarding materials

## Feedback

I'd love to hear from the community about:
- Which areas of complexity have been most challenging in your experience?
- What aspects of the current architecture work well and should be preserved?
- Any specific pain points or use cases that should be prioritized?
- Suggestions for maintaining backward compatibility?
Repository: EleutherAI/lm-evaluation-harness
Issue: api requests hanged  when too many request running concurrently
Root cause: Resource Mishandling

When evaluate with API
The code here https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/api_models.py#L537 launch request function coucurrently, if the requests number is extremely large, i.e. 20k.  There are too many amodel_call called, which would cause process hanged. 
```
async def amodel_call(
        self,
        session: ClientSession,
        messages: Union[List[List[int]], List[str], List[JsonChatStr]],
        *,
        generate: bool = True,
        cache_keys: list = None,
        ctxlens: Optional[List[int]] = None,
        gen_kwargs: Optional[Dict] = None,
        **kwargs,
    ) -> Union[List[str], List[Tuple[float, bool]], None]:
        # !!! Copy: shared dict for each request, need new object !!!
        gen_kwargs = copy.deepcopy(gen_kwargs)
        payload = self._create_payload(
            self.create_message(messages),
            generate=generate,
            gen_kwargs=gen_kwargs,
            seed=self._seed,
            **kwargs,
        )
        cache_method = "generate_until" if generate else "loglikelihood"
        try:
            async with session.post(
                self.base_url,
                json=payload,
                headers=self.header,
            ) as response:
                if not response.ok:
                    error_text = await response.text()
                    eval_logger.warning(
                        f"API request failed with error message: {error_text}. Retrying..."
                    )
                # raising exception will retry the request
                response.raise_for_status()
                outputs = await response.json()
            answers = (
                self.parse_generations(
                    outputs=outputs,
                )
                if generate
                else self.parse_logprobs(
                    outputs=outputs,
                    tokens=messages,
                    ctxlens=ctxlens,
                )
            )
            if cache_keys:
                for res, cache in zip(answers, cache_keys):
                    self.cache_hook.add_partial(cache_method, cache, res)
            return answers
        # If the retries also fail
        except RetryError:
            eval_logger.error(
                "API request failed after multiple retries. Please check the API status."
            )
            return None
```

![Image](https://github.com/user-attachments/assets/e74b5d11-9701-4487-8c9c-84a21567ca5a)
fixing by 

![Image](https://github.com/user-attachments/assets/63358401-91d8-4f88-b07f-2f213eea6225)

![Image](https://github.com/user-attachments/assets/0817726e-9cbb-493f-9078-c425da03e2b0)
Repository: EleutherAI/lm-evaluation-harness
Issue: AttributeError: module transformers has no attribute Qwen2AudioForConditionalGeneration
Root cause: External Dependency Breakage

Hi team,

I think there's a dependency issue from following the initial installation steps, it is resulting in the above error.

The fix seems to be coming from `pip install -U git+https://github.com/huggingface/transformers` but there's no backward compatibility!
Repository: EleutherAI/lm-evaluation-harness
Issue: Tokenizer Merges Built on the Fly When gguf_file Is Set, Even If External Tokenizer Is Provided
Root cause: Configuration Error

### 🔹 **Description**:

When evaluating quantized models in **GGUF format** using the `huggingface` backend in `lm-evaluation-harness`, the tokenizer attempts to build merges on the fly, even if a tokenizer path is explicitly provided by the user.

This behavior leads to extreme delays — in some cases, the tokenizer **takes over 2 hours to load or never finishes**.

---

### 🔸 **Problem Summary**:

* The current implementation always includes `gguf_file` in the tokenizer kwargs, which causes Hugging Face to use the embedded GGUF tokenizer.
* This embedded tokenizer rebuilds merges on the fly, causing significant loading delays.
* If the user passes an external tokenizer (e.g., from `tokenizer_dir`), it should be respected, and `gguf_file` should not be added.

---

### ✅ **Expected Behavior**:

* If a user provides a tokenizer, the code should **not** include `gguf_file`.
* This avoids unnecessary rebuilding of merges and drastically speeds up tokenizer loading (from hours to seconds).

---

###  **Logs**:

![Image](https://github.com/user-attachments/assets/51dd5472-2724-459c-9154-e666ed545833)

### 🛠 **Proposed Fix**:

Update https://github.com/EleutherAI/lm-evaluation-harness/blob/fea4d11d314959721b01ab58d9307a478343bd29/lm_eval/models/huggingface.py#L730 with the following logic:

```python
# gguf format embeds tokenizer and is not compatible with hf tokenizer `use_fast` param
if gguf_file is not None and not tokenizer:
    kwargs["gguf_file"] = gguf_file
```

This simple check ensures that if the user supplies a tokenizer, the embedded GGUF one is skipped.

---

### 🧪 **Result**:

Applying this fix:

* Prevents fallback to GGUF tokenizer when not needed.
* Avoids "merges building on the fly."
* Reduces tokenizer load time from hours to under a minute.

---

Happy to submit a PR if you’d like — thanks for maintaining this great tool!

Hi! Thanks for reporting this, and yes will appreciate a PR!
Repository: EleutherAI/lm-evaluation-harness
Issue: Any recommanded way to check attention scores?
Root cause: Documentation Deficiency

Hi,

I wonder whether there is any recommended way to return the attention scores for checking or fine-grained evaluation. 

It seems none of the current `output_type` could receive attention scores. Is there any example or tutorial on how to implement customized `output_type` and as well as `metrics`?

Many thanks.


A simple way currently could be to use the caching implementation (uses this [wrapper](https://github.com/EleutherAI/lm-evaluation-harness/blob/ff41a85631249aec89b57724cb9795bd49fe87e4/lm_eval/api/model.py#L232)) with `--use_cache <file.db>` and [save](https://github.com/EleutherAI/lm-evaluation-harness/blob/ff41a85631249aec89b57724cb9795bd49fe87e4/lm_eval/models/huggingface.py#L1296) the attention scores along with outputs.

Thank you very much, it is really helpful.
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue with the coqa task
Root cause: Validation Gap

The current `main` branch seems to have an issue with the coqa task. It was working in `lm-eval==0.2.0`.

```
Traceback (most recent call last):
  File "../../tasks/eval_harness/evaluate.py", line 9, in <module>
    from lm_eval import evaluator, tasks, utils
  File "/home/xiaoxiawu/.local/lib/python3.8/site-packages/lm_eval/evaluator.py", line 7, in <module>
    import lm_eval.tasks
                          File "/home/xiaoxiawu/.local/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 10, in <module>
    from . import coqa
  File "/home/xiaoxiawu/.local/lib/python3.8/site-packages/lm_eval/tasks/coqa.py", line 14, in <module>
    import lm_eval.datasets.coqa.coqa
ModuleNotFoundError: No module named 'lm_eval.datasets.coqa'
```

Previously noted by https://github.com/microsoft/Megatron-DeepSpeed/issues/40

Tagging @stas00 for his awareness.

I suspect that the issue is that the [custom datasets directory](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/datasets) has an empty [init.py](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/datasets/__init__.py).

Related PR fix: #309 
@leogao2 Any chance you can make a final review of this? Thanks!

Fixed after merging #309
Repository: EleutherAI/lm-evaluation-harness
Issue: Refactor CLI to subcommand structure

This issue documents our refactor of the CLI to a subcommand-based architecture in #2893, part of the broader effort to make the repo more modular and maintainable (see meta issue #3083).

The main driver was the need to load CLI commands from YAML configuration files (thanks to @artemorloff for the original implementation!), as complex commands from the CI were getting unwieldy. The new structure also gives us cleaner extension points and will be much easier to maintain going forward.

 new sub-commands:
- `lm_eval run`: Execute model evaluations (replaces the old default behavior)
- `lm_eval ls`: List available tasks, groups, and tags
- `lm_eval validate`: Validate task configs

Note:
- CI commands will override those from YAML config files
- The old `lm-eval --model hf ...` syntax still works for backward compatibility
Repository: EleutherAI/lm-evaluation-harness
Issue: Truncating from the left by default [QUESTION]
Root cause: Architectural Constraint

Why truncate explicitly from the left? This, in some cases, completely strips aways to query context from the start of the prompt.

The specific code is in lm_eval/models/huggingface.py:
```
                # when too long to fit in context, truncate from the left
                if self.backend == "causal":
                    total_length = len(context_enc) + len(continuation_enc)
                    if total_length > self.max_length + 1:
                        eval_logger.warning(
                            f"Combined length of context ({len(context_enc)}) and continuation ({len(continuation_enc)}) "
                            f"exceeds model's maximum length ({self.max_length}). "
                            f"Truncating {total_length - self.max_length + 1} tokens from the left."
                        )
                    inp = torch.tensor(
                        (context_enc + continuation_enc)[-(self.max_length + 1) :][:-1],
                        dtype=torch.long,
                        device=self.device,
                    )
                    (inplen,) = inp.shape
```

This would be useful if it's a boolean one passes in to choose which truncation one prefers, I think. Most tasks have the question at the start of the prompt, not the end, in which case left truncation makes sense.

I guess the edge case is not to truncate the continuation, of course.

I can see why this is implemented as left truncation: for loglikelihood you want to keep `continuation_enc` intact and avoid chopping off the target itself, so you sacrifice context on the left when over the limit.

That said, this can be problematic for prompts where the main question or instruction lives at the beginning, since that context is exactly what gets dropped. I agree that it would be very useful to have a configurable truncation side/strategy (e.g., left vs right) instead of hard-coding left truncation.
Repository: EleutherAI/lm-evaluation-harness
Issue: hendrycks MATH remove_boxed() does not handle \fbox{} despite fallback logic in last_boxed_only_string()
Root cause: Algorithmic Error

In the hendrycks_math/utils.py, the function last_boxed_only_string() correctly falls back to \fbox{...} if \boxed{...} is not present. However, remove_boxed() only handles \boxed{} but not \fbox{} . Is this intentional?
Repository: EleutherAI/lm-evaluation-harness
Issue: Attempted to load model 'hf', but no model for this name found
Root cause: Environment Incompatibility

I installed the harness following the instructions-

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

And then attempted to evaluate the example model 

lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

But I get the error- 


Traceback (most recent call last):
  File "/mnt/task_runtime/llm_venv/lib/python3.10/site-packages/lm_eval/api/registry.py", line 36, in get_model
    return MODEL_REGISTRY[model_name]
KeyError: 'hf'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/task_runtime/llm_venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/mnt/task_runtime/llm_venv/lib/python3.10/site-packages/lm_eval/__main__.py", line 516, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/mnt/task_runtime/llm_venv/lib/python3.10/site-packages/lm_eval/utils.py", line 440, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/task_runtime/llm_venv/lib/python3.10/site-packages/lm_eval/evaluator.py", line 226, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/mnt/task_runtime/llm_venv/lib/python3.10/site-packages/lm_eval/api/registry.py", line 38, in get_model
    raise ValueError(
ValueError: Attempted to load model 'hf', but no model for this name found! Supported model names: local-completions, local-chat-completions, openai-completions, openai-chat-completions.

Any help appreciated! 

Hi! I can't [reproduce](https://colab.research.google.com/drive/1YQN1Zy1nWKPPBJ4wgln2wcUjGl_t1WPb?usp=sharing) this. Have you tried using a fresh environment?

Thanks for looking into it! Re-doing the whole process on a new environment works. I am not sure what caused it initially. 

I am seeing the same error upon starting in a fresh environment. Does transformers version affect whether hf model is registered or not?
Repository: EleutherAI/lm-evaluation-harness
Issue: FileNotFoundError: Unable to find package root within 3 upwardsof
Root cause: Validation Gap

When I ran the following command, an error occurred.

model_name=gsm8k
servername=http://localhost
OPENAI_API_KEY=EMPTY lm_eval --model local-chat-completions --tasks gsm8k --model_args model=${model_name},base_url=${servername}:8008/v1 --check_integrity



Traceback (most recent call last):
  File "/nfs-15/fengxiaobin/miniforge3/envs/llm_eval/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/nfs-15/fengxiaobin/Open_source_project/lm-evaluation-harness-main/lm_eval/__main__.py", line 450, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/nfs-15/fengxiaobin/Open_source_project/lm-evaluation-harness-main/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/nfs-15/fengxiaobin/Open_source_project/lm-evaluation-harness-main/lm_eval/evaluator.py", line 330, in simple_evaluate
    run_task_tests(task_list=tasks)
  File "/nfs-15/fengxiaobin/Open_source_project/lm-evaluation-harness-main/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/nfs-15/fengxiaobin/Open_source_project/lm-evaluation-harness-main/lm_eval/evaluator_utils.py", line 541, in run_task_tests
    package_root = find_test_root(start_path=pathlib.Path(__file__))
  File "/nfs-15/fengxiaobin/Open_source_project/lm-evaluation-harness-main/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/nfs-15/fengxiaobin/Open_source_project/lm-evaluation-harness-main/lm_eval/evaluator_utils.py", line 529, in find_test_root
    raise FileNotFoundError(
FileNotFoundError: Unable to find package root within 3 upwardsof /nfs-15/fengxiaobin/Open_source_project/lm-evaluation-harness-main/lm_eval/evaluator_utils.py
Repository: EleutherAI/lm-evaluation-harness
Issue: The "hellaswag_multilingual" in OKapi is missing the Chinese (zh) script.
Root cause: Unimplemented Feature Gap

The "[hellaswag_multilingual](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/okapi/hellaswag_multilingual) " in OKapi is missing the Chinese (zh) script. However, I notice that their paper do exist the chinese evaluation, and other tasks under OKapi also have the zh script.
Repository: EleutherAI/lm-evaluation-harness
Issue: Testing  Llama-2-7b-chat-hf with pubmedqa  yields abnormal results
Root cause: Documentation Deficiency

I tested the Llama-2-7b-chat-hf model using the pubmedqa test set, but the result was very high, reaching 0.734. Does anyone know where I went wrong.

start command:
CUDA_VISIBLE_DEVICES=1 lm_eval --model hf \
    --model_args pretrained=/Models/Llama-2-7b-chat-hf,dtype=auto,add_bos_token=True\
    --gen_kwargs temperature=0.0,top_p=1.0,do_sample=False \
    --tasks pubmedqa \
    --batch_size auto \
    --num_fewshot 0 \
    --log_samples \
    --output_path ./results/llama-2-7b-chat-hf_20250709_pubmedqa.log \

Result:
Running loglikelihood requests:   0%|          | 0/1500 [00:00<?, ?it/s]
Running loglikelihood requests:   0%|          | 1/1500 [00:21<8:55:17, 21.43s/it]
Running loglikelihood requests:  13%|█▎        | 193/1500 [00:23<01:58, 11.00it/s]
Running loglikelihood requests:  26%|██▌       | 385/1500 [00:26<00:49, 22.56it/s]
Running loglikelihood requests:  38%|███▊      | 577/1500 [00:28<00:26, 34.49it/s]
Running loglikelihood requests:  51%|█████▏    | 769/1500 [00:30<00:15, 46.64it/s]
Running loglikelihood requests:  64%|██████▍   | 961/1500 [00:31<00:09, 59.00it/s]
Running loglikelihood requests:  77%|███████▋  | 1153/1500 [00:33<00:04, 71.86it/s]
Running loglikelihood requests:  90%|████████▉ | 1345/1500 [00:34<00:01, 89.45it/s]
Running loglikelihood requests: 100%|██████████| 1500/1500 [00:34<00:00, 43.51it/s]
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
2025-07-09:23:20:49 INFO     [loggers.evaluation_tracker:209] Saving results aggregated
2025-07-09:23:20:49 INFO     [loggers.evaluation_tracker:298] Saving per-sample results for: pubmedqa
Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 64
hf (pretrained=/Models/Llama-2-7b-chat-hf,dtype=auto,add_bos_token=True), gen_kwargs: (temperature=0.0,top_p=1.0,do_sample=False), limit: None, num_fewshot: 0, batch_size: auto (64)
| Tasks  |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
|--------|------:|------|-----:|------|---|----:|---|-----:|
|pubmedqa|      1|none  |     0|acc   |↑  |0.734|±  |0.0198|

Hi! Do you have a reference to check against?

In Table one of  [Safemarge](https://arxiv.org/abs/2503.17239) orginal socre on llama-2-7b-chat is 55.20 .
Repository: EleutherAI/lm-evaluation-harness
Issue: failed: unindent does not match any outer indentation level (<string>, line 13)
Root cause: Validation Gap

Hello, I hope you can help me solve this problem. I evaluated llama3.1-8B-Instruct on humaneval_instruct, and my pass@1 is 0. I found this indentation problem in the output result. How should I solve it? Thank you very much for helping me. Here is my command.

`HF_ALLOW_CODE_EVAL=1 lm_eval --model hf \
--model_args pretrained=/SSD/00/zmy/LLM/Meta-Llama-3.1-8B-Instruct/,trust_remote_code=True \
--tasks humaneval_instruct \
--device cuda:0 \
--num_fewshot 0 \
--limit 1 \
--output_path ./results/humaneval_0shot.json \
--gen_kwargs temperature=0.0,top_p=1.0
--verbosity debug \
--confirm_run_unsafe_code`

What this looks to me is that your model generates invalid code during the evaluation, not an issue with the LM Eval Harness itself.

However without the actual error messages or example output it's not really easy to figure out what went wrong.
Repository: EleutherAI/lm-evaluation-harness
Issue: Will MATH and MATH500 be added?
Root cause: Unimplemented Feature Gap

questions is in title

Several MATH variants are supported already. Have you tried searching the repo for "MATH"?

+1 I searched https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks and there seems no MATH500. Therefore I would appreciate it if I could know what is its name / will it be added

We already have MATH implemented. Is MATH500 just a subset but otherwise identical to the original Hendryks MATH dataset (and therefore should be evaluated the same way)?

IIRC yes

Currently there is a PR for improving the answer extraction of MATH, once that is merged I see no issue in adding the MATH500 task.