zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [BUG] bug prevents mixing batched and non-batched metrics
Root cause: Algorithmic Error

## Describe the bug
The `apply_metric` function in `lighteval/metrics/__init__.py` has a bug that prevents mixing batched and non-batched metrics. When both types of metrics are present in the same evaluation, the function fails to correctly compute and combine results from both metric types, leading to incomplete or incorrect metric outputs.

## Root Cause
The `apply_metric` function has a logical flaw: it uses a single `for metric in metrics:` loop with an `if/else` statement that only executes one branch per function call. If the first metric is batched, it will process all batched metrics but completely skip all non-batched metrics. If the first metric is non-batched, it will try to process all metrics (including batched ones) as non-batched, calling them incorrectly and ignoring their batching requirements. The function currently assumes all metrics would be of the same type rather than handling mixed batched/non

## To Reproduce
1. Create a list of metrics that includes both batched metrics (where `metric.batched_compute = True`) and non-batched metrics (where `metric.batched_compute = False`)
2. Call `apply_metric(responses, docs, metrics)` with this mixed list of metrics
3. Observe that the function either:
   - Only processes one type of metric (batched or non-batched) 
   - Produces incorrect output structure
   - Fails to combine results from both metric types properly

**Minimal working example:**
```python
from lighteval.metrics import apply_metric

# Assume we have:
# - batched_metric with batched_compute = True
# - non_batched_metric with batched_compute = False
metrics = [batched_metric, non_batched_metric]

# This will not work correctly with the buggy version
results = apply_metric(responses, docs, metrics)
```

## Expected behavior
The function should correctly handle a mixed list of batched and non-batched metrics by:
1. Separating metrics into batched and non-batched groups
2. Computing batched metrics once for all samples
3. Computing non-batched metrics individually for each sample
4. Combining results from both types into a unified output structure where each sample gets results from all metrics

The output should be a list of dictionaries, one per sample, containing the combined results from all metrics regardless of their batching behavior.

## Version info
- Operating system: Linux
- lighteval version: commit 7ed2636ed53142c0cf5e4aa752970ae62bcf1c24 (before the fix)
- Environment: pip/conda environment with lighteval
Repository: huggingface/lighteval
Issue: [BUG] lighteval does not support openai endpoints but says it does
Root cause: Documentation Deficiency

## Describe the bug
In the README and when running help, there are references to openai endpoint support, but it appears to no longer exist.
```
$ lighteval endpoint --help
                                                                                                        
 Usage: lighteval endpoint [OPTIONS] COMMAND [ARGS]...                                                  
                                                                                                        
 Evaluate models using some endpoint (tgi, inference endpoint, openai) as backend.                      
                                                                                                        
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Evaluation Backends ────────────────────────────────────────────────────────────────────────────────╮
│ inference-endpoint    Evaluate models using inference-endpoints as backend.                          │
│ tgi                   Evaluate models using TGI as backend.                                          │
│ litellm               Evaluate models using LiteLLM as backend.                                      │
│ inference-providers   Evaluate models using LiteLLM as backend.                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

```
$ lighteval endpoint openai --help
Usage: lighteval endpoint [OPTIONS] COMMAND [ARGS]...
Try 'lighteval endpoint --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ No such command 'openai'.                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

## Expected behavior
If the openai/openai-compatible support was removed, please remove it from your documentation or indicate clearly that it is no longer supported.


hi ! thanks for the notice, we did forget to remove some ref to openai endpoints. we used to have it but it is now managed by litellm endpoint, will remove :)
Repository: huggingface/lighteval
Issue: [FT] Improvements for LLM judges with litellm backend
Root cause: Unimplemented Feature Gap

## Issue encountered
As part of a community task I have been collaborating on, I've encountered various challenges with the litellm judge backend. I've already implemented the changes and will open a PR right after posting this issue. The challenges were:

- Judge outputs are currently not cached. If the evaluation script fails, the responses from the judge need to be regenerated, leading to higher evaluation costs.
- Not all inference providers support the same set of parameters when generating chat completions.
- The maximum number of generated tokens is currently hardcoded to 512 in the litellm backend
- Currently, 100 concurrent requests are performed, potentially leading to the client running into rate limits. 

## Solution/Feature

The `JudgeLM` and `JudgeLLM` constructors now accept a `backend_options` parameter (dict). In case of the litellm backend, this is then converted to a new dataclass `LitellmBackendOptions`, which allows to specify whether to use caching or not, how many concurrent requests should be performed, and whether to increase the number of output tokens in case of reasoning models or not.

Additionally, the litellm backend will now by default ignore chat completion arguments that are not supported by the currently used inference provider. The `max_tokens` parameter is now respected by the litellm backend instead of using a hardcoded value.
Repository: huggingface/lighteval
Issue: [FT] Additional LiteLLM model config options and better context length estimate
Root cause: Unimplemented Feature Gap

## Issue encountered
Currently, there are a few things that are note exposed through the `LiteLLMModelConfig` that can be very useful when running evaluations:

- It would be nice to have a `verbose` flag, in case you want to debug something related to litellm
- If one knows the maximum context length of your model, it would be nice if we could set that instead of relying on the default length of 4096 that is currently hardcoded in the `max_length` property
- Different APIs can differ in their robustness, and they might have different rate limits. It would be nice to configure the number of retries that are performed when calling the API, as well as the waiting time in between requests, and maybe a timeout if a request takes too long.

Additionally, it would be nice to apply the current strategy for the o1 model in `_prepare_max_new_tokens` to other reasoning models as well.

## Solution/Feature
I created a PR that implements the suggested changes above. It introduces new options in the `LiteLLMModelConfig`:

```python
"""
(...)
verbose (bool):
    Whether to enable verbose logging. Default is False.
max_model_length (int | None):
    Maximum context length for the model. If None, infers the model's default max length.
api_max_retry (int):
    Maximum number of retries for API requests. Default is 8.
api_retry_sleep (float):
    Initial sleep time (in seconds) between retries. Default is 1.0.
api_retry_multiplier (float):
    Multiplier for increasing sleep time between retries. Default is 2.0.
timeout (float):
    Request timeout in seconds. Default is None (no timeout).
(...)
"""
```

The increase in the allowed number of tokens is now calculated for all models that are recognized as reasoning models by litellm (as indicated by their `supports_reasoning` function). Instead of having hardcoded upper bounds, we use litellm's `get_max_tokens` helper function, or, if this fails, we query the maximum context length from different endpoints on OpenRouter. If the specified provider is present in that list, we get the information right from OpenRouter. Otherwise, we will choose the minimum context length among all OpenRouter providers to ensure that it works at least with all providers listed there. If this also fails, we will return the default context length of 4096, the same one as currently hardcoded.

In order to use the `suggest_reasoning` function of litellm, I had to update the minimum required version of litellm in the pyproject.toml file to 1.66.0.
Repository: huggingface/lighteval
Issue: To remember for version upgrades
GitHub labels: to-keep

Theoretically everything up to 1.0.0 is considered unstable and prone to change at any time:

> Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.

At Hugging Face, we try and apply the rule that while we have major version zero, minor releases (`0.x.0`) may add new features and break things (the equivalent of a major release), while patch releases (`0.0.x`) behave very closely to patches as understood by semantic versioning:

> Patch version Z (x.y.Z | x > 0) MUST be incremented if only backward compatible bug fixes are introduced. A bug fix is defined as an internal change that fixes incorrect behavior.

---

Now for development versions, this is a bit out of the control of semantic versioning IMO but the way that we do it in `transformers`/`diffusers`/`huggingface_hub` is to update the version defined in the init to the "development of the next version".

What that means is that:

For `transformers`, where there are no plans to go to v5 for now, we would do as such:

1. We release version `v4.38.0`
2. The next version we'll release, if everything goes well, is going to be `v4.39.0`
3. We put `v4.39.0.dev0` as we're developing that next version.

We're not putting `v4.38.1.dev0` as that would mean "we're preparing for the upcoming patch; patch which may or may not happen, and that should only contain bugfixes and no new features.

For `huggingface_hub`, there are no plan to go to v1 for now, so:

1. We release version `v0.21.0`
2. The next version we'll release, if everything goes well, is going to be `v0.22.0`
3. We put `v0.22.0.dev0` as we're developing that next version.

If, however, we were planning on releasing a v1 for `huggingface_hub`, we might instead put `v1.0.0.dev0`. We won't want to put `v0.21.1.dev0` at any point however.

---

Finally, we only put `.dev0` right now (and no `.dev1`, `.dev2`, etc), but we could eventually add them with patches as separators. So we'd go:
- `v4.38.0` Released
- `v4.39.0.dev0` in the init
- `v4.38.1` Released
- `v4.39.0.dev1` in the init
Here what's most important is that you stick to it, imo.

_Originally posted by @LysandreJik in https://github.com/huggingface/lighteval/pull/77#discussion_r1514537715_
Repository: huggingface/lighteval
Issue: How to use a configuration file?
Root cause: Documentation Deficiency

The documentation makes references to using configuration yaml files like [here](https://huggingface.co/docs/lighteval/main/en/use-litellm-as-backend) but it doesn't give the name of the file or which option to feed the config to lighteval. I tried making a `config.yaml`, `config.yml` in the current directory and trying a `--config` option (doesn't exist).

hi ! simply pass it in place of the model config. You can find how to use it with:

`lighteval accelerate --help`
Repository: huggingface/lighteval
Issue: [EVAL] Add MathVista
Root cause: Unimplemented Feature Gap

## Evaluation short description
- Why is this evaluation interesting?
MathVista is one of the most popular and commonly reported benchmark in MultiModal LLM like Qwen2.5_VL and InternVL3_5
- How used is it in the community?
To evaluate multimodal vision reasoning capabilities on Math questions

## Evaluation metadata
Provide all available
- Paper url:https://arxiv.org/pdf/2310.02255
- Github url: https://github.com/lupantech/MathVista
- Dataset url: https://huggingface.co/datasets/AI4Math/MathVista

https://mathvista.github.io/


hey @ThakurRajAnand @NathanHB can i work on this?
Repository: huggingface/lighteval
Issue: [EVAL] Add SLR-Bench
Root cause: Unimplemented Feature Gap

## Evaluation short description
- Why is this evaluation interesting?
Scalable Logical Reasoning Benchmark contains 19k+ prompts spanning 20 curriculum levels
Systematic progression, from simple attribute checks to complex recursion
Ideal for both model evaluation and curriculum-based model training
While models readily solve the initial levels, performance drops as complexity increases
- How used is it in the community?
To evaluate reasoning abilities of today’s Large Langauge Models

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2506.15787
- Github url: https://github.com/ml-research/ScalableLogicalReasoning
- Dataset url: https://huggingface.co/datasets/AIML-TUDA/SLR-Bench
Repository: huggingface/lighteval
Issue: Push details to hub does not work
Root cause: Validation Gap

Removing the `all` section of the result file has broken the create metadata card mechanism.

To reproduce:

```
 python  run_evals_accelerate.py --model_args "pretrained=HuggingFaceH4/zephyr-7b-beta" --tasks "lighteval|triviaqa|0|0"  --override_batch_size 1  --output_dir="tmp/"  --max_samples 1 --save_details --push_details_to_hub --results_org open-llm-leaderboard --push_results_to_hub
```

Also, having `datasets` version below 2.18.0 yields an error when fetching some files for the details dataset.

Good catch!
Repository: huggingface/lighteval
Issue: [BUG] TInyBenchmark metrics need conversion from string to Metric
Root cause: Validation Gap

## Describe the bug
When running `lighteval tasks inspect "lighteval|gsm8k|0"`, an error is encountered: `'str' object has no attribute 'category'`.  This is because the tinyBenchmark tasks metrics have not been updated to be `Metric` classes (https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/extended/tiny_benchmarks/main.py#L259) 

## To Reproduce
Run `lighteval tasks inspect "lighteval|gsm8k|0"` in the command line.

## Expected behavior
Command successfully executes and task information for gsm8k is displayed to the user.

## Version info
Commit 5137e03 on main
Repository: huggingface/lighteval
Issue: [BUG] Handle BERTScore baseline files
Root cause: Unimplemented Feature Gap

## Describe the bug
The default BERTScorer has a `rescale_with_baseline` argument that is set to `True`. When this flag is set, the `BERTScorer` class requires a `baseline_path` to be set, otherwise[ it will attempt to load TSV files](https://github.com/huggingface/lighteval/blob/main/src/lighteval/metrics/imports/bert_scorer.py#L376) that are expected to be in the `src/lighteval/metrics/imports` directory.

## To Reproduce
Requires #990 to be applied to fix overflow error.

```python
from lighteval.models.dummy.dummy_model import DummyModelConfig
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters


def main():
    task = "helm|summarization:xsum|0"

    model_config = DummyModelConfig(
        model_name="dummy",
        seed=1234,
    )

    pipeline_params = PipelineParameters(
        launcher_type=ParallelismManager.NONE,
        max_samples=1
    )

    evaluation_tracker = EvaluationTracker(
        output_dir="./results",
        save_details=True,
        push_to_hub=False,
    )

    pipeline = Pipeline(
        tasks=task,
        pipeline_parameters=pipeline_params,
        evaluation_tracker=evaluation_tracker,
        model_config=model_config,
    )
    pipeline.evaluate()

if __name__ == "__main__":
    main()
```

## Expected behavior
Either a warning is displayed to the user to let them know that the baselines need to be manually downloaded and where they can be acquired, or lighteval will handle the management of loading the baseline files. It looks like the files are downloaded from [here](https://github.com/Tiiiger/bert_score/tree/master/bert_score/rescale_baseline). 

I'm happy to create a PR to fix this, but need to know the preferred method.

## Version info
`main` on `c7a063aee58dc14f51841a6b4f977cd914e10a8f`
Repository: jitsi/jiwer
Issue: Getting different results based on punctuation
Root cause: Validation Gap

Is this intended behavior?

```python
print(wer('hello', 'hello.'))
print(wer('hello', 'hello!'))
```

Gives

```
0.0
1.0
```

No, when I worked on this the speech-to-text engine I used did not insert `!` or `?`. I will add it to the list of characters which are filtered out. 

@nikvaessen imho this should not happen at all - or it should be possible to disable this behavior. 

`wer()` should take an iterable and calculate the WER based on those elements. The input should not be modified any further - also no lower-casing. Imho one should not assume that this is what the caller of `wer()` wants. It's the callers' responsibility to pass in preprocessed input.

Thanks for this input. I've updated the library so that it's easier for users to control what pre-processing is done.
Repository: jitsi/jiwer
Issue: Could someone please explain what is difference between wer(ref, hypo) when I concatenate string and wer(ref_list, hypo_list)?
Root cause: Documentation Deficiency

I am trying to calculate wer between model transcriptions and ground truths but am getting too much difference in WER, sometime even surpassing 5%! The wer using list gives better wer than concatenated string. Could someone be kind enough to explain the difference and which is better to use? Thank you!

You can try to visualize the alignment and spot why the amount of sub/del/ins decrease when you concatenate strings. 

See https://jitsi.github.io/jiwer/reference/alignment/#alignment.visualize_alignment.

Generally, if the sentences are independent, you should not concatenate them.
Repository: jitsi/jiwer
Issue: Does the jiwer 4.0 have not be released？

Does the jiwer 4.0 have not be released？

I am waiting for https://github.com/huggingface/evaluate/pull/661 to be merged.

Hi @nikvaessen, сould you please specify when this will be released? I only just got familiar with this package today, and it’s really great, thank you! However, I was a bit misled by the documentation, as it lists functionality from version 4, such as new methods in the alignments file and some new properties, but they are actually not yet available for use ((

Feel free to add version 4.0 to your project by `uv add git+https://github.com/jitsi/jiwer` while I wait for my changes to be merged by the huggingface team.

It is now released on pipy.
Repository: jitsi/jiwer
Issue: Implement a `NoTransform` operation
Root cause: Documentation Deficiency

what would you think about adding a "NoTransform" placeholder class ?

This would be used, if user wishes to do all the transforms himself, example here:
https://github.com/KarelVesely84/jiwer/commit/1c5f29d7de590ad3d00710a7535122be99d6b0ee#diff-aae25c8b196777e4abeb01845ca517591f65580977ee7d1f639b5bbf16e26d9d
(plus a registration is necessary in `transforms.py` header to `__all__` variable)

Or can there be something similar already in the code ?

Thank you & best regards
Karel

_Originally posted by @KarelVesely84 in https://github.com/jitsi/jiwer/issues/108#issuecomment-2655917039_
            

This behavior can currently be done with:

```python3
import jiwer

ref = [['already', 'transformed']]
hyp = "needs to be transformed"

jiwer.process_words(ref, hyp, reference_transform=jiwer.Compose([]))
```

Can you specify why the default transformation is problematic in your case? It is quite minimal.

```python3
wer_default = tr.Compose(
    [
        tr.RemoveMultipleSpaces(),
        tr.Strip(),
        tr.ReduceToListOfListOfWords(),
    ]
)
```

Aha, I see, you are right. 
(Maybe I am just a little "obsessed" by doing things on my own)

Thank you for the suggestion.
K.
Repository: jitsi/jiwer
Issue: jiwer with wrong numpy version induces hang without reported wrong information
Root cause: External Dependency Breakage

Firstly, with jiwer=1.3.2 and numpy=2.0.2, 
wer = jiwer.wer(truth, hypothesis, standardize=True) would stuck in
Traceback (most recent call last):
File "/root/conformer/EfficientConformer/main.py", line 227, in <module>
     main(args)
   File "/root/conformer/EfficientConformer/main.py", line 128, in main
     model.fit(dataset_train, 
    ^^^^^^^^^^^^^^^^^^^^^^^^
   File "/root/conformer/EfficientConformer/models/model.py", line 313, in fit
     wer, truths, preds, val_loss = self.evaluate(dataset, val_steps, verbose_val, eval_loss=True)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/root/conformer/EfficientConformer/models/model.py", line 502, in evaluate
     wer = jiwer.wer(speech_true, speech_pred, standardize=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/root/miniconda3/lib/python3.12/site-packages/jiwer/wer.py", line 75, in wer
     distance = _edit_distance(t, h)
                ^^^^^^^^^^^^^^^^^^^^
   File "/root/miniconda3/lib/python3.12/site-packages/jiwer/wer.py", line 252, in _edit_distance
     m[i, j] = min(
               ^^^^
But it never reported wrong information

Solution:
update jiwer's version and
change
wer = jiwer.wer(truth, hypothesis, standardize=True)
into
transformation = jiwer.Compose([
    jiwer.RemoveWhiteSpace(),
    jiwer.ToLowerCase(),  
    jiwer.ReduceToListOfListOfWords(word_delimiter=" ")  
])
wer = jiwer.wer(truth, hypothesis, truth_transform=transformation, hypothesis_transform=transformation)
Repository: jitsi/jiwer
Issue: v3.1.0 src/jiwer/alignment.py have no visualize_error_counts() and collect_error_counts methods，why
Root cause: Documentation Deficiency

v3.1.0 src/jiwer/alignment.py have no visualize_error_counts() and collect_error_counts methods，why

Hi, I had the same problem because the method cannot be found with the code from the documentation:

```
import jiwer
print(jiwer.visualize_error_counts(out))
```

I suspect that this is a bug.

However, the implementation actually exists for both methods you mentioned. As a workaround I just copied the whole file and imported the methods from there: https://github.com/jitsi/jiwer/blob/c1b0d5e005431f5ce4fa6797f48639a8ccaa5042/src/jiwer/alignment.py#L266
https://github.com/jitsi/jiwer/blob/c1b0d5e005431f5ce4fa6797f48639a8ccaa5042/src/jiwer/alignment.py#L229

Solved now that version 4.0 is released on pipy.
Repository: jitsi/jiwer
Issue: Releasing jiwer-gml as a conda package
Root cause: Unimplemented Feature Gap

Would it be possible to make jiwer-gml available as a conda package?

I do not own/control https://pypi.org/project/jiwer-gml/. This repository is published as https://pypi.org/project/jiwer/

Sorry, they link to this repository on [jiwer-gml](https://pypi.org/project/jiwer-gml/).

As far as I know, jiwer-gml is an exact fork of jiwer published by someone else (I don't know why). Anyway, even the `jiwer` version on conda is not published by me. 

Okay, thank you