Repository: stanford-crfm/helm
Issue: Allow Skipping `summ_eval` Metrics When Using LLMJury
Root cause: Architectural Constraint

Currently, when running HELM with summarization metrics enabled, the framework attempts to import and use `summ_eval`. If the module is not installed, the run fails with:

```text
ModuleNotFoundError: No module named 'summ_eval'
helm.common.optional_dependencies.OptionalDependencyNotInstalled: Optional dependency summ_eval is not installed.
```

This happens **even when configuring evaluation with `LLMJury`**, which does not require `summ_eval`.

---

**Problem:**

* Users who want to evaluate summarization with `LLMJury` still need to install `summ_eval` due to hard imports in `summarization_metrics.py`.
* This adds unnecessary dependencies and introduces version conflicts (e.g., `Blanc` requiring `torch>=1.0,<2.0`).
* The framework does not currently provide a way to selectively **skip or disable `summ_eval` metrics** when alternative metrics (like LLMJury) are chosen.

**Expected Behavior:**

* If `LLMJury` is explicitly configured as the metric provider, HELM should not attempt to import or initialize `summ_eval`.
* Summarization evaluation should proceed without requiring unused dependencies.
* Optional dependencies should be checked only if the specific metric is requested.


**Actual Behavior:**

* HELM tries to load `summ_eval` by default, regardless of whether `LLMJury` is configured.
* Runs fail unless users manually install `summ_eval` and its dependencies.

---

cmd

```shell
helm-run --run-entries aci_bench:jury_config_path=/home/redacted/judges.yaml --suite 0.0.1 --max-eval-instances 5 --local-path /home/redacted/config --models-to-run openai/gpt-4o-mini --output-path /home/redacted/benchmark_output --disable-cache
```

@yifanmai @MiguelAFH 
Can you please help me to solve this issue?

and my solution to resolve this issue:

[common_metric_specs.py](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/metrics/common_metric_specs.py#L109)

> [!TIP]
>```python
> def get_summarization_metric_specs(args: Dict[str, Any]) -> List[MetricSpec]:
>     try:
>         return [
>             MetricSpec(class_name="helm.benchmark.metrics.summarization_metrics.SummarizationMetric", args=args)
>         ] + get_basic_metric_specs([])
>     except ModuleNotFoundError as e:
>         handle_module_not_found_error(e, ["summarization"])
>         hwarn("or Configure the LLMJuryMetric, Skipping summarization metrics.")
>     finally:
>         return get_basic_metric_specs([])
> ```

Hi @chakravarthik27,

Can you try running:
```
pip install "crfm-helm[summarization,medhelm]"
```

This should install the missing dependencies. You’re right that LLMJury itself doesn’t require that module, but some of the other summarization metrics do.

We’ve also updated our [documentation](https://crfm-helm.readthedocs.io/en/latest/medhelm/)
 with step-by-step instructions for installing MedHELM. Could you let me know if you still see the same issue after running the command?

The `summ-eval` dependency is used in the codebase (see [[helm/benchmark/metrics/summarization_metrics.py#L69-L72](https://github.com/stanford-crfm/helm/blob/06f80f3df3e881fed3bbff7d6b8425909e47235b/src/helm/benchmark/metrics/summarization_metrics.py#L69-L72)](https://github.com/stanford-crfm/helm/blob/06f80f3df3e881fed3bbff7d6b8425909e47235b/src/helm/benchmark/metrics/summarization_metrics.py#L69-L72)), but it is not listed in `pyproject.toml`.

When adding it explicitly with `uv add "crfm-helm[summarization,medhelm]" `and  `uv add summ-eval`, it pulls in `blanc`, which in turn forces `torch<2.0`. This leads to version conflicts with modern PyTorch in the development environment.



@MiguelAFH