Repository: stanford-crfm/helm
Issue: Support for numpy 2.x
Root cause: External Dependency Breakage

I'm starting to play with this repo, and I'm noticing the dependencies are quite strict. Specifically numpy is restricted to the 1.x series, which is going to make it more and more difficult to integrate with other libraries as times goes on. I was wondering what the plan was for loosening / modernizing some of these dependencies? 

I think it's quite reasonable to loosen the numpy dependency. Looking at the [NumPy 2 migration guide](https://numpy.org/devdocs/numpy_2_0_migration_guide.html), I don't think any of the breaking changes will impact us.

I think its a bit more complex, as there are a lot of dependencies on other libraries that will be impacted. A quick upgrade of numpy in a venv where helm was working gives an immediate problem with matplotlib. However, after upgrading matplotlib and seaborn, I did get 291 tests to pass, but I expect there are code paths that aren't fully tested in a local pytest run.

I also tried updating versions of scipy, scikit-image, pandas, scikit-learn but started to get failures when I installed the "[all]" dependencies.

I submitted a test PR here: https://github.com/stanford-crfm/helm/pull/3681 to see what happens on the CI side.

Is HELM running all of its benchmarks in its own main Python processes? In other words, if a specific scenario corrupts the process will that break everything downstream in the pipeline? If so, it may be worth thinking about (at least optionally) containerizing specific scenarios so each can operate with their own special set of dependencies which may be incompatible. 

EDIT: the failures I saw locally might be due to incompatibility between version in cache files. Running things from scratch in the CI seemed to work with versions loosened for some packages.  
 
---

What I've historically done to ensure I have a stable version of packages as well as am supporting the latest and greatest from various libraries is use ">=" in as many of my dependencies as possible. Then in the CI, I install a "loose" version where I allow the latest version of each dependency to be installed. This ensures that if something changes upstream, I see a failure in the CI, and I can attempt to fix it, or pin the version with a ",<" if I need to punt and go back to it later.

On the flip side, I have a "strict" version where I replace all the ">=" with "==" requirements and that let's me test the known stable versions of all packages. It becomes easy to see when there is an upstream error because "loose" will fail and "strict" will pass.

As far as I know, none of the mainstream package managers support this paradigm. But it's not too hard to replace these requirements on the fly in CI.  Unfortunately, I don't have a tool for this at the moment, because when I designed a solution for the problem pyproject.toml wasn't popular yet, and I rely on setup.py to have custom code translate requirements.txt files into install requires on the fly.



Thanks for investigating; this is very helpful.

HELM runs everything in the same process. HELM doesn't have scenario containerization (or process separation, or other kinds of modularization), though I agree that this is a good idea for many reasons.

HELM have something similar to the loose / strict system. The loose dependencies are described in [setup.cfg](https://github.com/stanford-crfm/helm/blob/main/setup.cfg). Whenever this file is updated, we compile a lockfile using [`uv pip compile`](https://docs.astral.sh/uv/pip/compile/) with [this GitHub Action](https://github.com/stanford-crfm/helm/blob/230c5e2f6672da9e073d22fffe4a776c7d7fa2ae/.github/workflows/update-requirements.yml) into [requirements.txt](https://github.com/stanford-crfm/helm/blob/main/requirements.txt). This guarantees that there is _some_ version resolution that will pass the tests, but does not guarantee that every possible version resolution will pass the tests.

If I were starting a new project today, I would probably pick [uv](https://docs.astral.sh/uv/) for dependency management. I have also tried poetry, which does something similar, but I definitely prefer uv over poetry.

Lockfiles are similar, but differ in some key ways. Lockfiles are great for reproducing development environments, but they don't force a test of minimum versions. If `uv pip compile` had the ability to force all packages to the minimum allowed version, and you tested that minimum in each CI run, then that would be the same. 

I agree its combinatorially impossible to test every version combination, but as long as you have something that covers older versions, newer versions, minimal, and full installs, I think that's enough.

From what I see you have test.yaml which:

* Installs the minimal package from the setup.cfg requirements and tests it
* Runs the developer install which installs pinned requirements,  installs the package in development mode, and then runs tets.

So you are going to effectively get a "loose-minimal" test and a "strict-all" test. But you won't get "loose-all" and "strict-minimal". I think strict-minimal isn't that important, but I think a loose-all test will start to reveal errors as time goes by and packages evolve (especially with tensorflow as a dependency!).

Uv does have some nice tooling, and I use the basic pip and venv stuff on the regular now. I need to evaluate uv's dependency management a bit more. I never really liked poetry and I think `uv add` probably still won't get you there if you need to support a wide range of packages on a wide range of Python versions (which is important if you want to run arbitrary benchmarks, but then you run into the problem of benchamarks not having robust versioning and you wind up needing a containerization approach anyway).

FWIW, I did write a tool in my [xdev package](https://github.com/Erotemic/xdev) to help find minimal versions of packages for each Python version and then generate the appropriate requirements.txt entries for them. E.g. `xdev availpkg torch` will write out:

```
torch>=2.6.0     ; python_version < '4.0'  and python_version >= '3.13'    # Python 3.13+
torch>=2.2.0     ; python_version < '3.13' and python_version >= '3.12'    # Python 3.12
torch>=2.0.0     ; python_version < '3.12' and python_version >= '3.11'    # Python 3.11
torch>=1.11.0    ; python_version < '3.11' and python_version >= '3.10'    # Python 3.10
torch>=1.7.1     ; python_version < '3.10' and python_version >= '3.9'     # Python 3.9
torch>=2.6.0     ; python_version < '3.9'  and python_version >= '3.9.0'    # Python 3.9.0
torch>=1.7.0     ; python_version < '3.9.0' and python_version >= '3.8'     # Python 3.8
torch>=2.0.0     ; python_version < '3.8'  and python_version >= '3.8.0'    # Python 3.8.0
torch>=1.11.0    ; python_version < '3.8.0' and python_version >= '3.7.0'    # Python 3.7.0
torch>=1.0.0     ; python_version < '3.7.0' and python_version >= '3.7'     # Python 3.7
torch>=1.7.1     ; python_version < '3.7'  and python_version >= '3.6.2'    # Python 3.6.2
torch>=1.7.0     ; python_version < '3.6.2' and python_version >= '3.6.1'    # Python 3.6.1
torch>=1.0.0     ; python_version < '3.6.1' and python_version >= '3.6'     # Python 3.6
torch>=1.0.0     ; python_version < '3.6'  and python_version >= '3.5'     # Python 3.5
torch>=1.0.0     ; python_version < '3.5'  and python_version >= '2.7'     # Python 2.7
```

It's not perfect, but I manually fix it up based on what versions of Python and the package I actually want to support. But that gives me some control over the minimum package version for each Python version that currently `uv` and other dependency managers don't do.

Note: I don't think helm has to do any of this. The spirit of the issue is just to support more modern versions of packages. I'm just mentioning it as I think its still an interesting unsolved problem in the Python dependency managements space. 


FYI: It looks like uv has a resolution strategy that can handle these "loose" versions. I haven't tested it yet, but it looks like if you install with `uv pip install --resolution lowest-direct` then its minimizes the satisfying versions of the dependencies, which is nearly the same as "loose" (might be exactly the same; I'd have to think about it). 

https://docs.astral.sh/uv/concepts/resolution/#resolution-strategy