Repository: UKGovernmentBEIS/inspect_ai
Issue: Modern dependency management
Root cause: Architectural Constraint

In my view, the way this repo handles its dependencies is a weak point. Dependencies are largely unversioned.

This makes bugs like these an inevitability:
- https://github.com/UKGovernmentBEIS/inspect_ai/issues/695
- https://github.com/UKGovernmentBEIS/inspect_ai/issues/41

If you're open to it, I think it would be great to switch to `poetry` for dependency resolution (or other modern tooling if you have a preference). I'm happy to handle making the change (including necessary changes to CI workflows etc). But for such a change to the core tooling I'd want the maintainers' blessing before I make PR.



If you're unfamiliar with `poetry`, here are two key pages to get you started:

* https://python-poetry.org/docs/basic-usage/#installing-dependencies
* https://python-poetry.org/docs/dependency-specification/

There are several advantages to using `poetry`, but if you don't want to use new tools, I think the absolute minimum that should be done is to specify dependency versions more tightly using version specifiers like `>=`, `<`, `~=`. See https://peps.python.org/pep-0440/#version-specifiers

Our approach to dependency versions is to be liberal on purpose _until_ proven that we need a narrower dependency. The reason for this is that if too many libraries bound their versions then package solvers (conda, pip, uv, etc.) often can't resolve  situations where multiple packages are aggressively bounding. Alternatively, when upper bounds are used often solvers fall back to very old versions. We want to give solvers latitude to fit us into a deployment in an optimal fashion, again only applying version bounds when we have evidence that we need one.

The problem of excessive upper-bounding pinning is not unique to Poetry but it does seem to have been encouraged/facilitated by it. Here is a well reasoned critique of this practice that explains why it scales so poorly to a large ecosystem of libraries: https://iscinumpy.dev/post/bound-version-constraints/

Because of the above, packages necessarily need to take a different approach than deployments to versions. Deployments (i.e. your specific application and mix of packages) should hard lock onto package versions so you are reproducible and aren't exposed to random breakage. Packages though to be good citizens need to be as flexible as possible in their version requirements (even to the point of dynamically adapting to different versions of packages they depend on).

In terms of examples, this issue (https://github.com/UKGovernmentBEIS/inspect_ai/issues/41) was evidence for needing a narrower requirement so we applied it. For the Google issue it's an optional dependency but we do require at runtime version 0.8.1 or higher (the fact that they made a breaking change in 0.8.3 is hard to solve for in advance!). We could hard bind to 0.8.1 but that's arbitrary and users could be missing important bug fixes if we do that. I'm more inclined to rely on package authors to have some modicum of respect for semantic versioning (and then tack when they don't).

Currently, our core requirements.txt does not do very much version bounding (except for where we've seen evidence its required like `typing_extensions`, `s3fs`, etc.) I am open to adding some lower bounds here if there are known issues they will help us avoid. In terms of optional dependencies, there we are more aggressive about lower bounding (typically requiring package versions from the past few months). You can see those requirements applied here in the `MIN_VERSION` constants for each provider: https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/model/_providers/providers.py.







Thanks for the thoughtful reply!

Definitely agreed that deployments should install dependencies from a lockfile, while libraries should specify much looser (or possibly no) constraints. 

I especially liked the ISciNumpy.dev post. I wasn't familiar with the debate about whether upper bounds should ever be used, and it's shifted my perspective. Thanks!

I think there are two distinct questions for inspect:
1. Should the inspect repo use a lockfile (e.g. `poetry.lock`)?
2. How should dependencies be specified in `pyproject.toml`?

(Technically, a third question is: which dependency resolution algorithm is best? Poetry and pip each have their own dependency resolution algorithm. Historically, `pip` had a much less sophisticated dependency resolver. However, my vague impression is that with `pip>=22.0`, which uses `resolvelib`, it has caught up significantly. I was not able to quickly find more useful information about the pros and cons of modern `pip` and `poetry` resolution algorithms.)

**Should the inspect repo use a lockfile?**
My answer would be a strong yes. I think _any_ project that is (or ever might be) installed more than once should have a lockfile. This includes all but the most trivial projects. For example:

* if your project has more than one engineer working on it
* if you'll ever install the project on a new laptop
* if you have a CI pipeline

Without a lockfile, you do not have reproducible environments to develop the package or run tests. You are fully at the mercy of external changes in the dependency resolution. It's the dreaded 'it works on my machine'.

To express the same thing in terms of your deployment/library distinction: yes, inspect is a library, but every engineer's laptop, and GitHub actions, are 'deployments' that will be subject to random breakage without a lockfile.

Even for users of inspect who are not directly developing inspect, a lockfile is very useful when investigating a bug that might be caused by a dependency mismatch. The lockfile gives a 'known good' version of dependencies that inspect has been tested with. If we take the example of https://github.com/UKGovernmentBEIS/inspect_ai/issues/695, I suspected a dependency problem as the cause, but I had no way of checking what version of `google-generativeai` inspect had been tested with in your CI pipeline. Proving that the bug was (or that it was not) due to dependency versioning would have involved relatively painstaking trial and error, so I didn't do it. With a lockfile, this becomes trivial. In the case of https://github.com/UKGovernmentBEIS/inspect_ai/issues/695, I could have told you immediately in the issue report that it was a dependency specification problem.

**How should dependencies be specified?**

> We could hard bind to 0.8.1 but that's arbitrary and users could be missing important bug fixes if we do that.

I completely agree, we don't want to pin `==0.8.1`. I don't think anyone is advocating for pinning exact versions in `pyproject.toml`. I think that would be terrible! 

As far as I can tell, the positions in this debate are:

- The ISciNumpy.dev post argues that upper bounds should only be used when a newer version of a dependency is proven to be incompatible. Otherwise, they should never be used, not even for major versions.
- Poetry's default is [caret (`^`) versioning](https://python-poetry.org/docs/dependency-specification/#caret-requirements), which allows updates that are (supposed to be) compatible under semantic versioning.

I'm a bit confused about what your position is on this. Do you want to use semantic versioning or do you want to use no upper bounds? 

On the one hand, you linked to the ISciNumpy.dev post, and said in https://github.com/UKGovernmentBEIS/inspect_ai/issues/695 "we are going to avoid upper-bounding like the plauge". On the other hand, you said in this thread "I'm more inclined to rely on package authors to have some modicum of respect for semantic versioning", which makes it sound like you do want to use semantic versioning. These seem like contradictory positions. 

As I said, I got a lot out of the ISciNumpy.dev post, and it did shift my views. I'm not sure I'm persuaded of the strongest version of its claims (no upper-bounding at all). I think there is still a trade-off here, between breakage caused by the lack of upper bound, and breakage caused by insufficient flexibility.

The strongest argument made by the ISciNumpy.dev post is that overly specific constraints cannot be fixed downstream, while insufficiently specific constraints can be fixed downstream once identified:

> If a library requires pyparsing>=3 and another library requests pyparsing<3, that’s the end, you are out of business.

This means there is a strong asymmetry between the two types of breakage: breakage caused by being too specific, when it occurs, is much worse. However, this could still be worth it, if the breakage caused by being too specific is much rarer. I could imagine this being the case if you only restrict major versions.

In any event, if you have a strong view on this topic I'm happy to defer to it.



Also FWIW, I started having a brief look at what very popular Python packages (by stars on GitHub) do:

- `scikit-learn`: [no upper-bounds](https://github.com/scikit-learn/scikit-learn/blob/934df30a7f976e2fd902b39fb0ff1c026b9c230c/pyproject.toml#L9C1-L14C2) 
- `django`: [no upper-bounds](https://github.com/django/django/blob/99dcc59237f384d7ade98acfd1cae8d90e6d60ab/pyproject.toml#L9-L13) 
- `fastapi`: [upper-bounds starlette and pydantic](https://github.com/fastapi/fastapi/blob/febf6b6a52e36e0d64920fb54670ad7fc29b2eaa/pyproject.toml#L44-L46). At a glance, neither seem to me to be due to known incompatibility.
  - `starlette`: https://github.com/fastapi/fastapi/pull/12431
  - `pedantic`: https://github.com/fastapi/fastapi/pull/9816/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711

- `flask`: [no upper-bounds](https://github.com/pallets/flask/blob/2fec0b206c6e83ea813ab26597e15c96fab08be7/pyproject.toml#L23C1-L30C2) 
- `requests`: [upper-bounds 3 out of 4 of its dependencies](https://github.com/psf/requests/blob/7335bbf480adc8e6aa88feb2022797a549a00aa3/setup.cfg#L7-L10). At a glance, none seem to be due to known incompatibility. 
  - `charset_normalizer`: https://github.com/psf/requests/pull/6261
  - `idna`: https://github.com/psf/requests/pull/4889
  - `urllib3`: https://github.com/psf/requests/pull/6545
- `mitmproxy`: uses [very tight bounding for almost all its dependencies](https://github.com/mitmproxy/mitmproxy/blob/8ef97b1f6786cde5b79789d0ef903c47d4951eb3/pyproject.toml#L34-L62). These are not due to known incompatibilities but rather seem to be some `dependabot` policy. If you do a git blame on the lines that specify the dependencies you can see that the changes were authored by `dependabot`
- `scrapy`: [no upper-bounds](https://github.com/scrapy/scrapy/blob/6d65708cb7f7b49b72fc17486fecfc1caf62e0af/setup.py#L8-L25)



> Without a lockfile, you do not have reproducible environments to develop the package or run tests. You are fully at the mercy of external changes in the dependency resolution. It's the dreaded 'it works on my machine'.
> 
> To express the same thing in terms of your deployment/library distinction: yes, inspect is a library, but every engineer's laptop, and GitHub actions, are 'deployments' that will be subject to random breakage without a lockfile.

Have worked on many projects with and without lockfiles (the vast majority of our projects at AISI do in fact use Poetry lockfiles). In the case of Inspect, we are in some ways courting the variability so that we can discover issues with both newer and older versions as they come up. i.e. our users will not have lockfiles they'll just have a combination of older versions + very up to date versions. We want to expose ourselves to this same variability so we can proactively discover issues. 

That said, if we regularly experienced pain associated with not being able to get/keep things running we'd absolutely go for lockfiles. So far it has been more of a benefit than a hinderance but we will happily shift gears if it becomes obvious we should.

> I'm a bit confused about what your position is on this. Do you want to use semantic versioning or do you want to use no upper bounds?

When I say "semantic versioning" that's actually shorthand for just relying on the general practice of libraries doing their best not to cause breakage (I don't literally mean we will assume that semantic versioning works, which the referenced post does a good job pointing out doesn't really in practice). We allow a wide range of versions, then when we see issues attempt to resolve them by being maximally permissive (i.e. even dynamically supporting multiple versions if possible). Breakage happens, but so far we've only seen it from model API packages (which are often evolved alongside service changes) which is why we do more aggressively lower bounding there. 

Philosophically, we really try to make our software "fit" the user's environment even if it causes us some inconvenience. I've seen too many projects struggle with adoption due to overly specific requirements. Developers of packages sometimes think their package is the center of the universe when that couldn't be further from the truth from a user standpoint!

All of that said, I think we currently suffer from not quite enough lower bounding, as often picking reasonable (e.g. ~ 2 year old) lower bounds can help users avoid the bugs and compatibility issues that tend to be addressed over time. We just need to set aside the time to make some good judgements there.





I went through each dependency to check to see if a good case could be made for a lower-bound. In 2 or 3 cases we actually had a hard requirement from a version (typically though 2 or 3 yrs old though so it didn't seem to come up in practice). In several other cases packages have had releases that take care of Python or OS compatibility issues or made material improvements to stability, and I set lower bounds in those cases as well.

Updated requirements.txt in this commit: https://github.com/UKGovernmentBEIS/inspect_ai/pull/736