Repository: EleutherAI/lm-evaluation-harness
Issue: Write tests for calling CLI arguments downstream to ensure correctly-returned types
Root cause: Validation Gap

Issue:  Currently we don't assert that passed-in args are the correct type passed to downstream code,  causing runtime exceptions. See example of [issue here](https://github.com/EleutherAI/lm-evaluation-harness/issues/1515).

There's a couple approaches we can try here, from least effort to most as far as I see it, most manual to most automated: 

1. Adding `assert `statements to all current arg-level actions in `cli_evaluate()`
2. Writing several unit tests that take in mock parsed args and unit-testing only `cli_evaluate()`
3. Breaking up `cli_evaluate()` into likely into two methods, one that starts from the method signature, and one that starts at line 309, where the evaluation logger starts, so we can test the part of the method where we assign variables from the CLI directly, and unit test that, 

Happy to discuss any of these and open to other approaches if they've been previously discussed, and also happy to take this as an issue. 

In my opinion, raising proper errors in the main code (not the tests) is preferable to using asserts.
Pros:
* this repo can be used as a submodule or package in a larger pipeline that can try to catch specific bugs.
* described here https://stackoverflow.com/a/40183030

Proper exceptions can also be pytested:
https://stackoverflow.com/questions/23337471/how-do-i-properly-assert-that-an-exception-gets-raised-in-pytest


One downside of raising errors in the code ahead of merge to main is that you'd have to know ahead of runtime that the code you're adding to the command line args requires new error handling. But, if all of the arguments are checked in a pre-merge test, you wouldn't be able to run the package to begin with and merge the code. 

What we'd like to test is automatically adding new command line args that [get parsed correctly downstream:](https://github.com/EleutherAI/lm-evaluation-harness/pull/1487/files#diff-45b7cf15ed225746696faea6973a591c9526c8a78f486e28b11f63635e543666R293) 

As a specific case, in this case, this environment variable should be a `string` rather than a `boolean`:

```python
   if args.trust_remote_code:
        os.environ["HF_DATASETS_TRUST_REMOTE_CODE"] = (
            args.trust_remote_code if args.trust_remote_code else True
        )
        args.model_args = (
            args.model_args
            + f",trust_remote_code={os.environ['HF_DATASETS_TRUST_REMOTE_CODE']}"
        )
```

Ultimately this causes an error in `cli_evaluate()`, so we could perhaps start by adding try/catch exceptions to each of the input CLI args. 



Maybe pydantic can help somehow
https://github.com/pydantic/pydantic-settings/issues/209
https://www.youtube.com/watch?v=7aBRk_JP-qY
https://youtu.be/zN4VCb0LbQI?t=514

> this environment variable should be a `string `rather than a `boolean`

As defined in documentation
https://docs.python.org/3.8/library/os.html#os.environ
_A [mapping](https://docs.python.org/3.8/glossary.html#term-mapping) object representing the **string** environment._
So, this specific problem should probably be catched by typecheckers at pre-commit time.


Closed with https://github.com/EleutherAI/lm-evaluation-harness/pull/1566