Repository: EleutherAI/lm-evaluation-harness
Issue: Passing wandb_args to `simple_evaluate`
Root cause: Documentation Deficiency

The way `wandb_args` is [currently implemented](https://github.com/EleutherAI/lm-evaluation-harness/pull/1339), they're [initialized directly](https://github.com/EleutherAI/lm-evaluation-harness/blob/96d185fa6232a5ab685ba7c43e45d1dbb3bb906d/lm_eval/__main__.py#L335) as parsed from the command line argument and never passed to the `simple_evaluate` [call](https://github.com/EleutherAI/lm-evaluation-harness/blob/96d185fa6232a5ab685ba7c43e45d1dbb3bb906d/lm_eval/evaluator.py#L32):

```
lm_eval \
    --model hf \
    --model_args pretrained=microsoft/phi-2,trust_remote_code=True \
    --tasks hellaswag,mmlu_abstract_algebra \
    --device cuda:0 \
    --batch_size 8 \
    --output_path output/phi-2 \
    --limit 10 \
    --wandb_args project=lm-eval-harness-integration \
    --log_samples
```

Is there a way we could initialize these outside of the CLI? 

Looks like wandb simply logs the results dict returned from `simple_evaluate`. Could just add [this](https://github.com/EleutherAI/lm-evaluation-harness/blob/96d185fa6232a5ab685ba7c43e45d1dbb3bb906d/lm_eval/__main__.py#L335-L342) post-hoc?

We could but we also need to pass the [wandb parameters like project name, entity, and group](https://docs.wandb.ai/ref/python/init) to start with to make sure the logs are going to the right place when the job kicks off - where might be a good place for that? 

Are there patterns in the library for already doing something similar, for example kicking off a vLLM server or maybe connecting to an HF organization repo to upload an artifact, etc?

Not sure I understand. Seems to be just this:
https://github.com/EleutherAI/lm-evaluation-harness/blob/cc771ecac8964f0afab76e95e264e69528430ff2/lm_eval/__main__.py#L213
where are only `arg` it's using is this comma delimited string:
https://github.com/EleutherAI/lm-evaluation-harness/blob/cc771ecac8964f0afab76e95e264e69528430ff2/lm_eval/__main__.py#L182
https://github.com/EleutherAI/lm-evaluation-harness/blob/cc771ecac8964f0afab76e95e264e69528430ff2/lm_eval/logging_utils.py#L93-L99

We could modify `WandbLogger` to accept the actual string (or the wandb `init` kwargs) rather than the `argparser` object as it's not using any of the other args. Everything else, it gets from the results dict.

Yup, that makes sense. If you're kicking off a [job via](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage) `simple_evaluate` without passing a string from CLI input though, do these wandb arguments also make it into the [method parameters?](https://github.com/EleutherAI/lm-evaluation-harness/blob/cc771ecac8964f0afab76e95e264e69528430ff2/lm_eval/evaluator.py#L38)  

Seems so. Most of the methods are pretty static and the only stateful object seems to be `wandb.run()` which it initalizes/attaches to during `init`. I think it should also be safe to initialise it after `simple_evaluate` returns if you want to incorporate any of the [config](https://github.com/EleutherAI/lm-evaluation-harness/blob/cc771ecac8964f0afab76e95e264e69528430ff2/lm_eval/evaluator.py#L255) values in the job init (they are already [logged](https://github.com/EleutherAI/lm-evaluation-harness/blob/cc771ecac8964f0afab76e95e264e69528430ff2/lm_eval/logging_utils.py#L111) along with the results).

I thought this might be easier to show with a reprex, so I created a notebook to instantiate an evaluation based on the `simple_evaluate` api: https://gist.github.com/veekaybee/0cc69166f38c81ad6362a263d2fcf934, foregoing CLI and running an eval directly. 

Could you take a look and let me know if there's a place directly to pass in the wandb parameters? It doesn't seem I can add them to [simple_evaluate](https://github.com/EleutherAI/lm-evaluation-harness/blob/cc771ecac8964f0afab76e95e264e69528430ff2/lm_eval/evaluator.py#L38) because wandb is not explicitly available outside of `__main__`. If it's something we end up needing to add to simple_evaluate, I'm happy to contribute a PR. 

> I thought this might be easier to show with a reprex, so I created a notebook to instantiate an evaluation based on the `simple_evaluate` api: https://gist.github.com/veekaybee/0cc69166f38c81ad6362a263d2fcf934, foregoing CLI and running an eval directly.
> 
> Could you take a look and let me know if there's a place directly to pass in the wandb parameters? It doesn't seem I can add them to [simple_evaluate](https://github.com/EleutherAI/lm-evaluation-harness/blob/cc771ecac8964f0afab76e95e264e69528430ff2/lm_eval/evaluator.py#L38) because wandb is not explicitly available outside of `__main__`. If it's something we end up needing to add to simple_evaluate, I'm happy to contribute a PR.

I was thinking something like:
```python
import lm_eval
import wandb
from lm_eval.logging_utils import WandbLogger

results = lm_eval.simple_evaluate(...)

wandb_logger = WandbLogger(project="eval",...) # or empty if wandb.init() called before
wandb_logger.post_init(results)
wandb_logger.log_eval_result()
wandb_logger.log_eval_samples(results["samples"]) # if log_samples
```

or am I missing something?

Ah, got it! So in this case you'd instantiate the logger directly instead of having it be called as part of simple_evaluate. I was thinking based on previous use that everything is passed to simple_evaluate directly. That makes sense. Would it make sense to add some docs around this second approach here? https://github.com/EleutherAI/lm-evaluation-harness/pull/1339/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R311. Happy to do so. 

yeah, I just copied this from `__main__`. `simple_evaluate` is completely separate. I'll add this example to my PR but feel free to add/amend if I missed something!

cc @ayulockin