Repository: EleutherAI/lm-evaluation-harness
Issue: Hashing error when setting random seed for vllm model
Root cause: Validation Gap

It seems vllm engine uses an [initialization seed](https://github.com/allenai/oe-eval-internal/blob/f7319fddb3bb225c269a01d657affdaf139aa95b/oe_eval/models/eleuther_vllm_causallms.py#L38) to determine its randomness in sampling. I am using this feature to generate multiple different completions with scores by running the same evaluation code multiple times. However, it errored when I trying to set the seed in the command.

E.g., command:
`python -m oe_eval.run_eval --task '{"task_name": "gsm8k", "generation_kwargs": {"repeats": 1, "temperature": 0.7, "top_p": 1.0, "max_gen_toks": 4096, "do_sample": true, "truncate_context": true}, "limit": 1000, "random_subsample_seed": 42, "use_chat_format": true}' --batch-size 1000 --output-dir output/oe-eval-bon-candidates-new/tulu-2-13b-n-16/gsm8k/run-1 --save-raw-requests true --model allenai/tulu-2-13b --model-args '{"model_path": "allenai/tulu-2-13b", "seed": 14609, "max_length": 8192, "model_type": "vllm"}'`

Error message:
```
Traceback (most recent call last):
  File "/net/nfs.cirrascale/allennlp/yizhongw/miniconda3/envs/oe-eval/lib/python3.10/runpy.py", line 196, in _run_module_as_main                                                                           return _run_code(code, main_globals, None,
  File "/net/nfs.cirrascale/allennlp/yizhongw/miniconda3/envs/oe-eval/lib/python3.10/runpy.py", line 86, in _run_code                                                                                      exec(code, run_globals)                                                                                                                                                                              File "/net/nfs.cirrascale/allennlp/yizhongw/open-instruct-public/oe-eval-internal/oe_eval/run_eval.py", line 758, in <module>
    run_eval(args_dict)                                                                                                                                                                                  File "/net/nfs.cirrascale/allennlp/yizhongw/open-instruct-public/oe-eval-internal/oe_eval/run_eval.py", line 407, in run_eval                                                                            model_hash = hash_dict(model_config, MODEL_DEFAULTS)
  File "/net/nfs.cirrascale/allennlp/yizhongw/open-instruct-public/oe-eval-internal/oe_eval/utils.py", line 136, in hash_dict                                                                              raise ValueError(f"Hashing error! Key {k} not in defaults!")
ValueError: Hashing error! Key seed not in defaults!
```

This is due to `seed` not set in the [default configs](https://github.com/allenai/oe-eval-internal/blob/f7319fddb3bb225c269a01d657affdaf139aa95b/oe_eval/default_configs.py#L3-L25). I am not sure if it should be set there as [huggingface model class](https://github.com/allenai/oe-eval-internal/blob/f7319fddb3bb225c269a01d657affdaf139aa95b/oe_eval/models/eleuther_huggingface.py#L22) doesn't seem to use this argument. Another workaround is to ignore seed in the hashing [here](https://github.com/allenai/oe-eval-internal/blob/f7319fddb3bb225c269a01d657affdaf139aa95b/oe_eval/utils.py#L14), but I am not sure if it should be ignored as well.

Hi! Looks like you're using a (private) fork. `seed` should work on `main` :)