Repository: EleutherAI/lm-evaluation-harness
Issue: How to specify evaluation times using different seeds?
Root cause: Unimplemented Feature Gap

I want to use different seed to sample different few-shot examples for evaluation, but I don't find an argument to specify that.

The restuls include a stderr, so I aslo wonder if there is a default setting about evalution serveral times?

Still haven't figured out that.

I think harness doesn't support evaluating in different seeds

No explicit argument to specify seed settings but they are set [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/7afae7b5e8dbd9da6f41cdf43bdd309840673911/lm_eval/evaluator.py#L81-L85) for the `big-refactor` branch.

The stderr is calculated through bootstrapping.

> No explicit argument to specify seed settings but they are set [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/7afae7b5e8dbd9da6f41cdf43bdd309840673911/lm_eval/evaluator.py#L81-L85) for the `big-refactor` branch.
> 
> The stderr is calculated through bootstrapping.

Thank you! It works.

> No explicit argument to specify seed settings but they are set [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/7afae7b5e8dbd9da6f41cdf43bdd309840673911/lm_eval/evaluator.py#L81-L85) for the `big-refactor` branch.
> 
> The stderr is calculated through bootstrapping.

Sorry, I find I make a mistake after changing the seeds you mentioned -- I used two models. I have checked it does not work after modifying seeds here. The correct position to specify a seed for sampling fewshot examples is [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/7afae7b5e8dbd9da6f41cdf43bdd309840673911/lm_eval/api/task.py#L667).

it is wired that there is no argument to specify a seed. basically, langauge models are sensitive about the demonstrations, so it is difficult to get a stable result or persuasive conclusion by evaluating on only one group demonstrations.


> it is wired that there is no argument to specify a seed. basically, langauge models are sensitive about the demonstrations, so it is difficult to get a stable result or persuasive conclusion by evaluating on only one group demonstrations.

I think it was a deliberate design choice as the harness was designed to provide a consistent evaluation framework to compare across different models/runs for a given benchmark task. 

Might be a good idea to consolidate the seed settings for debugging purposes if nothing else. @haileyschoelkopf  

Yes, I think we definitely can and should add manual setting of the seed!

> basically, langauge models are sensitive about the demonstrations, so it is difficult to get a stable result or persuasive conclusion by evaluating on only one group demonstrations.

We'd like to expose more configurability and control over input demonstrations going forward, to enable easier control + study of this. See `lm_eval/api/samplers.py` in the new `main` branch for where we intend to add more advanced configurations (such as drawing examples from classes proportionally/in a balanced way) or to allow others to add them. 

>We'd like to expose more configurability and control over input demonstrations going forward, to enable easier control + study of this. 

I replace the hard-coded seed with an environment varaible, it is like:
```
# rnd=random.Random(1234) # https://github.com/EleutherAI/lm-evaluation-harness/blob/7afae7b5e8dbd9da6f41cdf43bdd309840673911/lm_eval/api/task.py#L667
rnd=random.Random(int(os.environ.get("HARNESS_FEWSHOT_SEED", 1234)))
```
it works well :)


>  in the new main branch for where we intend to add more advanced configurations (such as drawing examples from classes proportionally/in a balanced way) or to allow others to add them.

Agree that "balanced sampling" is an important option! 



closed by #1412 !