Repository: EleutherAI/lm-evaluation-harness
Issue: Potential Bug- Results change with batch size
Root cause: Algorithmic Error

I am running the same tests with different batch sizes and getting different results. 

TLDR; with batch 1,5 and 32 the accuracy numbers are 0.3215, 0.3183, 0.3023 respectively
```
(venv) root@b410b7a53b03:/home/lm-evaluation-harness# python main.py --model hf-causal-experimental --model_args pretrained=mosaicml/mpt-7b --tasks hendrycksTest-philosophy --no_cache  --batch_size 5
Selected Tasks: ['hendrycksTest-philosophy']
Loading checkpoint shards: 100%|█████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.33s/it]
Task: hendrycksTest-philosophy; number of docs: 311
Task: hendrycksTest-philosophy; document 0; context prompt (starting on next line):
The following are multiple choice questions (with answers) about philosophy.

Anscombe criticizes as absurd Kant’s idea of:
A. the thing in itself.
B. the categorical imperative.
C. the phenomenal self.
D. legislating for oneself.
Answer:
(end of prompt on previous line)
Requests: [Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' A')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' B')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' C')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' D')[0]
]
Running loglikelihood requests
100%|██████████████████████████████████████████████████████████████████████| 1244/1244 [00:20<00:00, 60.06it/s]
{
  "results": {
    "hendrycksTest-philosophy": {
      "acc": 0.3215434083601286,
      "acc_stderr": 0.026527724079528872,
      "acc_norm": 0.3215434083601286,
      "acc_norm_stderr": 0.026527724079528872
    }
  },
  "versions": {
    "hendrycksTest-philosophy": 1
  },
  "config": {
    "model": "hf-causal-experimental",
    "model_args": "pretrained=mosaicml/mpt-7b",
    "num_fewshot": 0,
    "batch_size": "5",
    "batch_sizes": [],
    "device": null,
    "no_cache": true,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal-experimental (pretrained=mosaicml/mpt-7b), limit: None, provide_description: False, num_fewshot: 0, batch_size: 5
|          Task          |Version| Metric |Value |   |Stderr|
|------------------------|------:|--------|-----:|---|-----:|
|hendrycksTest-philosophy|      1|acc     |0.3215|±  |0.0265|
|                        |       |acc_norm|0.3215|±  |0.0265|

(venv) root@b410b7a53b03:/home/lm-evaluation-harness# python main.py --model hf-causal-experimental --model_args pretrained=mosaicml/mpt-7b --tasks hendrycksTest-philosophy --no_cache  --batch_size 1
Selected Tasks: ['hendrycksTest-philosophy']
Loading checkpoint shards: 100%|█████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.31s/it]
Task: hendrycksTest-philosophy; number of docs: 311
Task: hendrycksTest-philosophy; document 0; context prompt (starting on next line):
The following are multiple choice questions (with answers) about philosophy.

Anscombe criticizes as absurd Kant’s idea of:
A. the thing in itself.
B. the categorical imperative.
C. the phenomenal self.
D. legislating for oneself.
Answer:
(end of prompt on previous line)
Requests: [Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' A')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' B')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' C')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' D')[0]
]
Running loglikelihood requests
100%|██████████████████████████████████████████████████████████████████████| 1244/1244 [00:39<00:00, 31.10it/s]
{
  "results": {
    "hendrycksTest-philosophy": {
      "acc": 0.3183279742765273,
      "acc_stderr": 0.026457225067811032,
      "acc_norm": 0.3183279742765273,
      "acc_norm_stderr": 0.026457225067811032
    }
  },
  "versions": {
    "hendrycksTest-philosophy": 1
  },
  "config": {
    "model": "hf-causal-experimental",
    "model_args": "pretrained=mosaicml/mpt-7b",
    "num_fewshot": 0,
    "batch_size": "1",
    "batch_sizes": [],
    "device": null,
    "no_cache": true,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal-experimental (pretrained=mosaicml/mpt-7b), limit: None, provide_description: False, num_fewshot: 0, batch_size: 1
|          Task          |Version| Metric |Value |   |Stderr|
|------------------------|------:|--------|-----:|---|-----:|
|hendrycksTest-philosophy|      1|acc     |0.3183|±  |0.0265|
|                        |       |acc_norm|0.3183|±  |0.0265|

(venv) root@b410b7a53b03:/home/lm-evaluation-harness# python main.py --model hf-causal-experimental --model_args pretrained=mosaicml/mpt-7b --tasks hendrycksTest-philosophy --no_cache  --batch_size 32
Selected Tasks: ['hendrycksTest-philosophy']
Loading checkpoint shards: 100%|█████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.31s/it]
Task: hendrycksTest-philosophy; number of docs: 311
Task: hendrycksTest-philosophy; document 0; context prompt (starting on next line):
The following are multiple choice questions (with answers) about philosophy.

Anscombe criticizes as absurd Kant’s idea of:
A. the thing in itself.
B. the categorical imperative.
C. the phenomenal self.
D. legislating for oneself.
Answer:
(end of prompt on previous line)
Requests: [Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' A')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' B')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' C')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' D')[0]
]
Running loglikelihood requests
100%|██████████████████████████████████████████████████████████████████████| 1244/1244 [00:19<00:00, 65.34it/s]
{
  "results": {
    "hendrycksTest-philosophy": {
      "acc": 0.3022508038585209,
      "acc_stderr": 0.02608270069539965,
      "acc_norm": 0.3022508038585209,
      "acc_norm_stderr": 0.02608270069539965
    }
  },
  "versions": {
    "hendrycksTest-philosophy": 1
  },
  "config": {
    "model": "hf-causal-experimental",
    "model_args": "pretrained=mosaicml/mpt-7b",
    "num_fewshot": 0,
    "batch_size": "32",
    "batch_sizes": [],
    "device": null,
    "no_cache": true,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal-experimental (pretrained=mosaicml/mpt-7b), limit: None, provide_description: False, num_fewshot: 0, batch_size: 32
|          Task          |Version| Metric |Value |   |Stderr|
|------------------------|------:|--------|-----:|---|-----:|
|hendrycksTest-philosophy|      1|acc     |0.3023|±  |0.0261|
|                        |       |acc_norm|0.3023|±  |0.0261|
```

There is something off with the harness. Also, I noticed that the accuracy DECREASES as the fewshot INCREASES !! 
Can the maintainers (@haileyschoelkopf ) please look into this ?

TLDR; accuracy foes from 0.35 to 0.31 (!!) as fewshot goes from 1 to 5
```
(venv) root@038193943f10:/lm-evaluation-harness# python main.py --model hf-causal --model_args pretrained=mosai
cml/mpt-7b-instruct --tasks hendrycksTest-philosophy --num_fewshot 1 --no_cache  --batch_size 1
Selected Tasks: ['hendrycksTest-philosophy']
Using device 'cuda'
Loading checkpoint shards: 100%|█████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.67s/it]
Task: hendrycksTest-philosophy; number of docs: 311
Task: hendrycksTest-philosophy; document 0; context prompt (starting on next line):
The following are multiple choice questions (with answers) about philosophy.

Psychological egoism is:
A. an ethical theory about how we ought to behave.
B. a generalization concerning the way people tend to behave.
C. a claim about human nature and the ways people are capable of behaving.
D. none of the above.
Answer: C

Anscombe criticizes as absurd Kant’s idea of:
A. the thing in itself.
B. the categorical imperative.
C. the phenomenal self.
D. legislating for oneself.
Answer:
(end of prompt on previous line)
Requests: [Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nPsychological egoism is:\nA. an ethical theory about how we ought to behave.\nB. a generalization concerning the way people tend to behave.\nC. a claim about human nature and the ways people are capable of behaving.\nD. none of the above.\nAnswer: C\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' A')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nPsychological egoism is:\nA. an ethical theory about how we ought to behave.\nB. a generalization concerning the way people tend to behave.\nC. a claim about human nature and the ways people are capable of behaving.\nD. none of the above.\nAnswer: C\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' B')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nPsychological egoism is:\nA. an ethical theory about how we ought to behave.\nB. a generalization concerning the way people tend to behave.\nC. a claim about human nature and the ways people are capable of behaving.\nD. none of the above.\nAnswer: C\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' C')[0]
, Req_loglikelihood('The following are multiple choice questions (with answers) about philosophy.\n\nPsychological egoism is:\nA. an ethical theory about how we ought to behave.\nB. a generalization concerning the way people tend to behave.\nC. a claim about human nature and the ways people are capable of behaving.\nD. none of the above.\nAnswer: C\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:', ' D')[0]
]
Running loglikelihood requests
100%|██████████████████████████████████████████████████████████████████████| 1244/1244 [00:26<00:00, 46.88it/s]
{
  "results": {
    "hendrycksTest-philosophy": {
      "acc": 0.35691318327974275,
      "acc_stderr": 0.02721042037593402,
      "acc_norm": 0.35691318327974275,
      "acc_norm_stderr": 0.02721042037593402
    }
  },
  "versions": {
    "hendrycksTest-philosophy": 1
  },
  "config": {
    "model": "hf-causal",
    "model_args": "pretrained=mosaicml/mpt-7b-instruct",
    "num_fewshot": 1,
    "batch_size": "1",
    "batch_sizes": [],
    "device": null,
    "no_cache": true,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal (pretrained=mosaicml/mpt-7b-instruct), limit: None, provide_description: False, num_fewshot: 1, batch_size: 1
|          Task          |Version| Metric |Value |   |Stderr|
|------------------------|------:|--------|-----:|---|-----:|
|hendrycksTest-philosophy|      1|acc     |0.3569|±  |0.0272|
|                        |       |acc_norm|0.3569|±  |0.0272|

(venv) root@038193943f10:/lm-evaluation-harness# python main.py --model hf-causal --model_args pretrained=mosaicml/mpt-7b-instruct --tasks hendrycksTest-philosophy --num_fewshot 5 --no_cache  --batch_size 1
Selected Tasks: ['hendrycksTest-philosophy']
Using device 'cuda'
Loading checkpoint shards: 100%|█████████████████████████████████████████████████| 2/2 [00:02<00:00,  1.37s/it]
Task: hendrycksTest-philosophy; number of docs: 311
Task: hendrycksTest-philosophy; document 0; context prompt (starting on next line):
The following are multiple choice questions (with answers) about philosophy.

Psychological egoism is:
A. an ethical theory about how we ought to behave.
B. a generalization concerning the way people tend to behave.
C. a claim about human nature and the ways people are capable of behaving.
D. none of the above.
Answer: C

According to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:
A. pleasure.
B. happiness.
C. good.
D. virtue.
Answer: C

According to d'Holbach, people always act according to _____.
A. free choices
B. dictates of the soul
C. necessary natural laws
D. undetermined will
Answer: C

Before Tolstoy's Christian conversion, what was his perspective on the meaning of life?
A. optimist
B. satisfied
C. nominally religious
D. pessimist
Answer: D

The study of reality in the broadest sense, an inquiry into the elemental nature of the universe and the things in it, is known as _____.
A. metaphysics
B. epistemology
C. quantum physics
D. axiology
Answer: A

Anscombe criticizes as absurd Kant’s idea of:
A. the thing in itself.
B. the categorical imperative.
C. the phenomenal self.
D. legislating for oneself.
Answer:
(end of prompt on previous line)
Requests: [Req_loglikelihood("The following are multiple choice questions (with answers) about philosophy.\n\nPsychological egoism is:\nA. an ethical theory about how we ought to behave.\nB. a generalization concerning the way people tend to behave.\nC. a claim about human nature and the ways people are capable of behaving.\nD. none of the above.\nAnswer: C\n\nAccording to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:\nA. pleasure.\nB. happiness.\nC. good.\nD. virtue.\nAnswer: C\n\nAccording to d'Holbach, people always act according to _____.\nA. free choices\nB. dictates of the soul\nC. necessary natural laws\nD. undetermined will\nAnswer: C\n\nBefore Tolstoy's Christian conversion, what was his perspective on the meaning of life?\nA. optimist\nB. satisfied\nC. nominally religious\nD. pessimist\nAnswer: D\n\nThe study of reality in the broadest sense, an inquiry into the elemental nature of the universe and the things in it, is known as _____.\nA. metaphysics\nB. epistemology\nC. quantum physics\nD. axiology\nAnswer: A\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:", ' A')[0]
, Req_loglikelihood("The following are multiple choice questions (with answers) about philosophy.\n\nPsychological egoism is:\nA. an ethical theory about how we ought to behave.\nB. a generalization concerning the way people tend to behave.\nC. a claim about human nature and the ways people are capable of behaving.\nD. none of the above.\nAnswer: C\n\nAccording to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:\nA. pleasure.\nB. happiness.\nC. good.\nD. virtue.\nAnswer: C\n\nAccording to d'Holbach, people always act according to _____.\nA. free choices\nB. dictates of the soul\nC. necessary natural laws\nD. undetermined will\nAnswer: C\n\nBefore Tolstoy's Christian conversion, what was his perspective on the meaning of life?\nA. optimist\nB. satisfied\nC. nominally religious\nD. pessimist\nAnswer: D\n\nThe study of reality in the broadest sense, an inquiry into the elemental nature of the universe and the things in it, is known as _____.\nA. metaphysics\nB. epistemology\nC. quantum physics\nD. axiology\nAnswer: A\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:", ' B')[0]
, Req_loglikelihood("The following are multiple choice questions (with answers) about philosophy.\n\nPsychological egoism is:\nA. an ethical theory about how we ought to behave.\nB. a generalization concerning the way people tend to behave.\nC. a claim about human nature and the ways people are capable of behaving.\nD. none of the above.\nAnswer: C\n\nAccording to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:\nA. pleasure.\nB. happiness.\nC. good.\nD. virtue.\nAnswer: C\n\nAccording to d'Holbach, people always act according to _____.\nA. free choices\nB. dictates of the soul\nC. necessary natural laws\nD. undetermined will\nAnswer: C\n\nBefore Tolstoy's Christian conversion, what was his perspective on the meaning of life?\nA. optimist\nB. satisfied\nC. nominally religious\nD. pessimist\nAnswer: D\n\nThe study of reality in the broadest sense, an inquiry into the elemental nature of the universe and the things in it, is known as _____.\nA. metaphysics\nB. epistemology\nC. quantum physics\nD. axiology\nAnswer: A\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:", ' C')[0]
, Req_loglikelihood("The following are multiple choice questions (with answers) about philosophy.\n\nPsychological egoism is:\nA. an ethical theory about how we ought to behave.\nB. a generalization concerning the way people tend to behave.\nC. a claim about human nature and the ways people are capable of behaving.\nD. none of the above.\nAnswer: C\n\nAccording to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:\nA. pleasure.\nB. happiness.\nC. good.\nD. virtue.\nAnswer: C\n\nAccording to d'Holbach, people always act according to _____.\nA. free choices\nB. dictates of the soul\nC. necessary natural laws\nD. undetermined will\nAnswer: C\n\nBefore Tolstoy's Christian conversion, what was his perspective on the meaning of life?\nA. optimist\nB. satisfied\nC. nominally religious\nD. pessimist\nAnswer: D\n\nThe study of reality in the broadest sense, an inquiry into the elemental nature of the universe and the things in it, is known as _____.\nA. metaphysics\nB. epistemology\nC. quantum physics\nD. axiology\nAnswer: A\n\nAnscombe criticizes as absurd Kant’s idea of:\nA. the thing in itself.\nB. the categorical imperative.\nC. the phenomenal self.\nD. legislating for oneself.\nAnswer:", ' D')[0]
]
Running loglikelihood requests
100%|██████████████████████████████████████████████████████████████████████| 1244/1244 [00:45<00:00, 27.42it/s]
{
  "results": {
    "hendrycksTest-philosophy": {
      "acc": 0.31511254019292606,
      "acc_stderr": 0.0263852737034645,
      "acc_norm": 0.31511254019292606,
      "acc_norm_stderr": 0.0263852737034645
    }
  },
  "versions": {
    "hendrycksTest-philosophy": 1
  },
  "config": {
    "model": "hf-causal",
    "model_args": "pretrained=mosaicml/mpt-7b-instruct",
    "num_fewshot": 5,
    "batch_size": "1",
    "batch_sizes": [],
    "device": null,
    "no_cache": true,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal (pretrained=mosaicml/mpt-7b-instruct), limit: None, provide_description: False, num_fewshot: 5, batch_size: 1
|          Task          |Version| Metric |Value |   |Stderr|
|------------------------|------:|--------|-----:|---|-----:|
|hendrycksTest-philosophy|      1|acc     |0.3151|±  |0.0264|
|                        |       |acc_norm|0.3151|±  |0.0264|
```

It's a known issue that changing batch size induces non-determinism on GPUs. Have you found that performance decreases monotonously with batch size? If so, that could be a sign of a problem. However with only two data points it's hard to draw any conclusions.

Seems duplicative of https://github.com/EleutherAI/lm-evaluation-harness/issues/873