Repository: EleutherAI/lm-evaluation-harness
Issue: [big-refactor] Question with parallelize=True
Root cause: Documentation Deficiency

With parallelize=True, I think it should be like fsdp and each gpu should use similar memory:
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100-SXM...  On   | 00000000:3F:00.0 Off |                    0 |
| N/A   39C    P0    95W / 400W |  57607MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-SXM...  On   | 00000000:44:00.0 Off |                    0 |
| N/A   46C    P0   360W / 400W |  29373MiB / 81920MiB |     92%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA A100-SXM...  On   | 00000000:62:00.0 Off |                    0 |
| N/A   36C    P0   204W / 400W |  29373MiB / 81920MiB |     18%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA A100-SXM...  On   | 00000000:68:00.0 Off |                    0 |
| N/A   39C    P0    78W / 400W |  29373MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA A100-SXM...  On   | 00000000:A9:00.0 Off |                    0 |
| N/A   39C    P0    83W / 400W |  29373MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA A100-SXM...  On   | 00000000:AD:00.0 Off |                    0 |
| N/A   32C    P0    73W / 400W |  29373MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA A100-SXM...  On   | 00000000:D2:00.0 Off |                    0 |
| N/A   33C    P0    93W / 400W |  36099MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA A100-SXM...  On   | 00000000:D5:00.0 Off |                    0 |
| N/A   35C    P0    77W / 400W |   1057MiB / 81920MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
```
As you can see, the first gpu is much more cosuming and at each time, only one gpu is running.

Thanks for noting this! I've added more description to the README about this option https://github.com/EleutherAI/lm-evaluation-harness/pull/638 , let me know if this clears things up. 

We do however support FSDP in addition to `parallelize=True`, which @fattorib will be documenting in this readme section soon!

Feel free to reopen if there is any confusion!