Repository: EleutherAI/lm-evaluation-harness
Issue: RuntimeError: Invalid device string: '0'
Root cause: Validation Gap

I tried demo in README.md and experience this error. I do have a GPU on the machine. 

```
 python main.py     --model hf-causal     --model_args pretrained=EleutherAI/gpt-j-6B     --tasks lambada_openai,hellaswag     --device 0

/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/computation/expressions.py:20: UserWarning: Pandas requires version '2.7.3' or newer of 'numexpr' (version '2.7.1' currently installed).
  from pandas.core.computation.check import NUMEXPR_INSTALLED
2023-03-22 17:33:22.377990: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
--------------------------------------------------------------------------
WARNING: No preset parameters were found for the device that Open MPI
detected:

  Local host:            152-70-119-217
  Device name:           mlx5_0
  Device vendor ID:      0x02c9
  Device vendor part ID: 4126

Default device parameters will be used, which may result in lower
performance.  You can edit any of the files specified by the
btl_openib_device_param_files MCA parameter to set values for your
device.

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_no_device_params_found to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           152-70-119-217
  Local device:         mlx5_0
  Local port:           1
  CPCs attempted:       udcm
--------------------------------------------------------------------------
Selected Tasks: ['lambada_openai', 'hellaswag']
Traceback (most recent call last):
  File "main.py", line 108, in <module>
    main()
  File "main.py", line 79, in main
    results = evaluator.simple_evaluate(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 64, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/base.py", line 114, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/models/huggingface.py", line 188, in __init__
    self.model.to(self._device)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1811, in to
    return super().to(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/torch/nn/modules/module.py", line 908, in to
    device, dtype, non_blocking, convert_to_format = torch._C._nn._parse_to(*args, **kwargs)
RuntimeError: Invalid device string: '0'
```


```
Wed Mar 22 17:34:15 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A10          On   | 00000000:06:00.0 Off |                    0 |
|  0%   32C    P8    21W / 150W |      2MiB / 23028MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
```

don't specify device or use --device cuda:0

@rgwt123 Thanks and it works! The doc should be updated.

Thanks for the PR @Jeffwan !