Repository: EleutherAI/lm-evaluation-harness
Issue: RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)` error in some models
Root cause: Resource Mishandling

Hi, when I try to test psmathur/orca_mini_v3_13b, I get the following error 

> 2024-04-08:23:27:17,531 INFO     [__main__.py:251] Verbosity set to INFO
2024-04-08:23:27:21,823 INFO     [__main__.py:335] Selected Tasks: ['hellaswag']
2024-04-08:23:27:21,825 INFO     [evaluator.py:131] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
2024-04-08:23:27:21,825 INFO     [evaluator.py:177] Initializing hf model, with arguments: {'pretrained': 'psmathur/orca_mini_v3_13b', 'tokenizer': 'psmathur/orca_mini_v3_13b', 'dtype': 'float16', 'parallelize': True}
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:05<00:00,  1.93s/it]
2024-04-08:23:27:32,157 WARNING  [big_modeling.py:433] Some parameters are on the meta device device because they were offloaded to the cpu.
/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/datasets/load.py:1461: FutureWarning: The repository for hellaswag contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/hellaswag
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
2024-04-08:23:27:39,213 INFO     [task.py:395] Building contexts for hellaswag on rank 0...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10042/10042 [00:03<00:00, 3062.03it/s]
2024-04-08:23:27:43,273 INFO     [evaluator.py:379] Running loglikelihood requests
Running loglikelihood requests:   0%|                                                                                                 | 0/40168 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/local/home/tcelik/miniconda3/envs/dslenv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/local/home/tcelik/lm-evaluation-harness/lm_eval/__main__.py", line 341, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/local/home/tcelik/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/local/home/tcelik/lm-evaluation-harness/lm_eval/evaluator.py", line 251, in simple_evaluate
    results = evaluate(
  File "/local/home/tcelik/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/local/home/tcelik/lm-evaluation-harness/lm_eval/evaluator.py", line 390, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/local/home/tcelik/lm-evaluation-harness/lm_eval/api/model.py", line 336, in loglikelihood
    return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
  File "/local/home/tcelik/lm-evaluation-harness/lm_eval/models/huggingface.py", line 1049, in _loglikelihood_tokens
    self._model_call(batched_inps, **call_kwargs), dim=-1
  File "/local/home/tcelik/lm-evaluation-harness/lm_eval/models/huggingface.py", line 764, in _model_call
    return self.model(inps).logits
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1176, in forward
    outputs = self.model(
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1019, in forward
    layer_outputs = decoder_layer(
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 740, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 639, in forward
    query_states = self.q_proj(hidden_states)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/local/home/tcelik/miniconda3/envs/dslenv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 116, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
Running loglikelihood requests:   0%|                                                                                                 | 0/40168 [00:01<?, ?it/s]

The code I am using is:

`CUDA_VISIBLE_DEVICES=0,2 taskset -c 0-15,32-47 lm_eval --model hf \
    --tasks gsm8k \
    --model_args pretrained=psmathur/orca_mini_v3_13b,tokenizer=psmathur/orca_mini_v3_13b,dtype=float16,parallelize=True \
    --batch_size 1`

The same problem happens for hellaswag dataset. To make sure problem was not about cuda visible devices or task set I ran:

`CUDA_VISIBLE_DEVICES=0,2 taskset -c 0-15,32-47 lm_eval --model hf \
    --tasks hellaswag \
    --model_args pretrained=EleutherAI/gpt-j-6B,dtype=float16,parallelize=True \
    --batch_size auto`

which works without problem. I have seen that #499 had a similar problem, but there was no followup from ops side. Funny thing is, checking [the model card](https://huggingface.co/pankajmathur/orca_mini_v3_13b), this model was tested using lm eval harness.

Hi! Did the potential bug causes I listed in that other issue help at all?

Typically this error means either 1) way out-of-memory or 2) more often, tokenizer size is misconfigured somehow so at some point the code is trying to read an out-of-bounds value in the input or output embedding layers.

I’d check what the vocab size supposedly is, and if it matches the tokenizer. maybe a few extra tokens were added to the tokenizer but that’s not reflected anywhere. Also worth trying to load and do a forward pass with the model outside lm_eval to see if it runs

Hi. I did check the vocab size beforehand but it turns out borrowing some more gpus (4 in total) makes it work. I didn't fully understand what you meant by "A really bad out-of-memory error " in previous question and didn't expect memory to not be enough since for some other cases I was correctly getting OOM. Sorry for repeating question and thank you for your help.

Glad it’s working!

See e.g. https://discuss.pytorch.org/t/cuda-error-cublas-status-not-initialized-when-calling-cublascreate-handle/125450 — if happening to run up close enough to the memory limit that cublas cannot initialize a handle then this error instead of the standard OOM will be raised