Repository: EleutherAI/lm-evaluation-harness
Issue: RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
Root cause: Environment Incompatibility

我在最新的框架下运行以下命令报错：
python -u main.py \
    --model hf-causal-experimental \
    --model_args pretrained=/root/nas/pretrained_model/llm_models/BELLE-7B-2M,use_accelerate=True \
    --tasks mrpc,mutual,wmt20-en-zh \
    --device cuda:0 \
    --no_cache
然后报错：
Traceback (most recent call last):
  File "/root/nas/LLM/lmeval2/lm-evaluation-harness/main.py", line 114, in <module>
    main()
  File "/root/nas/LLM/lmeval2/lm-evaluation-harness/main.py", line 85, in main
    results = evaluator.simple_evaluate(
  File "/root/nas/LLM/lmeval2/lm-evaluation-harness/lm_eval/utils.py", line 182, in _wrapper
    return fn(*args, **kwargs)
  File "/root/nas/LLM/lmeval2/lm-evaluation-harness/lm_eval/evaluator.py", line 86, in simple_evaluate
    results = evaluate(
  File "/root/nas/LLM/lmeval2/lm-evaluation-harness/lm_eval/utils.py", line 182, in _wrapper
    return fn(*args, **kwargs)
  File "/root/nas/LLM/lmeval2/lm-evaluation-harness/lm_eval/evaluator.py", line 247, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
  File "/root/nas/LLM/lmeval2/lm-evaluation-harness/lm_eval/base.py", line 186, in loglikelihood
    return self._loglikelihood_tokens(new_reqs)
  File "/root/nas/LLM/lmeval2/lm-evaluation-harness/lm_eval/base.py", line 344, in _loglikelihood_tokens
    self._model_call(batched_inps), dim=-1
  File "/root/nas/LLM/lmeval2/lm-evaluation-harness/lm_eval/models/huggingface.py", line 476, in _model_call
    return self.model(inputs)["logits"]
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/transformers/models/bloom/modeling_bloom.py", line 913, in forward
    transformer_outputs = self.transformer(
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/transformers/models/bloom/modeling_bloom.py", line 786, in forward
    outputs = block(
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/transformers/models/bloom/modeling_bloom.py", line 439, in forward
    attn_outputs = self.self_attention(
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/transformers/models/bloom/modeling_bloom.py", line 290, in forward
    fused_qkv = self.query_key_value(hidden_states)  # [batch_size, seq_length, 3 x hidden_size]
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/opt/conda/envs/lmeval/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
请问这是什么原因，我运行单gpu时没有报错

`--model hf-causal-experimental`可能有点问题

Hi! I am not sure what is causing the error. I see that you're running a local model?

Generally with this error code it means that either
1) There is an index error where the code is trying to access a tensor element that is out-of-bounds of the shape you have: for example if you're using the wrong tokenizer. Perhaps adding `tokenizer=<TOKENIZER_NAME>` to `--model_args` would help this if the tokenizer you want to use is available on the HF hub.
2) A really bad out-of-memory error or something similar, but your batch size is small so should not be happening. 
3) some weird CUDA driver issue

@Yun-Peng-Wang Hey, do you have any follow up questions? Was Hailey’s response helpful?