Repository: modelscope/evalscope
Issue:  CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
Root cause: Environment Incompatibility

## 问题描述 / Issue Description

使用quick-start样例时报错


## 使用的工具 / Tools Used
- Native / 原生框架



## 执行的代码或指令 / Code or Commands Executed


```bash
python evalscope/run.py \
 --model /fs/archive/share/Meta-Llama-3-8B-Instruct \
 --template-type llama3 \
 --datasets hellaswag
```

## 错误日志 / Error Log


```
Traceback (most recent call last):
  File "/home/u2022000909/data_selection/evalscope/evalscope/run.py", line 408, in <module>
    main()
  File "/home/u2022000909/data_selection/evalscope/evalscope/run.py", line 402, in main
    run_task(task_cfg)
  File "/home/u2022000909/data_selection/evalscope/evalscope/run.py", line 367, in run_task
    res_dict: dict = evaluator.eval(infer_cfg=infer_cfg, debug=debug)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 476, in eval
    answers_list: list = self.get_answers(subset_name=subset_name,
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 238, in get_answers
    answer_d: dict = self._pred_answer(input_d=input_prompt,
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 138, in _pred_answer
    ans: dict = self.model_adapter.predict(inputs=input_d, infer_cfg=infer_cfg)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/models/model_adapter.py", line 282, in predict
    pred_list: list = self.loglikelihood(inputs=inputs['data'], infer_cfg=infer_cfg)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/models/model_adapter.py", line 315, in loglikelihood
    logits = self.model(inputs_tokens)[0]
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1190, in forward
    outputs = self.model(
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 921, in forward
    position_embeddings = self.rotary_emb(hidden_states, position_ids)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/u2022000909/anaconda3/envs/evalscope/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 158, in forward
    freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - Ubuntu

- Python版本 / Python Version:
  - 3.10



请确认cuda环境是否正常，运行下面的代码：
```python
import torch
print(torch.__version__)
print(torch.cuda.is_available())
print(torch.cuda.current_device())
```

可能是之前的gpu内存不够了，更换了一个gpu解决了，谢谢

如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！