Spaces:
Running
on
CPU Upgrade
Running
on
CPU Upgrade
Problem submitting gpt-sw3 models (gpt-2 architecture)
#13
by
Ekgren
- opened
All gpt-sw3 models can be found here https://huggingface.co/AI-Sweden-Models
For example this one https://huggingface.co/AI-Sweden-Models/gpt-sw3-20b-instruct fails to be submitted to the benchmark, see attached screenshot.
The models are of the gpt-2 architecture with a sentencepiece tokenizer.
Thanks! Looking into this.
Update: the harness is able to handle it without issues, so I think I can upload the leaderboard code to get rid of the issue!
%cd /content/lm-evaluation-harness
!lm_eval --model hf --model_args pretrained=AI-Sweden-Models/gpt-sw3-20b-instruct,revision=main --tasks nq_open --device cuda --batch_size 1 --num_fewshot 8
2024-02-03:07:55:56,177 INFO [utils.py:148] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-02-03:07:55:56,177 INFO [utils.py:160] NumExpr defaulting to 8 threads.
2024-02-03:07:55:56,352 INFO [config.py:58] PyTorch version 2.1.0+cu121 available.
2024-02-03:07:55:56,353 INFO [config.py:95] TensorFlow version 2.15.0 available.
2024-02-03:07:55:56,354 INFO [config.py:108] JAX version 0.4.23 available.
2024-02-03 07:55:56.917234: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-03 07:55:56.917328: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-03 07:55:56.918882: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-03 07:55:58.092248: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-02-03:07:56:00,083 INFO [__main__.py:162] Verbosity set to INFO
2024-02-03:07:56:00,083 INFO [__init__.py:358] lm_eval.tasks.initialize_tasks() is deprecated and no longer necessary. It will be removed in v0.4.2 release. TaskManager will instead be used.
2024-02-03:07:56:04,311 INFO [__main__.py:238] Selected Tasks: ['nq_open']
2024-02-03:07:56:04,311 INFO [__main__.py:239] Loading selected tasks...
2024-02-03:07:56:04,366 INFO [huggingface.py:148] Using device 'cuda'
config.json: 100% 995/995 [00:00<00:00, 5.92MB/s]
model.safetensors.index.json: 100% 48.8k/48.8k [00:00<00:00, 4.96MB/s]
Downloading shards: 0% 0/9 [00:00<?, ?it/s]
[..]
Now it works! Thank you
Ekgren
changed discussion status to
closed