google/flan-ul2 · Is it possible to run this model on the CPU?

Jun 13, 2023

I have CPU only pytorch, I set the following in my code:

os.environ["CUDA_VISIBLE_DEVICES"] = "-1" 
tokenizer = AutoTokenizer.from_pretrained("google/flan-ul2") 
model = T5ForConditionalGeneration.from_pretrained("google/flan-ul2") 
model = model.to('cpu') 
torch.device("cpu")

and I just did this in bash:

export TORCH_CUDA_ARCH_LIST=""

and it still tells me this even though I am actively trying to to everything to stop any calls to CUDA:

anaconda3/envs/transformers/lib/python3.10/site-packages/torch/cuda/__init__.py", line 239, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

vmajor

Jun 14, 2023

I got the demo code to run on the CPU. I do not know yet why it does not want to do so when inside the actual code that I have for it, but it is clearly related to the code, not the model so it is up to me to fix.

vmajor changed discussion status to closed Jun 14, 2023