openlm-research

/

open_llama_7b_v2

like 112

This is likely a result of running it on CPU, where the half-precision ops are not supported. To use it on CPU, you need to convert the data type to float32 before you run any inference.

it would be helpful if you paste the sample code you were testing on.

If you see this line comment it out:
torch.set_default_tensor_type(torch.cuda.HalfTensor)

Where do we find that line? which file?

If you downloaded the model directly from Meta, there should be this python script: /llama/llama/generation.py
You can either comment out line 100, or update it to be:

    if torch.cuda.is_available():
        torch.set_default_tensor_type(torch.cuda.HalfTensor)

so that half-tensors are not used if you don't have CUDA.

It would be helpful if you can paste the code that gave you this error. I was getting this same error when trying out the code present in this tutorial: https://huggingface.co/blog/llama2

For me, changing torch_dtype from torch.float16 to torch.bfloat16 fixed the issue.

For me this replicated the issue in colab:
https://colab.research.google.com/drive/1SDN3rJhyL9EpDWuDVjyE3lJ6hV0Cfdd-?usp=sharing

The error was not resolved by changing torch_dtype from torch.float16 to torch.bfloat16

This is the code that generated the error. If you have a solution, may you also describe the proposed solution's rationale?

from transformers import AutoTokenizer
import transformers
import torch

model = "meta-llama/Llama-2-70b-chat-hf"

tokenizer = AutoTokenizer.from_pretrained(model)

pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

sequences = pipeline(
    'I liked "Breaking Bad" and "Band of Brothers". Do you have any recommendations of other shows I might like?\n',
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=200,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

This is the code that generated the error. If you have a solution, may you also describe the proposed solution's rationale?

from transformers import AutoTokenizer
import transformers
import torch

model = "meta-llama/Llama-2-70b-chat-hf"

tokenizer = AutoTokenizer.from_pretrained(model)

pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

sequences = pipeline(
    'I liked "Breaking Bad" and "Band of Brothers". Do you have any recommendations of other shows I might like?\n',
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=200,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

same issue, and by changing torch16 to torch32, its taking forever to load, and consumes 99% of the RAM space and the notebook stops then. If anyone knows why is it happening and any solution to it. Please let me know. Thankyou!