Required Sagemaker instance size?

#5
by lcrane - opened

If I wanted to run opt-66b in an AWS Sagemaker instance, what are the memory/GPU/CPU requirements? I have tried a couple experiments (example code snippet below) but it seems to run out of memory loading the model and fail silently.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-66b", use_fast=False)
model = AutoModelForCausalLM.from_pretrained("facebook/opt-66b", torch_dtype=torch.float16).cuda()

Sign up or log in to comment