mosaicml

/

mpt-30b-chat

like 198

I'm trying to get MPT-30b-Chat running on a ml.p3.8xlarge instance and I'm gettin an error that says I'm out of disc space "[Errno 28] No space left on device". I've tweaked the code from this post (https://hackernoon.com/how-to-run-mpt-7b-on-aws-sagemaker-mosaicmls-chatgpt-competitor) which can be found here (https://colab.research.google.com/drive/1kJr2LHHLKYkbnNutVYEkt2vrYsbO38aw?ref=hackernoon.com). Here is my current code:

!pip install -qU transformers accelerate einops langchain xformers
from torch import cuda, bfloat16
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig
import transformers

device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu'

name = 'mosaicml/mpt-30b-chat'

tokenizer = AutoTokenizer.from_pretrained(name, trust_remote_code=True)

config = transformers.AutoConfig.from_pretrained(name, trust_remote_code=True)
config.init_device = 'cuda:0'

model = AutoModelForCausalLM.from_pretrained(name,
                                             trust_remote_code=True,
                                             config=config,
                                             torch_dtype=bfloat16)

Would really appreciate any help on this.