mistralai

/

Mixtral-8x7B-Instruct-v0.1

like 3.92k

I am trying to deploy this model to Sagemaker using the provided deployment code and am getting this error:

File "text_generation_server/server.py", line 159, in serve_inner
    model = get_model(
  File "text_generation_server/models/__init__.py", line 291, in get_model
    raise ValueError("sharded is not supported for AutoModel")
> File "text_generation_server/server.py", line 159, in serve_inner model = get_model( File "text_generation_server/models/__init__.py", line 291, in get_model raise ValueError("sharded is not supported for AutoModel")

Deployment Code for Reference:

import json
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

try:
    role = sagemaker.get_execution_role()
except ValueError:
    iam = boto3.client('iam')
    role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']

# Hub Model configuration. https://huggingface.co/models
hub = {
    'HF_MODEL_ID':'mistralai/Mixtral-8x7B-Instruct-v0.1',
    'SM_NUM_GPUS': json.dumps(8)
}



# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    image_uri=get_huggingface_llm_image_uri("huggingface",version="1.1.0"),
    env=hub,
    role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.48xlarge",
    container_startup_health_check_timeout=300,
  )
  
# send request
predictor.predict({
    "inputs": "[INST] You are a pirate chatbot who always responds with Arr and pirate speak!\nThere's a llama on my lawn, how can I get rid of him? [/INST]",
})