tiiuae/falcon-7b-instruct · Deployment to Amazon SageMaker

May 30, 2023

I'm trying to deploy the model to Amazon SageMaker via the HuggingFaceModel class. When I try to use the code snippet provided with this model, I can deploy the endpoint, however the predict query fails with the following error:

"{
  "code": 400,
  "type": "InternalServerException",
  "message": "Loading /.sagemaker/mms/models/tiiuae__falcon-7b-instruct requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code\u003dTrue` to remove this error."
}"

I tried passing myself the trust_remote_code=True in the Pipeline code with a custom inference python script, but nothing changes - which makes me think that it just gets ignored by the Multi-Model Server in SageMaker.

Any suggestions on how to solve it?

rq-aszasz

Jun 8, 2023

I reproduced this error too.

I also tried setting the container_startup_health_check_timeout parameter to 1 hour, it didn't help.

rq-aszasz

Jun 8, 2023

Update:

Try with sagemaker>=2.163.0.

FalconLLM

Technology Innovation Institute org Jun 9, 2023

You can leverage the HuggingFace LLM Inference Container, as documented here, to deploy Falcon easily in SageMaker.

tiiuae
/

falcon-7b-instruct

Deployment to Amazon SageMaker - `trust_remote_code` issue