Amazon Sagemaker deployment issue for FLAN-T5 model family
Using the deployment script for Amazon Sagemaker as described on the FLAN-T5 model cards:
from sagemaker.huggingface import HuggingFaceModel
import sagemaker
role = sagemaker.get_execution_role()
hub = {
'HF_MODEL_ID':'google/flan-t5-small',
'HF_TASK':'text2text-generation'
}
huggingface_model = HuggingFaceModel(
transformers_version='4.17.0',
pytorch_version='1.10.2',
py_version='py38',
env=hub,
role=role,
)
predictor = huggingface_model.deploy(
initial_instance_count=1, # number of instances
instance_type='ml.m5.xlarge' # ec2 instance type
)
predictor.predict({
'inputs': "The answer to the universe is"
})
I receive the following error:
"AttributeError: 'T5LayerFF' object has no attribute 'config'"
Could someone please help with how to resolve this?
I had the same issue, I've updated transformers to the latest 4.24 version and it worked
I am using exact python example code and using transformers v. 4.25.1 and seeing
{
"code": 400,
"type": "InternalServerException",
"message": "Could not load model /.sagemaker/mms/models/google__flan-t5-xl with any of the following classes: (\u003cclass \u0027transformers.models.auto.modeling_auto.AutoModelForSeq2SeqLM\u0027\u003e, \u003cclass \u0027transformers.models.t5.modeling_t5.T5ForConditionalGeneration\u0027\u003e)."
}
And digging deeper in Cloudwatch logs I am seeing:
Could not load model {model} with any of the following classes: {class_tuple}
&
Could not load model /.sagemaker/mms/models/google__flan-t5-xl with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForSeq2SeqLM'>, <class 'transformers.models.t5.modeling_t5.T5ForConditionalGeneration'>
Might just be doing something incorrect, but wanted to post here in case something else. Thanks!
UPDATE:
Looks like maybe the issue is with me trying to load the xl version (flan-t5-xl) not seeing the same issue with (flan-t5-small)
Facing the same problem. Raised an issue here:
https://github.com/huggingface/transformers/issues/21402