Not able to deploy llama-7b-hf over sagemaker #39
from sagemaker.huggingface import HuggingFaceModel
import sagemaker
role = sagemaker.get_execution_role()
hub = {
'HF_MODEL_ID':'decapoda-research/llama-7b-hf',
'HF_TASK':'question-answering'
}
create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
transformers_version='4.12',
pytorch_version='1.9',
py_version='py38',
env=hub,
role=role,
)
deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
initial_instance_count=1, # number of instances
instance_type='ml.m5.2xlarge' # ec2 instance type
)
predictor.predict(data = "what is the capital of Pakistan?")
same error here!
+1 on the error, its most likely a bug on HF end. @philschmid can you please take a look?
the LLama architecture got added in Transformers 4.28.0 which is not yet available on SageMaker.
Also to mention LLama is not allowed for commercial use.
I do not understand models trained on public data are not available for commercial use by public. So strange
btw I finetuned it and it is amazing. That's the poem i generated using finetuned llama-7b.
Birds of the sky, soaring high,
Filling the air with melodious cry.
With wings that flutter and sway,
They dance in the breeze of the day.
Birds of the earth, hopping and pecking,
In search of worms and seeds to becking.
With beaks that chisel and scrape,
They dig in the soil and scrape.
Birds of the sea, diving and swooping,
In search of fish and crustaceous hopping.
With webbed feet that paddle and glide,
They swim in the ocean's tide.
Birds of the air, soaring high,
Filling the air with melodious cry.
With wings that flutter and sway,
They dance in the breeze of the day.
Birds of the earth, hopping and pecking,
In search of worms and seeds to becking.
With beaks that chisel and scrape,
They dig in the soil and scrape.
Birds of the sea, diving and swooping,
In search of fish and crustaceous hopping.
With webbed feet that paddle and glide,
They swim in the ocean's tide.
Birds of the air, soaring high,
Filling the air with melodious cry.
With wings that flutter and sway,
They dance in the breeze of the day.
Birds of the earth, hopping and pecking,
In search of worms and seeds to becking.
With beaks that chisel and scrape,
They dig in the soil and scrape.
Birds of the sea, diving and swooping,
In search of fish and crustaceous hopping.
With webbed feet that paddle and glide,
They swim in the ocean's tide.
Birds of the air, soaring high,
Filling the air with melodious cry.
With wings that flutter and sway,
They dance in the breeze of the day.
Seems like Yann LeCun disagrees. LLaMa is available for commercial use. https://twitter.com/ylecun/status/1629214560077901825?lang=en
any timeline for support on this? i would like to deploy llama as a sagemaker endpoint but get the same error (also same for any llama derivative model)