intfloat

/

e5-mistral-7b-instruct

like 413

I'm trying to deploy on an AWS Nvidia Tesla T4 · 4x GPU · 64 GB instance through Inference Endpoints, but I'm getting an error. The logs are as follows:

2024/01/29 14:17:51 ~ 2024-01-29 19:17:51,979 | INFO | Initializing model from directory:/repository
2024/01/29 14:17:51 ~ 2024-01-29 19:17:51,979 | INFO | No custom pipeline found at /repository/handler.py
2024/01/29 14:17:51 ~ 2024-01-29 19:17:51,979 | INFO | Using device GPU
2024/01/29 14:17:51 ~ Traceback (most recent call last):
2024/01/29 14:17:51 ~ File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 705, in lifespan
2024/01/29 14:17:51 ~ async with self.lifespan_context(app) as maybe_state:
2024/01/29 14:17:51 ~ File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 584, in __aenter__
2024/01/29 14:17:51 ~ await self._router.startup()
2024/01/29 14:17:51 ~ File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 682, in startup
2024/01/29 14:17:51 ~ await handler()
2024/01/29 14:17:51 ~ File "/app/webservice_starlette.py", line 57, in some_startup_task
2024/01/29 14:17:51 ~ inference_handler = get_inference_handler_either_custom_or_default_handler(HF_MODEL_DIR, task=HF_TASK)
2024/01/29 14:17:51 ~ File "/app/huggingface_inference_toolkit/handler.py", line 45, in get_inference_handler_either_custom_or_default_handler
2024/01/29 14:17:51 ~ return HuggingFaceHandler(model_dir=model_dir, task=task)
2024/01/29 14:17:51 ~ File "/app/huggingface_inference_toolkit/handler.py", line 17, in __init__
2024/01/29 14:17:51 ~ self.pipeline = get_pipeline(model_dir=model_dir, task=task)
2024/01/29 14:17:51 ~ File "/app/huggingface_inference_toolkit/utils.py", line 261, in get_pipeline
2024/01/29 14:17:51 ~ hf_pipeline = pipeline(task=task, model=model_dir, device=device, **kwargs)
2024/01/29 14:17:51 ~ File "/opt/conda/lib/python3.9/site-packages/transformers/pipelines/__init__.py", line 705, in pipeline
2024/01/29 14:17:51 ~ config = AutoConfig.from_pretrained(model, _from_pipeline=task, **hub_kwargs, **model_kwargs)
2024/01/29 14:17:51 ~ File "/opt/conda/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 998, in from_pretrained
2024/01/29 14:17:51 ~ config_class = CONFIG_MAPPING[config_dict["model_type"]]
2024/01/29 14:17:51 ~ File "/opt/conda/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 710, in __getitem__
2024/01/29 14:17:51 ~ raise KeyError(key)
2024/01/29 14:17:51 ~ KeyError: 'mistral'
4vur5 2024-01-29T19:17:51.981+00:00
2024/01/29 14:17:51 ~ Application startup failed. Exiting.

I configured the endpoint on the feature-extraction task with a Default container type.