Sagemaker SDK Deployment

#20
by Sayel - opened

What is the required container image version to deploy this model?

Mistral AI_ org

Hi! You can retrieve the setup steps for SageMaker from the model card by clicking on Deploy > Amazon SageMaker (top right).
The model should also be available from SageMaker Jumpstart soon.

sophiamyang changed discussion status to closed

Using the SDK I am getting this Sagemaker error:

torch.cuda.OutOfMemoryError: CUDA out of memory.
Tried to allocate 192.00 MiB. GPU 4 has a total capacty of 22.20 GiB of which 77.12 MiB is free. 
Process 42028 has 22.12 GiB memory in use. 
Of the allocated memory 20.85 GiB is allocated by PyTorch, and 111.76 MiB is reserved by PyTorch but unallocated.

If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  
See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Is anyone else getting the same error?

Sign up or log in to comment