runtime error

d negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-10-06 05:38:35.945234: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-10-06 05:38:35.945413: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero Traceback (most recent call last): File "app.py", line 14, in <module> sd_dreambooth_model_1=StableDiffusionPy(resolution, resolution, download_weights=False, jit_compile=True) File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/stable_diffusion_tf/stable_diffusion.py", line 24, in __init__ text_encoder, diffusion_model, decoder, encoder = get_models(img_height, img_width, download_weights=download_weights) File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/stable_diffusion_tf/stable_diffusion.py", line 229, in get_models embeds = CLIPTextTransformer()([input_word_ids, input_pos_ids]) File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/stable_diffusion_tf/clip_encoder.py", line 115, in __init__ super().__init__() File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/tensorflow/python/trackable/base.py", line 205, in _method_wrapper result = method(self, *args, **kwargs) File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler raise e.with_traceback(filtered_tb) from None File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 588, in ensure_initialized context_handle = pywrap_tfe.TFE_NewContext(opts) tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version

Container logs:

Fetching error logs...