runtime error

Space failed to start. Exit code: 1. Reason: installed properly. ['.'] ERROR:absl:At most one of `num_partitions` or `model_parallel_submesh` can be set. Got `num_partitions=1` and `model_parallel_submesh`=(1, 1, 1, 1). A ValueError will be raised beginning March 21, 2022. WARNING:jax._src.xla_bridge:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.) /home/user/.local/lib/python3.8/site-packages/jax/_src/xla_bridge.py:658: UserWarning: jax.host_count has been renamed to jax.process_count. This alias will eventually be removed; please update your code. warnings.warn( WARNING:absl:Tiling device assignment mesh by hosts, which may lead to reduced XLA collective performance. To avoid this, modify the model parallel submesh or run with more tasks per host. Traceback (most recent call last): File "app.py", line 240, in <module> inference_model = InferenceModel('/home/user/app/checkpoints/mt3/', 'mt3') File "app.py", line 89, in __init__ self.restore_from_checkpoint(checkpoint_path) File "app.py", line 135, in restore_from_checkpoint self._train_state = train_state_initializer.from_checkpoint_or_scratch( File "/home/user/.local/lib/python3.8/site-packages/t5x/utils.py", line 1288, in from_checkpoint_or_scratch return self.from_checkpoint( File "/home/user/.local/lib/python3.8/site-packages/t5x/utils.py", line 1267, in from_checkpoint train_states = [ File "/home/user/.local/lib/python3.8/site-packages/t5x/utils.py", line 1267, in <listcomp> train_states = [ File "/home/user/.local/lib/python3.8/site-packages/t5x/utils.py", line 1257, in from_checkpoints yield _restore_path(path, restore_cfg), path File "/home/user/.local/lib/python3.8/site-packages/t5x/utils.py", line 1249, in _restore_path return restore_checkpointer.restore( File "/home/user/.local/lib/python3.8/site-packages/t5x/checkpoints.py", line 1047, in restore raise ValueError(f'Path is not a valid T5X checkpoint: {ckpt_path}') ValueError: Path is not a valid T5X checkpoint: /home/user/app/checkpoints/mt3/

Container logs: