Repository: simpler-env/SimplerEnv
Issue: A segmentation fault occurred during the evaluation process on the headless server.
Root cause: Environment Incompatibility

Thank you for your great work. I encounter a bug when evaluating LAPA in simpler_env.
This is evaluation script:
```
CUDA_VISIBLE_DEVICES=${gpu_id} python -m simpler_env.main_inference_lapa \
     --policy-model ${policy_model} --ckpt-path ${ckpt_path} \
      --robot ${robot} --policy-setup widowx_bridge --action-scale-file ${action_scale_file} \
      --control-freq 5 --sim-freq 500 --max-episode-steps 60 \
      --env-name PutCarrotOnPlateInScene-v0 --scene-name ${scene_name} \
      --rgb-overlay-path ${rgb_overlay_path} \
      --robot-init-x ${robot_init_x} ${robot_init_x} 1 --robot-init-y ${robot_init_y} ${robot_init_y} 1 --obj-variation-mode episode --obj-episode-range 0 24\
      --robot-init-rot-quat-center 0 0 0 1 --robot-init-rot-rpy-range 0 0 1 0 0 1 0 0 1 \
      --load-llama-config 7b --update-llama-config "dict(action_vocab_size=245,sample_mode='text',theta=50000000,max_sequence_length=32768,scan_attention=False,scan_query_chunk_size=128,scan_key_chunk_size=128,scan_mlp=False,scan_mlp_chunk_size=8192,scan_layers=True)" \
      --vocab-file ${vocab_file} --vqgan-checkpoint ${vqgan_file} \
      --tokens-per-action 7 --tokens-per-delta 4 \
      --mesh-dim ${mesh_dim}
```

Whenever I run the function run_maniskill2_eval_single_episode() two or three times, the program will encounter a segmentation fault.
Every time, the program will crash before the return of run_maniskill2_eval_single_episode() for the third time (possibly during the destructor process).
```
scripts/lapa_bridge.sh: line 43: 3315270 Segmentation fault      (core dumped) CUDA_VISIBLE_DEVICES=${gpu_id} python -m simpler_env.main_inference_lapa --policy-model ${policy_model} --ckpt-path ${ckpt_path} --robot ${robot} --policy-setup widowx_bridge --action-scale-file ${action_scale_file} --control-freq 5 --sim-freq 500 --max-episode-steps 60 --env-name PutCarrotOnPlateInScene-v0 --scene-name ${scene_name} --rgb-overlay-path ${rgb_overlay_path} --robot-init-x ${robot_init_x} ${robot_init_x} 1 --robot-init-y ${robot_init_y} ${robot_init_y} 1 --obj-variation-mode episode --obj-episode-range 4 24 --robot-init-rot-quat-center 0 0 0 1 --robot-init-rot-rpy-range 0 0 1 0 0 1 0 0 1 --load-llama-config 7b --update-llama-config "dict(action_vocab_size=245,sample_mode='text',theta=50000000,max_sequence_length=32768,scan_attention=False,scan_query_chunk_size=128,scan_key_chunk_size=128,scan_mlp=False,scan_mlp_chunk_size=8192,scan_layers=True)" --vocab-file ${vocab_file} --vqgan-checkpoint ${vqgan_file} --tokens-per-action 7 --tokens-per-delta 4 --mesh-dim ${mesh_dim}
```
Some of my key dependent versions are as follows:
```
# Name                    Version                   Build  Channel
albumentations            2.0.8                    pypi_0    pypi
annotated-doc             0.0.4                    pypi_0    pypi
annotated-types           0.7.0                    pypi_0    pypi
anyio                     4.12.0                   pypi_0    pypi
apache-beam               2.54.0                   pypi_0    pypi
array-record              0.8.1                    pypi_0    pypi
asttokens                 3.0.1                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
async-timeout             5.0.1                    pypi_0    pypi
attrs                     25.4.0                   pypi_0    pypi
beautifulsoup4            4.14.3                   pypi_0    pypi
bzip2                     1.0.8                h5eee18b_6  
ca-certificates           2025.12.2            h06a4308_0  
cachetools                6.2.4                    pypi_0    pypi
certifi                   2025.11.12               pypi_0    pypi
charset-normalizer        3.4.4                    pypi_0    pypi
chex                      0.1.86                   pypi_0    pypi
click                     8.3.1                    pypi_0    pypi
cloudpickle               2.2.1                    pypi_0    pypi
coacd                     1.0.7                    pypi_0    pypi
contourpy                 1.3.2                    pypi_0    pypi
crcmod                    1.7                      pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
datasets                  2.13.0                   pypi_0    pypi
decorator                 5.2.1                    pypi_0    pypi
decord                    0.6.0                    pypi_0    pypi
dill                      0.4.0                    pypi_0    pypi
distrax                   0.1.5                    pypi_0    pypi
dlimp                     0.0.1                    pypi_0    pypi
dm-tree                   0.1.9                    pypi_0    pypi
dnspython                 2.8.0                    pypi_0    pypi
docopt                    0.6.2                    pypi_0    pypi
einops                    0.8.1                    pypi_0    pypi
etils                     1.13.0                   pypi_0    pypi
exceptiongroup            1.3.1                    pypi_0    pypi
executing                 2.2.1                    pypi_0    pypi
expat                     2.7.3                h7354ed3_4  
farama-notifications      0.0.4                    pypi_0    pypi
fastapi                   0.124.4                  pypi_0    pypi
fastavro                  1.12.1                   pypi_0    pypi
fasteners                 0.20                     pypi_0    pypi
filelock                  3.20.1                   pypi_0    pypi
flatbuffers               25.9.23                  pypi_0    pypi
flax                      0.7.0                    pypi_0    pypi

imageio-ffmpeg            0.6.0                    pypi_0    pypi
importlib-resources       6.5.2                    pypi_0    pypi
ipdb                      0.13.13                  pypi_0    pypi
ipython                   8.37.0                   pypi_0    pypi
jax                       0.4.23                   pypi_0    pypi
jax-jumpy                 1.0.0                    pypi_0    pypi
jaxlib                    0.4.23+cuda12.cudnn89          pypi_0    pypi
jedi                      0.19.2                   pypi_0    pypi
js2py                     0.74                     pypi_0    pypi
jsonpickle                3.4.2                    pypi_0    pypi
jsonschema                4.25.1                   pypi_0    pypi
jsonschema-specifications 2025.9.1                 pypi_0    pypi
keras                     2.15.0                   pypi_0    pypi
kiwisolver                1.4.9                    pypi_0    pypi
ld_impl_linux-64          2.44                 h153f514_2  
libclang                  18.1.1                   pypi_0    pypi
libexpat                  2.7.3                h7354ed3_4  
libffi                    3.4.4                h6a678d5_1  
libgcc                    15.2.0               h69a1729_7  
libgcc-ng                 15.2.0               h166f726_7  
libgomp                   15.2.0               h4751f2c_7  
libnsl                    2.0.0                h5eee18b_0  
libstdcxx                 15.2.0               h39759b7_7  
libstdcxx-ng              15.2.0               hc03a8fd_7  
libuuid                   1.41.5               h5eee18b_0  
libxcb                    1.17.0               h9b100fa_0  
libzlib                   1.3.1                hb25bd0a_0  
mani-skill2-real2sim      0.5.3                    pypi_0    pypi
markdown                  3.10                     pypi_0    pypi
markdown-it-py            4.0.0                    pypi_0    pypi
markupsafe                3.0.3                    pypi_0    pypi
matplotlib                3.10.8                   pypi_0    pypi
matplotlib-inline         0.2.1                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mediapy                   1.2.0                    pypi_0    pypi
ml-collections            1.1.0                    pypi_0    pypi
ml-dtypes                 0.3.2                    pypi_0    pypi
msgpack                   1.1.2                    pypi_0    pypi
multidict                 6.7.0                    pypi_0    pypi
multiprocess              0.70.14                  pypi_0    pypi
narwhals                  2.14.0                   pypi_0    pypi
ncurses                   6.5                  h7934f7d_0  
nest-asyncio              1.6.0                    pypi_0    pypi
numpy                     1.24.4                   pypi_0    pypi
nvidia-cublas-cu12        12.3.4.1                 pypi_0    pypi
nvidia-cuda-cupti-cu12    12.3.101                 pypi_0    pypi
nvidia-cuda-nvcc-cu12     12.3.103                 pypi_0    pypi
nvidia-cuda-nvrtc-cu12    12.3.103                 pypi_0    pypi
nvidia-cuda-runtime-cu12  12.3.101                 pypi_0    pypi
nvidia-cudnn-cu12         8.9.7.29                 pypi_0    pypi
nvidia-cufft-cu12         11.0.12.1                pypi_0    pypi
nvidia-curand-cu12        10.3.4.107               pypi_0    pypi
nvidia-cusolver-cu12      11.5.4.101               pypi_0    pypi
nvidia-cusparse-cu12      12.2.0.103               pypi_0    pypi
nvidia-nccl-cu12          2.19.3                   pypi_0    pypi
nvidia-nvjitlink-cu12     12.3.101                 pypi_0    pypi
oauthlib                  3.3.1                    pypi_0    pypi
objsize                   0.7.1                    pypi_0    pypi
opencv-python-headless    4.11.0.86                pypi_0    pypi
openssl                   3.0.18               hd6dcaed_0  
opt-einsum                3.4.0                    pypi_0    pypi
optax                     0.1.7                    pypi_0    pypi
orbax-checkpoint          0.5.16                   pypi_0    pypi
orjson                    3.11.5                   pypi_0    pypi
packaging                 25.0                     pypi_0    pypi
pandas                    2.3.3                    pypi_0    pypi
parso                     0.8.5                    pypi_0    pypi
pexpect                   4.9.0                    pypi_0    pypi
pillow                    12.0.0                   pypi_0    pypi
pip                       25.3               pyhc872135_0  
python                    3.10.19              h6fa692b_0  
python-dateutil           2.9.0.post0              pypi_0    pypi
pytz                      2025.2                   pypi_0    pypi
pyyaml                    6.0.3                    pypi_0    pypi
readline                  8.3                  hc2a1206_0  
referencing               0.37.0                   pypi_0    pypi
regex                     2025.11.3                pypi_0    pypi
requests                  2.32.5                   pypi_0    pypi
requests-oauthlib         2.0.0                    pypi_0    pypi
rich                      14.2.0                   pypi_0    pypi
rlds                      0.1.8                    pypi_0    pypi
rpds-py                   0.30.0                   pypi_0    pypi
rsa                       4.9.1                    pypi_0    pypi
rtree                     1.4.1                    pypi_0    pypi
ruckig                    0.15.3                   pypi_0    pypi
sapien                    2.2.2                    pypi_0    pypi
scipy                     1.12.0                   pypi_0    pypi
sentencepiece             0.2.1                    pypi_0    pypi
sentry-sdk                2.48.0                   pypi_0    pypi
setuptools                80.9.0          py310h06a4308_0  
simsimd                   6.5.8                    pypi_0    pypi
six                       1.17.0                   pypi_0    pypi
smmap                     5.0.2                    pypi_0    pypi
soupsieve                 2.8                      pypi_0    pypi
sqlite                    3.51.0               h2a70700_0  
stack-data                0.6.3                    pypi_0    pypi
starlette                 0.50.0                   pypi_0    pypi
stringzilla               4.5.1                    pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
tensorboard               2.15.2                   pypi_0    pypi
tensorboard-data-server   0.7.2                    pypi_0    pypi
tensorflow                2.15.1                   pypi_0    pypi
tensorflow-datasets       4.9.4                    pypi_0    pypi
tensorflow-estimator      2.15.0                   pypi_0    pypi
tensorflow-hub            0.16.0                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.37.1                   pypi_0    pypi
tensorflow-metadata       1.16.1                   pypi_0    pypi
tensorflow-probability    0.25.0                   pypi_0    pypi
tensorstore               0.1.74                   pypi_0    pypi
termcolor                 3.2.0                    pypi_0    pypi
tiktoken                  0.12.0                   pypi_0    pypi
tk                        8.6.15               h54e0aa7_0  
tokenizers                0.13.3                   pypi_0    pypi
toml                      0.10.2                   pypi_0    pypi
tomli                     2.3.0                    pypi_0    pypi
toolz                     1.1.0                    pypi_0    pypi
tqdm                      4.67.1                   pypi_0    pypi
traitlets                 5.14.3                   pypi_0    pypi
transformers              4.29.2                   pypi_0    pypi
transforms3d              0.4.2                    pypi_0    pypi
```
my VulkanInfo can output normally, and I ran this program on the A6000 server.
I also tried the following commands:
```
export SAPIEN_RENDER_SYSTEM=egl
export SAPIEN_NO_DISPLAY=1
export DISPLAY=""
```
Could you please tell me how I should solve this problem?

Could it be that the problem lies in the fact that I installed opencv-python-headless?

Additionally, I did not install the following dependencies: 
```
tf_agents==0.19.0
dm-reverb[tensorflow]==0.14.0
tfds-nightly==4.9.4.dev202403220044
tfp-nightly==0.25.0.dev20240322. 
```
Will this have any impact?

> Could it be that the problem lies in the fact that I installed opencv-python-headless?

I downloaded opencv-python but it still doesn't work.

I use gdb and get the information
```
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
tls_get_addr_tail (dtv=0x8822ca0, the_map=0x0, ti=<optimized out>, ti=<optimized out>) at ../elf/dl-tls.c:758
758     ../elf/dl-tls.c: No such file or directory.
#0  tls_get_addr_tail (dtv=0x8822ca0, the_map=0x0, ti=<optimized out>, ti=<optimized out>) at ../elf/dl-tls.c:758
#1  0x00007ffff7fe9dec in __tls_get_addr () at ../sysdeps/x86_64/tls_get_addr.S:55
#2  0x00007fe3ca89751e in ?? () from /lib/x86_64-linux-gnu/libnvidia-gpucomp.so.580.76.05
#3  0x00007ffff7c82fde in __cxa_finalize (d=0x7fe3cbe55720) at cxa_finalize.c:83
#4  0x00007fe3c7e55616 in ?? () from /lib/x86_64-linux-gnu/libnvidia-gpucomp.so.580.76.05
#5  0x00007fdea68b7500 in ?? ()
#6  0xfffffffffffffff8 in ?? ()
#7  0x00007fffffffb970 in ?? ()
#8  0x00007fe3cae0c7bd in ?? () from /lib/x86_64-linux-gnu/libnvidia-gpucomp.so.580.76.05
#9  0xfffffffffffffff8 in ?? ()
#10 0x00007ffff7d9cba5 in __GI__dl_catch_exception (exception=<optimized out>, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:182
#11 0x00007ffff7fe5f36 in _dl_close_worker (map=<optimized out>, force=<optimized out>) at dl-close.c:297
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
```

The problem occurred when the CUDA version of the graphics card driver was 13.0. I tried running the server with the graphics card driver set to 12.4, and it worked successfully.