Repository: tensorflow
Issue: TF 2.18 with GPU does not detect GPU, Cannot dlopen some GPU libraries, in a container
Labels: stat:awaiting tensorflower,type:build/install,comp:gpu,TF 2.18

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

2.18

### Custom code

Yes

### OS platform and distribution

Linux Centos 7.9, RHEL 8, RHEL 9

### Mobile device

_No response_

### Python version

3.11.0rc1

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

550.90.07

### GPU model and memory

_No response_

### Current behavior?

After discussing this on the [Apptainer Git](https://github.com/apptainer/apptainer/issues/2706#issuecomment-2613253071) we determined the latest TF-GPU running 2.18.0 does not register any GPUs. Older versions like 2.7.1-gpu work just fine.

`apptainer run --nv  /apps/Miniforge/lib/python3.12/site-packages/containers/tensorflow/tensorflow/latest-gpu/tensorflow-tensorflow-latest-gpu-sha256\:1f16fbd9be8bb84891de12533e332bbd500511caeb5cf4db501dbe39d422f9c7.sif python`


```
import tensorflow as tf
2025-01-24 15:03:27.629215: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1737749008.639844   35316 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1737749008.847756   35316 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-24 15:03:31.499335: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

```
```
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
W0000 00:00:1737749068.599039   35316 gpu_device.cc:2344] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Num GPUs Available:  0

```
```
>>> print(tf.__version__)
2.18.0
```


### Standalone code to reproduce the issue

```shell
shpc install tensorflow/tensorflow:latest-gpu

or

apptainer pull docker://tensorflow/tensorflow:latest-gpu

apptainer run --nv  /apps/Miniforge/lib/python3.12/site-packages/containers/tensorflow/tensorflow/latest-gpu/tensorflow-tensorflow-latest-gpu-sha256\:1f16fbd9be8bb84891de12533e332bbd500511caeb5cf4db501dbe39d422f9c7.sif python

python
import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
```

### Relevant log output

```shell
apptainer --debug exec --nv  /apps/Miniforge/lib/python3.12/site-packages/containers/tensorflow/tensorflow/latest-gpu/tensorflow-tensorflow-latest-gpu-sha256:1f16fbd9be8bb84891de12533e332bbd500511caeb5cf4db501dbe39d422f9c7.sif python
DEBUG   [U=0,P=1355208]    persistentPreRun()            Apptainer version: 1.3.6-1
DEBUG   [U=0,P=1355208]    persistentPreRun()            Parsing configuration file /etc/apptainer/apptainer.conf
DEBUG   [U=0,P=1355208]    SetBinaryPath()               Setting binary path to /usr/libexec/apptainer/bin:/usr/share/Modules/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/TurboVNC/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG   [U=0,P=1355208]    SetBinaryPath()               Using that path for all binaries
DEBUG   [U=0,P=1355208]    handleConfDir()               /root/.apptainer already exists. Not creating.
DEBUG   [U=0,P=1355208]    handleRemoteConf()            Ensuring file permission of 0600 on /root/.apptainer/remote.yaml
DEBUG   [U=0,P=1355208]    setUmask()                    Saving umask 0002 for propagation into container
DEBUG   [U=0,P=1355208]    checkEncryptionKey()          Checking for encrypted system partition
DEBUG   [U=0,P=1355208]    Init()                        Image format detection
DEBUG   [U=0,P=1355208]    Init()                        Check for sandbox image format
DEBUG   [U=0,P=1355208]    Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=0,P=1355208]    Init()                        Check for sif image format
DEBUG   [U=0,P=1355208]    Init()                        sif image format detected
VERBOSE [U=0,P=1355208]    SetGPUConfig()                'always use nv = yes' found in apptainer.conf
DEBUG   [U=0,P=1355208]    setNVLegacyConfig()           Using legacy binds for nv GPU setup
VERBOSE [U=0,P=1355208]    NvidiaIpcsPath()              persistenced socket /var/run/nvidia-persistenced/socket not found
DEBUG   [U=0,P=1355208]    findOnPath()                  Found "ldconfig" at "/sbin/ldconfig"
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding SHELL environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding SUDO_GID environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding HISTCONTROL environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding no_proxy environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding HOSTNAME environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding HISTSIZE environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding SBATCH_PARTITION environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding GUESTFISH_OUTPUT environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding SLURM_PARTITION environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding SUDO_COMMAND environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding SUDO_USER environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LMOD_DIR environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding PWD environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LOGNAME environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding MODULESHOME environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding MANPATH environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding GUESTFISH_RESTORE environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding __MODULES_SHARE_MANPATH environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding SSH_ASKPASS environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LANG environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LS_COLORS environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LMOD_SETTARG_FULL_SUPPORT environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding GUESTFISH_PS1 environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding https_proxy environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LMOD_VERSION environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding MODULEPATH_ROOT environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LMOD_PKG environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding TERM environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LESSOPEN environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding USER environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding NO_PROXY environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding MODULES_RUN_QUARANTINE environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LOADEDMODULES environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding SHLVL environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding BASH_ENV environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LMOD_sys environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding HTTPS_PROXY environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding GUESTFISH_INIT environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding HTTP_PROXY environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding http_proxy environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding S_COLORS environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding __MODULES_LMINIT environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding which_declare environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding XDG_DATA_DIRS environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding MODULEPATH environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding SUDO_UID environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding LMOD_CMD environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding MAIL environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding MODULES_CMD environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding BASH_FUNC_ml%% environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding BASH_FUNC_which%% environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding BASH_FUNC_module%% environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding BASH_FUNC_scl%% environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding BASH_FUNC__module_raw%% environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding _ environment variable
VERBOSE [U=0,P=1355208]    SetContainerEnv()             Not forwarding APPTAINER_DEBUG environment variable
DEBUG   [U=0,P=1355208]    SetContainerEnv()             Forwarding USER_PATH environment variable
VERBOSE [U=0,P=1355208]    SetContainerEnv()             Setting HOME=/root
VERBOSE [U=0,P=1355208]    SetContainerEnv()             Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG   [U=0,P=1355208]    InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=1355208]    SetuidMountAllowed()          Kernel squashfs mount allowed because running as root
DEBUG   [U=0,P=1355208]    init()                        Use starter binary /usr/libexec/apptainer/bin/starter
VERBOSE [U=0,P=1355208]    print()                       Set messagelevel to: 5
VERBOSE [U=0,P=1355208]    init()                        Starter initialization
VERBOSE [U=0,P=1355208]    is_suid()                     Check if we are running as setuid: 0
DEBUG   [U=0,P=1355208]    read_engine_config()          Read engine configuration
DEBUG   [U=0,P=1355208]    init()                        Wait completion of stage1
DEBUG   [U=0,P=1355224]    set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=1355224]    init()                        Spawn stage 1
DEBUG   [U=0,P=1355224]    func1()                       executablePath is /usr/libexec/apptainer/bin/starter
DEBUG   [U=0,P=1355224]    func1()                       starter was not relocated from /usr/libexec
DEBUG   [U=0,P=1355224]    func1()                       Install prefix is /usr
DEBUG   [U=0,P=1355224]    startup()                     apptainer runtime engine selected
VERBOSE [U=0,P=1355224]    startup()                     Execute stage 1
DEBUG   [U=0,P=1355224]    StageOne()                    Entering stage 1
DEBUG   [U=0,P=1355224]    InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=1355224]    prepareRootCaps()             Root full capabilities
DEBUG   [U=0,P=1355224]    prepareAutofs()               Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Found "/home" as autofs mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Found "/share" as autofs mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Found "/misc" as autofs mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Found "/net" as autofs mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Found "/mnt/smb/locker" as autofs mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Found "/mnt/smb/labshare" as autofs mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Found "/mnt/smb/staging" as autofs mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Could not keep file descriptor for home directory /root: no mount point
DEBUG   [U=0,P=1355224]    prepareAutofs()               Could not keep file descriptor for current working directory /root: no mount point
DEBUG   [U=0,P=1355224]    Init()                        Image format detection
DEBUG   [U=0,P=1355224]    Init()                        Check for sandbox image format
DEBUG   [U=0,P=1355224]    Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=0,P=1355224]    Init()                        Check for sif image format
DEBUG   [U=0,P=1355224]    Init()                        sif image format detected
DEBUG   [U=0,P=1355224]    setSessionLayer()             Using overlay because it is not disabled
DEBUG   [U=0,P=1355224]    PrepareConfig()               image driver is 
VERBOSE [U=0,P=1355208]    wait_child()                  stage 1 exited with status 0
DEBUG   [U=0,P=1355208]    cleanup_fd()                  Close file descriptor 4
DEBUG   [U=0,P=1355208]    cleanup_fd()                  Close file descriptor 5
DEBUG   [U=0,P=1355208]    cleanup_fd()                  Close file descriptor 6
DEBUG   [U=0,P=1355208]    init()                        Set child signal mask
DEBUG   [U=0,P=1355208]    init()                        Create socketpair for master communication channel
DEBUG   [U=0,P=1355208]    init()                        Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=0,P=1355208]    init()                        Spawn master process
DEBUG   [U=0,P=1355230]    set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=1355230]    create_namespace()            Create mount namespace
VERBOSE [U=0,P=1355208]    enter_namespace()             Entering in mount namespace
DEBUG   [U=0,P=1355208]    enter_namespace()             Opening namespace file ns/mnt
VERBOSE [U=0,P=1355230]    create_namespace()            Create mount namespace
VERBOSE [U=0,P=1355231]    init()                        Spawn RPC server
DEBUG   [U=0,P=1355208]    func1()                       executablePath is /usr/libexec/apptainer/bin/starter
DEBUG   [U=0,P=1355208]    func1()                       starter was not relocated from /usr/libexec
DEBUG   [U=0,P=1355208]    func1()                       Install prefix is /usr
DEBUG   [U=0,P=1355231]    func1()                       executablePath is /usr/libexec/apptainer/bin/starter
DEBUG   [U=0,P=1355231]    func1()                       starter was not relocated from /usr/libexec
DEBUG   [U=0,P=1355231]    func1()                       Install prefix is /usr
DEBUG   [U=0,P=1355208]    startup()                     apptainer runtime engine selected
VERBOSE [U=0,P=1355208]    startup()                     Execute master process
DEBUG   [U=0,P=1355231]    startup()                     apptainer runtime engine selected
VERBOSE [U=0,P=1355231]    startup()                     Serve RPC requests
DEBUG   [U=0,P=1355208]    InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=1355208]    setupSessionLayout()          Using Layer system: overlay
DEBUG   [U=0,P=1355208]    setupOverlayLayout()          Creating overlay SESSIONDIR layout
DEBUG   [U=0,P=1355208]    addRootfsMount()              Mount rootfs in read-only mode
DEBUG   [U=0,P=1355208]    addRootfsMount()              Image type is 4096
DEBUG   [U=0,P=1355208]    addRootfsMount()              Mounting block [squashfs] image: /share/apps/Miniforge/lib/python3.12/site-packages/containers/tensorflow/tensorflow/latest-gpu/tensorflow-tensorflow-latest-gpu-sha256:1f16fbd9be8bb84891de12533e332bbd500511caeb5cf4db501dbe39d422f9c7.sif
DEBUG   [U=0,P=1355208]    addKernelMount()              Checking configuration file for 'mount proc'
DEBUG   [U=0,P=1355208]    addKernelMount()              Adding proc to mount list
VERBOSE [U=0,P=1355208]    addKernelMount()              Default mount: /proc:/proc
DEBUG   [U=0,P=1355208]    addKernelMount()              Checking configuration file for 'mount sys'
DEBUG   [U=0,P=1355208]    addKernelMount()              Adding sysfs to mount list
VERBOSE [U=0,P=1355208]    addKernelMount()              Default mount: /sys:/sys
DEBUG   [U=0,P=1355208]    addDevMount()                 Checking configuration file for 'mount dev'
DEBUG   [U=0,P=1355208]    addDevMount()                 Adding dev to mount list
VERBOSE [U=0,P=1355208]    addDevMount()                 Default mount: /dev:/dev
DEBUG   [U=0,P=1355208]    addHostMount()                Not mounting host file systems per configuration
VERBOSE [U=0,P=1355208]    addBindsMount()               Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=0,P=1355208]    addBindsMount()               Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG   [U=0,P=1355208]    addHomeStagingDir()           Staging home directory (/root) at /var/lib/apptainer/mnt/session/root
DEBUG   [U=0,P=1355208]    addHomeMount()                Adding home directory mount [/var/lib/apptainer/mnt/session/root:/root] to list using layer: overlay
DEBUG   [U=0,P=1355208]    addTmpMount()                 Checking for 'mount tmp' in configuration file
DEBUG   [U=0,P=1355208]    addScratchMount()             Not mounting scratch directory: Not requested
DEBUG   [U=0,P=1355208]    addLibsMount()                Checking for 'user bind control' in configuration file
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libOpenCL.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libOpenGL.so.0 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-cfg.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libEGL.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-eglcore.so.550.120 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-ml.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvcuvid.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-gtk3.so.550.120 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-glvkspirv.so.550.120 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libcudadebugger.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLESv2.so.2 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-ptxjitcompiler.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLESv2.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGL.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLX_nvidia.so.0 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLESv1_CM.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-tls.so.550.120 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLdispatch.so.0 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-encode.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libOpenCL.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-ptxjitcompiler.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-encode.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-nvvm.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-glsi.so.550.120 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-opticalflow.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-opencl.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-egl-wayland.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLX.so.0 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvoptix.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-gpucomp.so.550.120 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGL.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLESv2_nvidia.so.2 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLESv1_CM_nvidia.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-nvvm.so.4 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libEGL.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-gtk2.so.550.120 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-ml.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-fbc.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-fbc.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvcuvid.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-opticalflow.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLX.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLESv1_CM.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-cfg.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libcuda.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libvdpau_nvidia.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-rtcore.so.550.120 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libcuda.so.1 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libOpenGL.so to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libEGL_nvidia.so.0 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libnvidia-glcore.so.550.120 to mount list
DEBUG   [U=0,P=1355208]    addLibsMount()                Add library /lib64/libGLdispatch.so to mount list
DEBUG   [U=0,P=1355208]    addFilesMount()               Checking for 'user bind control' in configuration file
DEBUG   [U=0,P=1355208]    addFilesMount()               Adding file /bin/nvidia-persistenced:/usr/bin/nvidia-persistenced to mount list
DEBUG   [U=0,P=1355208]    addFilesMount()               Adding file /bin/nvidia-cuda-mps-control:/usr/bin/nvidia-cuda-mps-control to mount list
DEBUG   [U=0,P=1355208]    addFilesMount()               Adding file /bin/nvidia-cuda-mps-server:/usr/bin/nvidia-cuda-mps-server to mount list
DEBUG   [U=0,P=1355208]    addFilesMount()               Adding file /bin/nvidia-smi:/usr/bin/nvidia-smi to mount list
DEBUG   [U=0,P=1355208]    addFilesMount()               Adding file /bin/nvidia-debugdump:/usr/bin/nvidia-debugdump to mount list
DEBUG   [U=0,P=1355208]    addResolvConfMount()          Adding /etc/resolv.conf to mount list
VERBOSE [U=0,P=1355208]    addResolvConfMount()          Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG   [U=0,P=1355208]    addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG   [U=0,P=1355208]    create()                      Mount all
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting tmpfs to /var/lib/apptainer/mnt/session
DEBUG   [U=0,P=1355208]    mountImage()                  Mounting loop device /dev/loop0 to /var/lib/apptainer/mnt/session/rootfs of type squashfs
DEBUG   [U=0,P=1355208]    createCwdDir()                Using /root as current working directory
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting overlay to /var/lib/apptainer/mnt/session/final
DEBUG   [U=0,P=1355208]    mountGeneric()                Unmounting and remounting overlay
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final
DEBUG   [U=0,P=1355208]    setPropagationMount()         Set RPC mount propagation flag to SLAVE
VERBOSE [U=0,P=1355208]    Passwd()                      Checking for template passwd file: /var/lib/apptainer/mnt/session/rootfs/etc/passwd
VERBOSE [U=0,P=1355208]    Passwd()                      Creating passwd content
VERBOSE [U=0,P=1355208]    Passwd()                      Creating template passwd file and injecting user data: /var/lib/apptainer/mnt/session/rootfs/etc/passwd
DEBUG   [U=0,P=1355208]    addIdentityMount()            Adding /etc/passwd to mount list
VERBOSE [U=0,P=1355208]    addIdentityMount()            Default mount: /etc/passwd:/etc/passwd
VERBOSE [U=0,P=1355208]    Group()                       Checking for template group file: /var/lib/apptainer/mnt/session/rootfs/etc/group
VERBOSE [U=0,P=1355208]    Group()                       Creating group content
DEBUG   [U=0,P=1355208]    addIdentityMount()            Adding /etc/group to mount list
VERBOSE [U=0,P=1355208]    addIdentityMount()            Default mount: /etc/group:/etc/group
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /dev to /var/lib/apptainer/mnt/session/final/dev
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /etc/localtime to /var/lib/apptainer/mnt/session/final/etc/localtime
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/etc/localtime
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /etc/hosts to /var/lib/apptainer/mnt/session/final/etc/hosts
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/etc/hosts
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /proc to /var/lib/apptainer/mnt/session/final/proc
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/proc
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting sysfs to /var/lib/apptainer/mnt/session/final/sys
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /root to /var/lib/apptainer/mnt/session/root
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/root
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /var/lib/apptainer/mnt/session/root to /var/lib/apptainer/mnt/session/final/root
DEBUG   [U=0,P=1355208]    func1()                       Container /tmp resolves to "/tmp"
DEBUG   [U=0,P=1355208]    func1()                       Container /var/tmp resolves to "/var/tmp"
VERBOSE [U=0,P=1355208]    func1()                       Default mount: /tmp:/tmp
VERBOSE [U=0,P=1355208]    func1()                       Default mount: /var/tmp:/var/tmp
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /tmp to /var/lib/apptainer/mnt/session/final/tmp
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/tmp
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /var/tmp to /var/lib/apptainer/mnt/session/final/var/tmp
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/var/tmp
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libOpenCL.so to /var/lib/apptainer/mnt/session/libs/libOpenCL.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libOpenCL.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libOpenGL.so.0 to /var/lib/apptainer/mnt/session/libs/libOpenGL.so.0
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libOpenGL.so.0
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-cfg.so.1 to /var/lib/apptainer/mnt/session/libs/libnvidia-cfg.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-cfg.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libEGL.so.1 to /var/lib/apptainer/mnt/session/libs/libEGL.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libEGL.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-eglcore.so.550.120 to /var/lib/apptainer/mnt/session/libs/libnvidia-eglcore.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-eglcore.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-ml.so to /var/lib/apptainer/mnt/session/libs/libnvidia-ml.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-ml.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvcuvid.so.1 to /var/lib/apptainer/mnt/session/libs/libnvcuvid.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvcuvid.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-gtk3.so.550.120 to /var/lib/apptainer/mnt/session/libs/libnvidia-gtk3.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-gtk3.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-glvkspirv.so.550.120 to /var/lib/apptainer/mnt/session/libs/libnvidia-glvkspirv.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-glvkspirv.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libcudadebugger.so.1 to /var/lib/apptainer/mnt/session/libs/libcudadebugger.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libcudadebugger.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLESv2.so.2 to /var/lib/apptainer/mnt/session/libs/libGLESv2.so.2
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLESv2.so.2
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-ptxjitcompiler.so.1 to /var/lib/apptainer/mnt/session/libs/libnvidia-ptxjitcompiler.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-ptxjitcompiler.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLESv2.so to /var/lib/apptainer/mnt/session/libs/libGLESv2.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLESv2.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGL.so to /var/lib/apptainer/mnt/session/libs/libGL.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGL.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLX_nvidia.so.0 to /var/lib/apptainer/mnt/session/libs/libGLX_nvidia.so.0
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLX_nvidia.so.0
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLESv1_CM.so.1 to /var/lib/apptainer/mnt/session/libs/libGLESv1_CM.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLESv1_CM.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-tls.so.550.120 to /var/lib/apptainer/mnt/session/libs/libnvidia-tls.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-tls.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLdispatch.so.0 to /var/lib/apptainer/mnt/session/libs/libGLdispatch.so.0
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLdispatch.so.0
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-encode.so.1 to /var/lib/apptainer/mnt/session/libs/libnvidia-encode.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-encode.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libOpenCL.so.1 to /var/lib/apptainer/mnt/session/libs/libOpenCL.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libOpenCL.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-ptxjitcompiler.so to /var/lib/apptainer/mnt/session/libs/libnvidia-ptxjitcompiler.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-ptxjitcompiler.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-encode.so to /var/lib/apptainer/mnt/session/libs/libnvidia-encode.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-encode.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-nvvm.so to /var/lib/apptainer/mnt/session/libs/libnvidia-nvvm.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-nvvm.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-glsi.so.550.120 to /var/lib/apptainer/mnt/session/libs/libnvidia-glsi.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-glsi.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-opticalflow.so to /var/lib/apptainer/mnt/session/libs/libnvidia-opticalflow.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-opticalflow.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-opencl.so.1 to /var/lib/apptainer/mnt/session/libs/libnvidia-opencl.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-opencl.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-egl-wayland.so.1 to /var/lib/apptainer/mnt/session/libs/libnvidia-egl-wayland.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-egl-wayland.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLX.so.0 to /var/lib/apptainer/mnt/session/libs/libGLX.so.0
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLX.so.0
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvoptix.so.1 to /var/lib/apptainer/mnt/session/libs/libnvoptix.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvoptix.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-gpucomp.so.550.120 to /var/lib/apptainer/mnt/session/libs/libnvidia-gpucomp.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-gpucomp.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGL.so.1 to /var/lib/apptainer/mnt/session/libs/libGL.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGL.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLESv2_nvidia.so.2 to /var/lib/apptainer/mnt/session/libs/libGLESv2_nvidia.so.2
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLESv2_nvidia.so.2
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLESv1_CM_nvidia.so.1 to /var/lib/apptainer/mnt/session/libs/libGLESv1_CM_nvidia.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLESv1_CM_nvidia.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-nvvm.so.4 to /var/lib/apptainer/mnt/session/libs/libnvidia-nvvm.so.4
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-nvvm.so.4
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libEGL.so to /var/lib/apptainer/mnt/session/libs/libEGL.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libEGL.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-gtk2.so.550.120 to /var/lib/apptainer/mnt/session/libs/libnvidia-gtk2.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-gtk2.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-ml.so.1 to /var/lib/apptainer/mnt/session/libs/libnvidia-ml.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-ml.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-fbc.so to /var/lib/apptainer/mnt/session/libs/libnvidia-fbc.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-fbc.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-fbc.so.1 to /var/lib/apptainer/mnt/session/libs/libnvidia-fbc.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-fbc.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvcuvid.so to /var/lib/apptainer/mnt/session/libs/libnvcuvid.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvcuvid.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-opticalflow.so.1 to /var/lib/apptainer/mnt/session/libs/libnvidia-opticalflow.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-opticalflow.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLX.so to /var/lib/apptainer/mnt/session/libs/libGLX.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLX.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLESv1_CM.so to /var/lib/apptainer/mnt/session/libs/libGLESv1_CM.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLESv1_CM.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-cfg.so to /var/lib/apptainer/mnt/session/libs/libnvidia-cfg.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-cfg.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libcuda.so to /var/lib/apptainer/mnt/session/libs/libcuda.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libcuda.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libvdpau_nvidia.so to /var/lib/apptainer/mnt/session/libs/libvdpau_nvidia.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libvdpau_nvidia.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-rtcore.so.550.120 to /var/lib/apptainer/mnt/session/libs/libnvidia-rtcore.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-rtcore.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libcuda.so.1 to /var/lib/apptainer/mnt/session/libs/libcuda.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libcuda.so.1
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libOpenGL.so to /var/lib/apptainer/mnt/session/libs/libOpenGL.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libOpenGL.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libEGL_nvidia.so.0 to /var/lib/apptainer/mnt/session/libs/libEGL_nvidia.so.0
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libEGL_nvidia.so.0
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libnvidia-glcore.so.550.120 to /var/lib/apptainer/mnt/session/libs/libnvidia-glcore.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libnvidia-glcore.so.550.120
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /lib64/libGLdispatch.so to /var/lib/apptainer/mnt/session/libs/libGLdispatch.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/libs/libGLdispatch.so
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /var/lib/apptainer/mnt/session/libs to /var/lib/apptainer/mnt/session/final/.singularity.d/libs
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/.singularity.d/libs
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /bin/nvidia-persistenced to /var/lib/apptainer/mnt/session/final/usr/bin/nvidia-persistenced
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/usr/bin/nvidia-persistenced
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /bin/nvidia-cuda-mps-control to /var/lib/apptainer/mnt/session/final/usr/bin/nvidia-cuda-mps-control
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/usr/bin/nvidia-cuda-mps-control
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /bin/nvidia-cuda-mps-server to /var/lib/apptainer/mnt/session/final/usr/bin/nvidia-cuda-mps-server
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/usr/bin/nvidia-cuda-mps-server
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /bin/nvidia-smi to /var/lib/apptainer/mnt/session/final/usr/bin/nvidia-smi
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/usr/bin/nvidia-smi
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /bin/nvidia-debugdump to /var/lib/apptainer/mnt/session/final/usr/bin/nvidia-debugdump
DEBUG   [U=0,P=1355208]    mountGeneric()                Remounting /var/lib/apptainer/mnt/session/final/usr/bin/nvidia-debugdump
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /var/lib/apptainer/mnt/session/etc/resolv.conf to /var/lib/apptainer/mnt/session/final/etc/resolv.conf
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /var/lib/apptainer/mnt/session/etc/passwd to /var/lib/apptainer/mnt/session/final/etc/passwd
DEBUG   [U=0,P=1355208]    mountGeneric()                Mounting /var/lib/apptainer/mnt/session/etc/group to /var/lib/apptainer/mnt/session/final/etc/group
VERBOSE [U=0,P=1355208]    addCwdMount()                 /root found within container
DEBUG   [U=0,P=1355208]    create()                      Chroot into /var/lib/apptainer/mnt/session/final
DEBUG   [U=0,P=1355231]    Chroot()                      Hold reference to host / directory
DEBUG   [U=0,P=1355231]    Chroot()                      Called pivot_root on /var/lib/apptainer/mnt/session/final
DEBUG   [U=0,P=1355231]    Chroot()                      Change current directory to host / directory
DEBUG   [U=0,P=1355231]    Chroot()                      Apply slave mount propagation for host / directory
DEBUG   [U=0,P=1355231]    Chroot()                      Called unmount(/, syscall.MNT_DETACH)
DEBUG   [U=0,P=1355231]    Chroot()                      Changing directory to / to avoid getpwd issues
DEBUG   [U=0,P=1355208]    create()                      Chdir into / to avoid errors
VERBOSE [U=0,P=1355230]    wait_child()                  rpc server exited with status 0
DEBUG   [U=0,P=1355230]    init()                        Set container privileges
DEBUG   [U=0,P=1355230]    apply_privileges()            Effective capabilities:   0x000001ffffffffff
DEBUG   [U=0,P=1355230]    apply_privileges()            Permitted capabilities:   0x000001ffffffffff
DEBUG   [U=0,P=1355230]    apply_privileges()            Bounding capabilities:    0x000001ffffffffff
DEBUG   [U=0,P=1355230]    apply_privileges()            Inheritable capabilities: 0x000001ffffffffff
DEBUG   [U=0,P=1355230]    apply_privileges()            Ambient capabilities:     0x000001ffffffffff
DEBUG   [U=0,P=1355230]    apply_privileges()            Set user ID to 0
DEBUG   [U=0,P=1355230]    set_parent_death_signal()     Set parent death signal to 9
DEBUG   [U=0,P=1355230]    func1()                       executablePath is /usr/libexec/apptainer/bin/starter
DEBUG   [U=0,P=1355230]    func1()                       executablePath does not exist, assuming default prefix
DEBUG   [U=0,P=1355230]    startup()                     apptainer runtime engine selected
VERBOSE [U=0,P=1355230]    startup()                     Execute stage 2
DEBUG   [U=0,P=1355230]    StageTwo()                    Entering stage 2
DEBUG   [U=0,P=1355230]    StartProcess()                Setting umask in container to 0002
DEBUG   [U=0,P=1355230]    func4()                       Not exporting "BASH_FUNC__module_raw%%" to container environment: invalid key
DEBUG   [U=0,P=1355230]    func4()                       Not exporting "BASH_FUNC_ml%%" to container environment: invalid key
DEBUG   [U=0,P=1355230]    func4()                       Not exporting "BASH_FUNC_module%%" to container environment: invalid key
DEBUG   [U=0,P=1355230]    func4()                       Not exporting "BASH_FUNC_scl%%" to container environment: invalid key
DEBUG   [U=0,P=1355230]    func4()                       Not exporting "BASH_FUNC_which%%" to container environment: invalid key
DEBUG   [U=0,P=1355230]    sylogBuiltin()                Sourcing /.singularity.d/env/01-base.sh
DEBUG   [U=0,P=1355230]    sylogBuiltin()                Sourcing /.singularity.d/env/10-docker2singularity.sh
DEBUG   [U=0,P=1355230]    sylogBuiltin()                Sourcing /.singularity.d/env/90-environment.sh
DEBUG   [U=0,P=1355230]    sylogBuiltin()                Sourcing /.singularity.d/env/94-appsbase.sh
DEBUG   [U=0,P=1355230]    sylogBuiltin()                Sourcing /.singularity.d/env/95-apps.sh
DEBUG   [U=0,P=1355230]    sylogBuiltin()                Sourcing /.singularity.d/env/99-base.sh
DEBUG   [U=0,P=1355230]    sylogBuiltin()                Sourcing /.singularity.d/env/99-runtimevars.sh
DEBUG   [U=0,P=1355230]    sylogBuiltin()                Running action command exec
DEBUG   [U=0,P=1355208]    PostStartProcess()            Post start process
Python 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
```