NeMo Megatron dataset helper makefile compiles output to write protected container folder (Singularity)
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

The [C++ dataset helper makefile](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/data/language_modeling/megatron/Makefile) of Megatron in NeMo attempts to write its output to `/usr/bin/ld`, causing training to crash when using singularity containers build off of your NVIDIA NGC NeMo containers on HPC clusters. 

```
0: [NeMo I 2023-01-18 08:00:59 gpt_dataset:488]  > elasped time to build and save doc-idx mapping (seconds): 40.719332
0: make: Entering directory '/opt/conda/lib/python3.8/site-packages/nemo/collections/nlp/data/language_modeling/megatron'
0: g++ -O3 -Wall -shared -std=c++11 -fPIC -fdiagnostics-color -I/opt/conda/include/python3.8 -I/opt/conda/lib/python3.8/site-packages/pybind11/include helpers.cpp -o helpers.cpython-38-x86_64-linux-gnu.so
0: /usr/bin/ld: cannot open output file helpers.cpython-38-x86_64-linux-gnu.so: Read-only file system
0: collect2: error: ld returned 1 exit status
0: make: *** [Makefile:23: helpers.cpython-38-x86_64-linux-gnu.so] Error 1
0: make: Leaving directory '/opt/conda/lib/python3.8/site-packages/nemo/collections/nlp/data/language_modeling/megatron'
0: [NeMo E 2023-01-18 08:01:30 dataset_utils:83] Making C++ dataset helpers module failed, exiting.
0: [NeMo W 2023-01-18 08:01:31 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py:431: UserWarning: Error handling mechanism for deadlock detection is uninitialized. Skipping check.
0:       rank_zero_warn("Error handling mechanism for deadlock detection is uninitialized. Skipping check.")
```

**Steps/Code to reproduce bug**

Build a singularity container for use in HPC. This is our definition file `nemo.def`, and we build it locally (outside of HPC environment) via `sudo singularity build nemo2209.sif nemo.def`:

```
From: nvcr.io/nvidia/nemo:22.09

%environment
    export LC_ALL=C
```
We transfer the image `nemo2209.sif` to HPC, and follow the [NeMo GPT training docs](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/gpt/gpt_training.html).

See this other issue for [sbatch config and launch script](https://github.com/NVIDIA/NeMo/issues/5819) (changing `--nodes=2` to `--nodes=1`). 

**Expected behavior**

Most users will probably use NeMo Megatron on HPC, where they don't have `sudo` rights and need to use Singularity instead of Docker. It would be nice if you would test that your documentation examples are launchable with Singularity containers on systems where you do not have root/sudo. A container that is already built should be able to launch training without errors, and without building/compiling extra stuff that needs to be written to write protected folders. 

Regular NVIDIA Pytorch containers work out of the box when converted to Singularity containers and used with Megatron-LM.

**Environment overview (please complete the following information)**
HPC cluster, Slurm.

**Environment details**
NGC Nemo containers 22.08 and 22.09.

**Additional context**

A100 GPUs.