Converting Mistral/Mixtral to Nemo format throws error: Received both `precision=bf16-mixed` and `plugins=<...nlp_overrides.PipelineMixedPrecisionPlugin>`
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Converting Mistral/Mixtral to Nemo format throws the following error:

```
Traceback (most recent call last):
  File "/workspace/NeMo/scripts/checkpoint_converters/convert_mistral_7b_hf_to_nemo.py", line 338, in <module>
    convert(args)
  File "/workspace/NeMo/scripts/checkpoint_converters/convert_mistral_7b_hf_to_nemo.py", line 196, in convert
    trainer = Trainer(plugins=plugins, accelerator='cpu', precision=precision, strategy=NLPDDPStrategy())
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/argparse.py", line 70, in insert_env_defaults
    return fn(self, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 401, in __init__
    self._accelerator_connector = _AcceleratorConnector(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 134, in __init__
    self._check_config_and_set_final_flags(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 271, in _check_config_and_set_final_flags
    raise ValueError(
ValueError: Received both `precision=bf16-mixed` and `plugins=<nemo.collections.nlp.parts.nlp_overrides.PipelineMixedPrecisionPlugin object at 0x7f2658622e30>`. Choose one.
```


**Environment overview (please complete the following information)**

```bash
git clone https://github.com/xingyaoww/nemo-util
git submodule update --init --recursive
# Then Build from the Dockerfile.
cd NeMo; docker build -t xingyaoww/nemo:main .

# And enter the docker
DOCKER_IMG=xingyaoww/nemo:main
WORK_DIR=`pwd`

# read model directory from ENV VAR, if exists add it to the docker run command
if [ -z "$MODEL_DIR" ]; then
    echo "MODEL_DIR is not set. Please set MODEL_DIR to the directory containing the model files."
else
    echo "MODEL_DIR is set to '$MODEL_DIR'"
    EXTRA_ARGS="-v $MODEL_DIR:/models"
fi

docker run \
    --gpus all \
    -e UID=$(id -u) \
    --shm-size=2g \
    --net=host \
    --ulimit memlock=-1 \
    --rm -it \
    $EXTRA_ARGS \
    -v ${PWD}:/workspace \
    -w /workspace \
    $DOCKER_IMG \
    bash -c "useradd --shell /bin/bash -u $UID -o -c '' -m nemo && usermod -aG root nemo && cd /workspace && su nemo -c 'git config --global credential.helper store' && su nemo"
```

**Steps/Code to reproduce bug**

```bash
#!/bin/bash
echo "You supposed to run this script inside the docker container"

MODEL_PATH=/models/Mistral-7B-v0.1/
OUTPUT_NAME=mistral-7b-base.nemo

# Override the installed version of Megatron-LM, but use the cloned version
export PYTHONPATH=$(pwd)/NeMo:$(pwd)/Megatron-LM:$PYTHONPATH

python3 NeMo/scripts/checkpoint_converters/convert_mistral_7b_hf_to_nemo.py \
    --input_name_or_path=$MODEL_PATH \
    --output_path=data/models/nemo/$OUTPUT_NAME \
    --precision=bf16-mixed
```


**Potential fixes**
I comment out the line that adds `PipelineMixedPrecisionPlugin` to `plugins` resolves this issue (check commit [here](PipelineMixedPrecisionPlugin)), not sure if this is the best/correct way to resolve this issue?


Looks like solved by https://github.com/NVIDIA/NeMo/pull/8908