eval_trainer.predict() gives AttributeError: 'PipelineMixedPrecisionPlugin' object has no attribute '_desired_input_dtype' error
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I encountered the following error when calling trainer_eval.predict() to test the fine-tuned LLM Model:

```
AttributeError                            Traceback (most recent call last)
Cell In[33], line 1
----> 1 response = trainer_eval.predict(model_eval, request_dl)
      2 for batch in response:
      3     for s in batch['sentences']:

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:864, in Trainer.predict(self, model, dataloaders, datamodule, return_predictions, ckpt_path)
    862 self.state.status = TrainerStatus.RUNNING
    863 self.predicting = True
--> 864 return call._call_and_handle_interrupt(
    865     self, self._predict_impl, model, dataloaders, datamodule, return_predictions, ckpt_path
    866 )

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py:44, in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
     42     if trainer.strategy.launcher is not None:
     43         return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
---> 44     return trainer_fn(*args, **kwargs)
     46 except _TunerExitException:
     47     _call_teardown_hook(trainer)

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:903, in Trainer._predict_impl(self, model, dataloaders, datamodule, return_predictions, ckpt_path)
    899 assert self.state.fn is not None
    900 ckpt_path = self._checkpoint_connector._select_ckpt_path(
    901     self.state.fn, ckpt_path, model_provided=model_provided, model_connected=self.lightning_module is not None
    902 )
--> 903 results = self._run(model, ckpt_path=ckpt_path)
    905 assert self.state.stopped
    906 self.predicting = False

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:987, in Trainer._run(self, model, ckpt_path)
    982 self._signal_connector.register_signal_handlers()
    984 # ----------------------------
    985 # RUN THE TRAINER
    986 # ----------------------------
--> 987 results = self._run_stage()
    989 # ----------------------------
    990 # POST-Training CLEAN UP
    991 # ----------------------------
    992 log.debug(f"{self.__class__.__name__}: trainer tearing down")

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:1028, in Trainer._run_stage(self)
   1026     return self._evaluation_loop.run()
   1027 if self.predicting:
-> 1028     return self.predict_loop.run()
   1029 if self.training:
   1030     with isolate_rng():

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/utilities.py:182, in _no_grad_context.<locals>._decorator(self, *args, **kwargs)
    180     context_manager = torch.no_grad
    181 with context_manager():
--> 182     return loop_run(self, *args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/prediction_loop.py:124, in _PredictionLoop.run(self)
    122     self.batch_progress.is_last_batch = data_fetcher.done
    123     # run step hooks
--> 124     self._predict_step(batch, batch_idx, dataloader_idx, dataloader_iter)
    125 except StopIteration:
    126     # this needs to wrap the `*_step` call too (not just `next`) for `dataloader_iter` support
    127     break

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/prediction_loop.py:229, in _PredictionLoop._predict_step(self, batch, batch_idx, dataloader_idx, dataloader_iter)
    226 assert data_fetcher is not None
    228 if not (using_dataloader_iter := isinstance(data_fetcher, _DataLoaderIterDataFetcher)):
--> 229     batch = trainer.precision_plugin.convert_input(batch)
    230     batch = trainer.lightning_module._on_before_batch_transfer(batch, dataloader_idx=dataloader_idx)
    231     batch = call._call_strategy_hook(trainer, "batch_to_device", batch, dataloader_idx=dataloader_idx)

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/plugins/precision/fsdp.py:131, in FSDPPrecision.convert_input(self, data)
    129 @override
    130 def convert_input(self, data: Any) -> Any:
--> 131     return apply_to_collection(data, function=_convert_fp_tensor, dtype=Tensor, dst_type=self._desired_input_dtype)

AttributeError: 'PipelineMixedPrecisionPlugin' object has no attribute '_desired_input_dtype'

```
**Steps/Code to reproduce bug**

I'm following the steps and guides outlined in the [LoRA.ipynb](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/lora.ipynb) notebook example of finetuning an LLM (megatron_gpt). However, I encountered the following error when testing/evaluating the finetuned model by executing the following cell:

```python
response = trainer_eval.predict(model_eval, request_dl)
for batch in response:
    for s in batch['sentences']:
        print(f"{s}\n\n")
```

**Expected behavior**

getting output prediction from the fine-tuned model.

I also have a question regarding how to merge the adapter with the base model and save them in a single .nemo file for deployment. Could you please assist?

**Environment overview (please complete the following information)**

pulling the nemofw-training:24.03 image container
on A100 GPU




This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.