Add MegatronNevaDeployable for serving multimodal models on Triton server
State: closed | Pull request: yes
Labels: name: stale

name: Run CICD

# What does this PR do ?

Adds a `MegatronNevaDeployable` class implementing `ITritonDeployable` enabling deployment of multimodal models like LLaVA/Neva to Triton inference server directly from .nemo checkpoints.

Currently this has only been tested with a LLaVA 1.5 model created with the instructions [here](https://docs.nvidia.com/nemo-framework/user-guide/latest/multimodalmodels/multimodallanguagemodel/neva/dataprep.html).

**Collection**: [Note which collection this PR will affect]
deploy/

# Changelog 
- Add `MegatronNevaDeployable` class
- Add `NemoQueryMultimodalPyTorch` class
- Add `test_neva_deploy.py` script for testing deployment
- Add `query_inframework.py` script for easy use of `NemoQueryMultimodalPyTorch`

# Usage
See `test_neva_deploy.py` for full example on how to use `MegatronNevaDeployable`.

```python
    megatron_deployable = MegatronNevaDeployable(args.nemo_checkpoint, args.num_gpus)
...
    nm = DeployPyTriton(
        model=megatron_deployable,
        triton_model_name=model_name,
        triton_model_version=1,
        max_batch_size=8,
        port=8000,
        address="0.0.0.0",
        streaming=False,
    )
    nm.deploy()
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.