NeMo 2.0 In-framework deployment support
State: closed | Pull request: yes
Labels: name: TTS

name: Run CICD

name: skip-docs

# What does this PR do ?

Adds nemo2 support for inframework deployment

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.deploy.nlp.megatronllm_deployable
nemo/deploy/nlp/megatronllm_deployable.py:266:0: C0301: Line too long (144/119) (line-too-long)
nemo/deploy/nlp/megatronllm_deployable.py:270:0: C0301: Line too long (248/119) (line-too-long)
nemo/deploy/nlp/megatronllm_deployable.py:297:0: C0301: Line too long (120/119) (line-too-long)
nemo/deploy/nlp/megatronllm_deployable.py:302:0: C0301: Line too long (122/119) (line-too-long)
nemo/deploy/nlp/megatronllm_deployable.py:412:0: C0301: Line too long (158/119) (line-too-long)
nemo/deploy/nlp/megatronllm_deployable.py:52:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/nlp/megatronllm_deployable.py:102:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/nlp/megatronllm_deployable.py:106:0: C0115: Missing class docstring (missing-class-docstring)
nemo/deploy/nlp/megatronllm_deployable.py:109:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/nlp/megatronllm_deployable.py:407:4: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.deploy.nlp.query_llm
nemo/deploy/nlp/query_llm.py:302:0: C0301: Line too long (120/119) (line-too-long)
nemo/deploy/nlp/query_llm.py:29:0: C0115: Missing class docstring (missing-class-docstring)
nemo/deploy/nlp/query_llm.py:16:0: W0611: Unused abstractmethod imported from abc (unused-import)
************* Module nemo.deploy.utils
nemo/deploy/utils.py:30:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/utils.py:63:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/utils.py:75:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/utils.py:80:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/utils.py:87:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/utils.py:92:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.lightning.megatron_parallel
nemo/lightning/megatron_parallel.py:238:0: C0301: Line too long (127/119) (line-too-long)
nemo/lightning/megatron_parallel.py:239:0: C0301: Line too long (140/119) (line-too-long)
nemo/lightning/megatron_parallel.py:240:0: C0301: Line too long (130/119) (line-too-long)
nemo/lightning/megatron_parallel.py:546:0: C0301: Line too long (129/119) (line-too-long)
nemo/lightning/megatron_parallel.py:553:0: C0301: Line too long (135/119) (line-too-long)
nemo/lightning/megatron_parallel.py:829:0: C0301: Line too long (137/119) (line-too-long)
nemo/lightning/megatron_parallel.py:1059:0: C0301: Line too long (136/119) (line-too-long)
nemo/lightning/megatron_parallel.py:1632:0: C0301: Line too long (128/119) (line-too-long)
nemo/lightning/megatron_parallel.py:1671:0: C0301: Line too long (146/119) (line-too-long)
nemo/lightning/megatron_parallel.py:64:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:65:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:67:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:102:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:106:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:306:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:330:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:356:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:382:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:518:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:561:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:565:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:619:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:654:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:660:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:666:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:673:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:680:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:714:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:722:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:738:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:765:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:777:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:799:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1325:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1500:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1506:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1512:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1516:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1521:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1526:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1554:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1600:8: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1622:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1695:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1734:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1748:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module scripts.deploy.nlp.deploy_inframework_triton
scripts/deploy/nlp/deploy_inframework_triton.py:31:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/deploy_inframework_triton.py:52:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/deploy_inframework_triton.py:59:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module scripts.deploy.nlp.query_inframework
scripts/deploy/nlp/query_inframework.py:21:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/query_inframework.py:42:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/query_inframework.py:65:0: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.35/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @oyilmaz-nvidia 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12300861492) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g