NeMo 2.0 In-framework deployment support
State: closed | Pull request: yes

# What does this PR do ?

Adds inframework deployment support and log probability output from trt-llm


Left some minor comments, overall LGTM

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.deploy.nlp.megatronllm_deployable
nemo/deploy/nlp/megatronllm_deployable.py:266:0: C0301: Line too long (144/119) (line-too-long)
nemo/deploy/nlp/megatronllm_deployable.py:270:0: C0301: Line too long (248/119) (line-too-long)
nemo/deploy/nlp/megatronllm_deployable.py:297:0: C0301: Line too long (120/119) (line-too-long)
nemo/deploy/nlp/megatronllm_deployable.py:302:0: C0301: Line too long (122/119) (line-too-long)
nemo/deploy/nlp/megatronllm_deployable.py:412:0: C0301: Line too long (158/119) (line-too-long)
nemo/deploy/nlp/megatronllm_deployable.py:52:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/nlp/megatronllm_deployable.py:102:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/nlp/megatronllm_deployable.py:106:0: C0115: Missing class docstring (missing-class-docstring)
nemo/deploy/nlp/megatronllm_deployable.py:109:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/nlp/megatronllm_deployable.py:407:4: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.deploy.nlp.query_llm
nemo/deploy/nlp/query_llm.py:29:0: C0115: Missing class docstring (missing-class-docstring)
nemo/deploy/nlp/query_llm.py:16:0: W0611: Unused abstractmethod imported from abc (unused-import)
************* Module nemo.deploy.utils
nemo/deploy/utils.py:30:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/utils.py:63:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/utils.py:75:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/utils.py:80:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/utils.py:87:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/utils.py:92:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.export.trt_llm.tensorrt_llm_run
nemo/export/trt_llm/tensorrt_llm_run.py:506:0: C0301: Line too long (125/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:510:0: C0301: Line too long (136/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:514:0: C0301: Line too long (123/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:557:0: C0301: Line too long (181/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:841:0: C0301: Line too long (153/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:524:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_run.py:533:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_run.py:591:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_run.py:33:0: W0611: Unused Mapping imported from tensorrt_llm.mapping (unused-import)
************* Module scripts.deploy.nlp.deploy_inframework_triton
scripts/deploy/nlp/deploy_inframework_triton.py:31:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/deploy_inframework_triton.py:52:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/deploy_inframework_triton.py:59:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module scripts.deploy.nlp.query_inframework
scripts/deploy/nlp/query_inframework.py:21:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/query_inframework.py:42:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/query_inframework.py:65:0: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.78/10
```

---

Thank you for improving NeMo's documentation!

Closing this since there is a newer version of this PR: https://github.com/NVIDIA/NeMo/pull/11523