Integrate lm-eval-harness for evaluations in NeMo
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Integrates [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/main) into NeMo to run evaluations on standard academic benchmarks.
Evaluations can be run by first deploying the model on PyTriton server via deploy method followed by running the evaluate method to do the actual evaluation.

**Collection**: This is an independent module and does not affect any collection.

# Changelog 
- Refactor `deploy` method in `nemo/collections/llm/api.py` by moving utility funcs to `nemo/collections/llm/evaluation/eval_utils.py`. Deploy method takes care of exporting nemo model to trtllm and deploying it on PyTriton server. 
- Deploy method also [starts rest service](https://github.com/NVIDIA/NeMo/pull/10621/files#diff-1afe0432f8c676bf15b5ebbc6628d0f1e9757ea6d3eb73a458c905080e77d50bR365-R369) using uvicorn that runs a FastAPI application which exposes OpenAI end point [/v1/completions](https://github.com/NVIDIA/NeMo/pull/10621/files#diff-5a9ea7d8a543abe1c43aba0c89551c8e661bedbdcbbdf7107750a7ea25870166R107). 
- The FastAPI application is defined in `nemo/deploy/service/rest_model_api.py` which contains the code to interact with the PyTriton Server to perform the actual model inference via `/v1/completions` endpoint.
- [Evaluate method](https://github.com/NVIDIA/NeMo/pull/10621/files#diff-1afe0432f8c676bf15b5ebbc6628d0f1e9757ea6d3eb73a458c905080e77d50bR382) is added to `nemo/collections/llm/api.py` to evaluate nemo model deployed on PyTriton server (via trtllm) using lm-evaluation-harness. 
Evaluate method takes in as input the rest service url "http://rest_service_http_address:rest_service_port/v1" and other evaluation params like eval task, num_fewshot, limit etc., and inference params like temperature, top_p, top_k, max_tokens_to_generate
- Evaluate method instantiates [NeMoFWLMEval class](https://github.com/NVIDIA/NeMo/pull/10621/files#diff-1afe0432f8c676bf15b5ebbc6628d0f1e9757ea6d3eb73a458c905080e77d50bR440) defined in `nemo/collections/llm/evaluation/eval_utils.py` that creates a wrapper to interface eval prompts from lm-eval-harness with the model deployed on PyTriton.
- [NeMoFWLMEval class ](https://github.com/NVIDIA/NeMo/pull/10621/files#diff-90b33cee942c51b274d1ded51e97e270442254275630ba88614867d5bd6296f4R18) defines two types of tasks: [generate_until](https://github.com/NVIDIA/NeMo/pull/10621/files#diff-90b33cee942c51b274d1ded51e97e270442254275630ba88614867d5bd6296f4R120) (for ex: gsm8k) and [loglikelihood](https://github.com/NVIDIA/NeMo/pull/10621/files#diff-90b33cee942c51b274d1ded51e97e270442254275630ba88614867d5bd6296f4R63) tasks (for ex: mmlu). 
- The PR also exposes [gather_context_logits, gather_generation_logits](https://github.com/NVIDIA/NeMo/pull/10621/files#diff-73ace52ae1e5be664174e2ce71267c14118d0476fbd42d8be60842c0311b1334R57-R58) flag to build the trtllm engine with logits. Eval code needs the generation logits to compute the logProb of the actual output token or label, hence its enabled (False by default).

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

[🤖]: Hi @athitten 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11735874189) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

[🤖]: Hi @athitten 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11751799322) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

> Can you please check this PR #11233 and make sure the export and deploy updates don't overlap with it?

@oyilmaz-nvidia there is no overlap between this PR and PR #11233. This PR exposes `output_generation_logits` which is required for evaluation and PR #11233 is exposing logProbs. It should be okay to have both, but we might have some minor conflicts merging the other after one of them is in.

<!-- pylint-output-strict-mode-true -->

beep boop 🤖: 🚨 The following files must be fixed before merge!

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

<!-- pylint-output-strict-mode-true -->

beep boop 🤖: 🚨 The following files must be fixed before merge!

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.llm.api
nemo/collections/llm/api.py:70:0: C0301: Line too long (135/119) (line-too-long)
nemo/collections/llm/api.py:226:0: C0301: Line too long (135/119) (line-too-long)
nemo/collections/llm/api.py:538:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.deploy.nlp.query_llm
nemo/deploy/nlp/query_llm.py:278:0: C0301: Line too long (120/119) (line-too-long)
nemo/deploy/nlp/query_llm.py:29:0: C0115: Missing class docstring (missing-class-docstring)
nemo/deploy/nlp/query_llm.py:16:0: W0611: Unused abstractmethod imported from abc (unused-import)
************* Module nemo.deploy.service.rest_model_api
nemo/deploy/service/rest_model_api.py:91:0: C0301: Line too long (155/119) (line-too-long)
nemo/deploy/service/rest_model_api.py:92:0: C0301: Line too long (152/119) (line-too-long)
nemo/deploy/service/rest_model_api.py:23:0: C0115: Missing class docstring (missing-class-docstring)
nemo/deploy/service/rest_model_api.py:41:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/service/rest_model_api.py:45:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/service/rest_model_api.py:49:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/service/rest_model_api.py:71:0: C0115: Missing class docstring (missing-class-docstring)
nemo/deploy/service/rest_model_api.py:84:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/service/rest_model_api.py:109:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/service/rest_model_api.py:12:0: W0611: Unused Path imported from pathlib (unused-import)
************* Module nemo.export.tensorrt_llm
nemo/export/tensorrt_llm.py:118:0: C0301: Line too long (148/119) (line-too-long)
nemo/export/tensorrt_llm.py:221:0: C0301: Line too long (123/119) (line-too-long)
nemo/export/tensorrt_llm.py:223:0: C0301: Line too long (135/119) (line-too-long)
nemo/export/tensorrt_llm.py:224:0: C0301: Line too long (141/119) (line-too-long)
nemo/export/tensorrt_llm.py:387:0: C0301: Line too long (186/119) (line-too-long)
nemo/export/tensorrt_llm.py:998:0: C0301: Line too long (184/119) (line-too-long)
nemo/export/tensorrt_llm.py:1032:0: C0301: Line too long (131/119) (line-too-long)
nemo/export/tensorrt_llm.py:1036:0: C0301: Line too long (135/119) (line-too-long)
nemo/export/tensorrt_llm.py:1094:0: C0301: Line too long (121/119) (line-too-long)
nemo/export/tensorrt_llm.py:67:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:527:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:817:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:838:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:849:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:854:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:931:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:1140:4: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.export.trt_llm.tensorrt_llm_build
nemo/export/trt_llm/tensorrt_llm_build.py:31:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_build.py:19:0: W0611: Unused Builder imported from tensorrt_llm.builder (unused-import)
nemo/export/trt_llm/tensorrt_llm_build.py:23:0: W0611: Unused add_lora imported from tensorrt_llm.models.modeling_utils (unused-import)
************* Module nemo.export.trt_llm.tensorrt_llm_run
nemo/export/trt_llm/tensorrt_llm_run.py:506:0: C0301: Line too long (125/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:510:0: C0301: Line too long (136/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:514:0: C0301: Line too long (123/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:557:0: C0301: Line too long (181/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:841:0: C0301: Line too long (153/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:524:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_run.py:533:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_run.py:591:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_run.py:33:0: W0611: Unused Mapping imported from tensorrt_llm.mapping (unused-import)
************* Module scripts.deploy.nlp.deploy_triton
scripts/deploy/nlp/deploy_triton.py:394:0: C0301: Line too long (121/119) (line-too-long)
scripts/deploy/nlp/deploy_triton.py:30:0: C0115: Missing class docstring (missing-class-docstring)
scripts/deploy/nlp/deploy_triton.py:49:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/deploy_triton.py:259:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/deploy_triton.py:371:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/deploy_triton.py:378:0: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.65/10
```

---

Thank you for improving NeMo's documentation!

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.llm.api
nemo/collections/llm/api.py:70:0: C0301: Line too long (135/119) (line-too-long)
nemo/collections/llm/api.py:226:0: C0301: Line too long (135/119) (line-too-long)
nemo/collections/llm/api.py:538:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.deploy.nlp.query_llm
nemo/deploy/nlp/query_llm.py:278:0: C0301: Line too long (120/119) (line-too-long)
nemo/deploy/nlp/query_llm.py:29:0: C0115: Missing class docstring (missing-class-docstring)
nemo/deploy/nlp/query_llm.py:16:0: W0611: Unused abstractmethod imported from abc (unused-import)
************* Module nemo.deploy.service.rest_model_api
nemo/deploy/service/rest_model_api.py:91:0: C0301: Line too long (155/119) (line-too-long)
nemo/deploy/service/rest_model_api.py:92:0: C0301: Line too long (152/119) (line-too-long)
nemo/deploy/service/rest_model_api.py:23:0: C0115: Missing class docstring (missing-class-docstring)
nemo/deploy/service/rest_model_api.py:41:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/service/rest_model_api.py:45:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/service/rest_model_api.py:49:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/service/rest_model_api.py:71:0: C0115: Missing class docstring (missing-class-docstring)
nemo/deploy/service/rest_model_api.py:84:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/service/rest_model_api.py:109:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/deploy/service/rest_model_api.py:12:0: W0611: Unused Path imported from pathlib (unused-import)
************* Module nemo.export.tensorrt_llm
nemo/export/tensorrt_llm.py:118:0: C0301: Line too long (148/119) (line-too-long)
nemo/export/tensorrt_llm.py:221:0: C0301: Line too long (123/119) (line-too-long)
nemo/export/tensorrt_llm.py:223:0: C0301: Line too long (135/119) (line-too-long)
nemo/export/tensorrt_llm.py:224:0: C0301: Line too long (141/119) (line-too-long)
nemo/export/tensorrt_llm.py:387:0: C0301: Line too long (186/119) (line-too-long)
nemo/export/tensorrt_llm.py:998:0: C0301: Line too long (184/119) (line-too-long)
nemo/export/tensorrt_llm.py:1032:0: C0301: Line too long (131/119) (line-too-long)
nemo/export/tensorrt_llm.py:1036:0: C0301: Line too long (135/119) (line-too-long)
nemo/export/tensorrt_llm.py:1094:0: C0301: Line too long (121/119) (line-too-long)
nemo/export/tensorrt_llm.py:67:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:527:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:817:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:838:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:849:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:854:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:931:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/tensorrt_llm.py:1140:4: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.export.trt_llm.tensorrt_llm_build
nemo/export/trt_llm/tensorrt_llm_build.py:31:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_build.py:19:0: W0611: Unused Builder imported from tensorrt_llm.builder (unused-import)
nemo/export/trt_llm/tensorrt_llm_build.py:23:0: W0611: Unused add_lora imported from tensorrt_llm.models.modeling_utils (unused-import)
************* Module nemo.export.trt_llm.tensorrt_llm_run
nemo/export/trt_llm/tensorrt_llm_run.py:506:0: C0301: Line too long (125/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:510:0: C0301: Line too long (136/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:514:0: C0301: Line too long (123/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:557:0: C0301: Line too long (181/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:841:0: C0301: Line too long (153/119) (line-too-long)
nemo/export/trt_llm/tensorrt_llm_run.py:524:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_run.py:533:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_run.py:591:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_run.py:33:0: W0611: Unused Mapping imported from tensorrt_llm.mapping (unused-import)
************* Module scripts.deploy.nlp.deploy_triton
scripts/deploy/nlp/deploy_triton.py:394:0: C0301: Line too long (121/119) (line-too-long)
scripts/deploy/nlp/deploy_triton.py:30:0: C0115: Missing class docstring (missing-class-docstring)
scripts/deploy/nlp/deploy_triton.py:49:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/deploy_triton.py:259:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/deploy_triton.py:371:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/deploy/nlp/deploy_triton.py:378:0: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.65/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @athitten 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11851906868) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g