nvidia-lm-eval 25.11.1
A framework for evaluating language models - packaged by NVIDIA
License: MIT
Requires Python: >=3.9

# NVIDIA NeMo Evaluator

The goal of NVIDIA NeMo Evaluator is to advance and refine state-of-the-art methodologies for model evaluation, and deliver them as modular evaluation packages (evaluation containers and pip wheels) that teams can use as standardized building blocks.

# Quick start guide

NVIDIA NeMo Evaluator provide you with evaluation clients, that are specifically built to evaluate model endpoints using our Standard API.

## Launching an evaluation for an LLM

1. Install the package
    ```
    pip install nvidia-lm-eval
    ```

3. (Optional) Set a token to your API endpoint if it's protected
    ```bash
    export MY_API_KEY="your_api_key_here"
    ```
4. List the available evaluations:
    ```bash
    $ nemo-evaluator ls
    Available tasks:
    * mmlu (in lm-evaluation-harness)
    * ifeval (in lm-evaluation-harness)
    * mmlu_pro (in lm-evaluation-harness)
    * math (in lm-evaluation-harness)
    ...
    ```
5. Run the evaluation of your choice:
   ```bash
   nemo-evaluator run_eval \
       --eval_type mmlu_pro \
       --model_id meta/llama-3.1-70b-instruct \
       --model_url https://integrate.api.nvidia.com/v1/chat/completions \
       --model_type chat \
       --api_key_name MY_API_KEY \
       --output_dir /workspace/results
   ```
6. Gather the results
    ```bash
    cat /workspace/results/results.yml
    ```

# Command-Line Tool

Each package comes pre-installed with a set of command-line tools, designed to simplify the execution of evaluation tasks. Below are the available commands and their usage for the `lm_eval` (`lm-evaluation-harness`):

## Commands

### 1. **List Evaluation Types**

```bash
nemo-evaluator ls
```

Displays the evaluation types available within the harness.

### 2. **Run an evaluation**

The `nemo-evaluator run_eval` command executes the evaluation process. Below are the flags and their descriptions:

### Required flags
* `--eval_type <string>`
The type of evaluation to perform
* `--model_id <string>`
The name or identifier of the model to evaluate.
* `--model_url <url>`
The API endpoint where the model is accessible.
* `--model_type <string>`
The type of the model to evaluate, currently either "chat", "completions", or "vlm".
* `--output_dir <directory>`
The directory to use as the working directory for the evaluation. The results, including the results.yml output file, will be saved here.

### Optional flags
* `--api_key_name <string>`
The name of the environment variable that stores the Bearer token for the API, if authentication is required.
* `--run_config <path>`
Specifies the path to a  YAML file containing the evaluation definition.

### Example

```bash
nemo-evaluator run_eval \
    --eval_type ifeval \
    --model_id my_model \
    --model_type chat \
    --model_url http://localhost:8000 \
    --output_dir ./evaluation_results
```

If the model API requires authentication, set the API key in an environment variable and reference it using the `--api_key_name` flag:

```bash
export MY_API_KEY="your_api_key_here"

nemo-evaluator run_eval \
    --eval_type ifeval \
    --model_id my_model \
    --model_type chat \
    --model_url http://localhost:8000 \
    --api_key_name MY_API_KEY \
    --output_dir ./evaluation_results
```

# Configuring evaluations via YAML

Evaluations in NVIDIA NeMo Evaluator are configured using YAML files that define the parameters and settings required for the evaluation process. These configuration files follow a standard API which ensures consistency across evaluations.

Example of a YAML config:
```yaml
config:
  type: ifeval
  params:
    parallelism: 50
    limit_samples: 20
target:
  api_endpoint:
    model_id: meta/llama-3.1-8b-instruct
    type: chat
    url: https://integrate.api.nvidia.com/v1/chat/completions
    api_key: NVIDIA_API_KEY
```

The priority of overrides is as follows:
1. command line arguments
2. user config (as seen above)
3. task defaults (defined per task type)
4. framework defaults

`--dry_run` option allows you to print the final run configuration and command without executing the evaluation.

### Example:

```bash
nemo-evaluator run_eval \
    --eval_type mmlu_pro_instruct \
    --model_id my_model \
    --model_type chat \
    --model_url http://localhost:8000 \
    --output_dir .evaluation_results \
    --dry_run
```

Output:

```bash
Rendered config:

command: '{% if target.api_endpoint.api_key is not none %}OPENAI_API_KEY=${{target.api_endpoint.api_key}}{%
  endif %} lm-eval --tasks {{config.params.task}}{% if config.params.extra.num_fewshot
  is defined %} --num_fewshot {{ config.params.extra.num_fewshot }}{% endif %} --model
  {% if target.api_endpoint.type == "completions" %}local-completions{% elif target.api_endpoint.type
  == "chat" %}local-chat-completions{% endif %} --model_args "base_url={{target.api_endpoint.url}},model={{target.api_endpoint.model_id}},tokenized_requests=false,{%
  if target.api_endpoint.type == "completions" %}tokenizer={{config.params.extra.tokenizer}}{%
  endif %},num_concurrent={{config.params.parallelism}}{% if config.params.max_new_tokens
  is not none %},max_gen_toks={{ config.params.max_new_tokens }}{% endif %},timeout={{
  config.params.timeout }},max_retries={{ config.params.max_retries }},stream={{ target.api_endpoint.stream
  }}" --log_samples --output_path {{config.output_dir}} --use_cache {{config.output_dir}}/lm_cache
  {% if config.params.limit_samples is not none %}--limit {{config.params.limit_samples}}{%
  endif %} {% if target.api_endpoint.type == "chat" %}--fewshot_as_multiturn --apply_chat_template
  {% endif %} {% if config.params.extra.args is defined %} {{config.params.extra.args}}
  {% endif %} {% if config.params.temperature is not none or config.params.top_p is
  not none %}--gen_kwargs="{% if config.params.temperature is not none %}temperature={{
  config.params.temperature }},{% endif %}{% if config.params.top_p is not none %}top_p={{
  config.params.top_p}}{% endif %}"{% endif %}'
framework_name: lm-evaluation-harness
pkg_name: lm_eval
config:
  output_dir: .evaluation_results
  params:
    limit_samples: null
    max_new_tokens: 1024
    max_retries: 5
    parallelism: 10
    task: mmlu_pro
    temperature: 1.0e-07
    timeout: 30
    top_p: 0.9999999
    extra:
      tokenizer: meta-llama/Llama-3.1-70B-Instruct
      num_fewshot: 0
  supported_endpoint_types:
  - chat
  type: mmlu_pro_instruct
target:
  api_endpoint:
    api_key: null
    model_id: my_model
    stream: false
    type: chat
    url: http://localhost:8000


Rendered command:

 lm-eval --tasks mmlu_pro --num_fewshot 0 --model local-chat-completions --model_args "base_url=http://localhost:8000,model=my_model,tokenized_requests=false,,num_concurrent=10,max_gen_toks=1024,timeout=30,max_retries=5,stream=False" --log_samples --output_path .evaluation_results --use_cache .evaluation_results/lm_cache  --fewshot_as_multiturn --apply_chat_template   --gen_kwargs="temperature=1e-07,top_p=0.9999999"
```

# FAQ

## Deploying a model as an endpoint

NVIDIA NeMo Evaluator utilize a client-server communication architecture to interact with the model. As a prerequisite, the **model must be deployed as an endpoint with a NIM-compatible API**.

Users have the flexibility to deploy their model using their own infrastructure and tooling.

Servers with APIs that conform to the OpenAI/NIM API standard are expected to work seamlessly out of the box.