Adding a Custom Task

Lighteval provides a flexible framework for creating custom evaluation tasks. This guide explains how to create and integrate new tasks into the evaluation system.

Task Categories

Before creating a custom task, consider which category it belongs to:

Core Evaluations

Core evaluations are evaluations that only require standard logic in their metrics and processing, and that we will add to our test suite to ensure non-regression through time. They already see high usage in the community.

Extended Evaluations

Extended evaluations are evaluations that require custom logic in their metrics (complex normalization, an LLM as a judge, etc.), that we added to facilitate the life of users. They already see high usage in the community.

Community Evaluations

Community evaluations are submissions by the community of new tasks.

A popular community evaluation can move to become an extended or core evaluation over time.

You can find examples of custom tasks in the community_tasks directory.

Step-by-Step Creation of a Custom Task

To contribute your custom task to the Lighteval repository, you would first need to install the required dev dependencies by running pip install -e .[dev] and then run pre-commit install to install the pre-commit hooks.

Step 1: Create the Task File

First, create a Python file under the community_tasks directory.

Step 2: Define the Prompt Function

You need to define a prompt function that will convert a line from your dataset to a document to be used for evaluation.

from lighteval.tasks.requests import Doc

# Define as many as you need for your different tasks
def prompt_fn(line: dict, task_name: str):
    """Defines how to go from a dataset line to a doc object.
    Follow examples in src/lighteval/tasks/default_prompts.py, or get more info
    about what this function should do in the README.
    """
    return Doc(
        task_name=task_name,
        query=line["question"],
        choices=[f" {c}" for c in line["choices"]],
        gold_index=line["gold"],
    )

Step 3: Choose or Create Metrics

You can either use an existing metric (defined in lighteval.metrics.metrics.Metrics) or create a custom one.

Using Existing Metrics

from lighteval.metrics import Metrics

# Use an existing metric
metric = Metrics.ACCURACY

Creating Custom Metrics

from lighteval.metrics.utils.metric_utils import SampleLevelMetric
import numpy as np

custom_metric = SampleLevelMetric(
    metric_name="my_custom_metric_name",
    higher_is_better=True,
    category="accuracy",
    sample_level_fn=lambda x: x,  # How to compute score for one sample
    corpus_level_fn=np.mean,  # How to aggregate the sample metrics
)

Step 4: Define Your Task

You can define a task with or without subsets using LightevalTaskConfig.

Simple Task (No Subsets)

from lighteval.tasks.lighteval_task import LightevalTaskConfig

# This is how you create a simple task (like HellaSwag) which has one single subset
# attached to it, and one evaluation possible.
task = LightevalTaskConfig(
    name="myothertask",
    prompt_function=prompt_fn,  # Must be defined in the file or imported
    suite=["community"],
    hf_repo="your_dataset_repo_on_hf",
    hf_subset="default",
    hf_avail_splits=["train", "test"],
    evaluation_splits=["test"],
    few_shots_split="train",
    few_shots_select="random_sampling_from_train",
    metrics=[metric],  # Select your metric in Metrics
    generation_size=256,
    stop_sequence=["\n", "Question:"],
)

Task with Multiple Subsets

If you want to create a task with multiple subsets, add them to the SAMPLE_SUBSETS list and create a task for each subset.

SAMPLE_SUBSETS = ["subset1", "subset2", "subset3"]  # List of all the subsets to use for this eval

class CustomSubsetTask(LightevalTaskConfig):
    def __init__(
        self,
        name,
        hf_subset,
    ):
        super().__init__(
            name=name,
            hf_subset=hf_subset,
            prompt_function=prompt_fn,  # Must be defined in the file or imported
            hf_repo="your_dataset_name",
            metrics=[custom_metric],  # Select your metric in Metrics or use your custom_metric
            hf_avail_splits=["train", "test"],
            evaluation_splits=["test"],
            few_shots_split="train",
            few_shots_select="random_sampling_from_train",
            suite=["community"],
            generation_size=256,
            stop_sequence=["\n", "Question:"],
        )

SUBSET_TASKS = [CustomSubsetTask(name=f"mytask:{subset}", hf_subset=subset) for subset in SAMPLE_SUBSETS]

Step 5: Add Tasks to the Table

Then you need to add your task to the TASKS_TABLE list.

# STORE YOUR EVALS

# Tasks with subsets:
TASKS_TABLE = SUBSET_TASKS

# Tasks without subsets:
# TASKS_TABLE = [task]

Step 6: Creating a requirement file

If your task has requirements, you need to create a requirement.txt file with only the required dependencies so that anyone can run your task.

Running Your Custom Task

Once your file is created, you can run the evaluation with the following command:

lighteval accelerate \
    "model_name=HuggingFaceH4/zephyr-7b-beta" \
    "community|{custom_task}|{fewshots}" \
    --custom-tasks {path_to_your_custom_task_file}

Example Usage

# Run a custom task with zero-shot evaluation
lighteval accelerate \
    "model_name=openai-community/gpt2" \
    "community|myothertask|0" \
    --custom-tasks community_tasks/my_custom_task.py

# Run a custom task with few-shot evaluation
lighteval accelerate \
    "model_name=openai-community/gpt2" \
    "community|myothertask|3" \
    --custom-tasks community_tasks/my_custom_task.py

< > Update on GitHub