Repository: oumi-ai/oumi
File: src/oumi/core/configs/params/evaluation_params.py (Python)

Code before:
    evaluation_backend: str = MISSING
    """The evaluation backend to use for the current task."""

    task_name: Optional[str] = None
    """The task to evaluate or the custom evaluation function to use.

    For LM Harness evaluations (when the evaluation_backend is set to
    EvaluationBackend.LM_HARNESS), the `task_name` corresponds to a predefined task
    to evaluate on (e.g. "mmlu"). A list of all supported tasks by the LM Harness
    backend can be found by running: `lm-eval --tasks list`.

    For custom evaluations (when evaluation_backend is set to EvaluationBackend.CUSTOM),
    the `task_name` should be the registry key for the custom evaluation function to be
    used. Users can register new evaluation functions using the decorator
    `@register_evaluation_function`.
    """

    num_samples: Optional[int] = None
    """Number of samples/examples to evaluate from this dataset.

    Mostly for debugging, in order to reduce the runtime.
    If not set (None): the entire dataset is evaluated.
    If set, this must be a positive integer.
    """

    eval_kwargs: dict[str, Any] = field(default_factory=dict)
    """Additional keyword arguments to pass to the evaluation function.

    This allows for passing any evaluation-specific parameters that are not
    covered by other fields in TaskParams classes.
    """

    def get_evaluation_backend(self) -> EvaluationBackend:
        """Returns the evaluation backend as an Enum."""
        if not self.evaluation_backend:
            raise ValueError(
                "Missing `evaluation_backend`. When running evaluations, it is "
                "necessary to specify the evaluation backend to use for EACH task. "
                "The available backends can be found in the following enum: "
                "`oumi.core.configs.params.evaluation_params.EvaluationBackend`. "
                f"Current options: {EvaluationTaskParams.list_evaluation_backends()}."
            )
        elif self.evaluation_backend == EvaluationBackend.LM_HARNESS.value:
            return EvaluationBackend.LM_HARNESS
        elif self.evaluation_backend == EvaluationBackend.ALPACA_EVAL.value:
            return EvaluationBackend.ALPACA_EVAL
        elif self.evaluation_backend == EvaluationBackend.CUSTOM.value:
            return EvaluationBackend.CUSTOM
        else:
            raise ValueError(f"Unknown evaluation backend: {self.evaluation_backend}")


Review comment (suggestion):
Consider adding a check in __post_init__  that log_samples is not re-defined in `eval_kwargs` as well (it can lead to confusing  and potentially error-prone behavior).  Example: https://github.com/oumi-ai/oumi/blob/0748703fceacc78e01e04f78d657058a0c6b6fa9/src/oumi/core/configs/params/data_params.py#L169

Code after:
    evaluation_backend: str = MISSING
    """The evaluation backend to use for the current task."""

    task_name: Optional[str] = None
    """The task to evaluate or the custom evaluation function to use.

    For LM Harness evaluations (when the evaluation_backend is set to
    EvaluationBackend.LM_HARNESS), the `task_name` corresponds to a predefined task
    to evaluate on (e.g. "mmlu"). A list of all supported tasks by the LM Harness
    backend can be found by running: `lm-eval --tasks list`.

    For custom evaluations (when evaluation_backend is set to EvaluationBackend.CUSTOM),
    the `task_name` should be the registry key for the custom evaluation function to be
    used. Users can register new evaluation functions using the decorator
    `@register_evaluation_function`.
    """

    num_samples: Optional[int] = None
    """Number of samples/examples to evaluate from this dataset.

    Mostly for debugging, in order to reduce the runtime.
    If not set (None): the entire dataset is evaluated.
    If set, this must be a positive integer.
    """

    log_samples: Optional[bool] = False
    """Whether to log the samples used for evaluation.

    If not set (False): the model samples used for evaluation will not be logged.
    If set to True: the model samples generated during inference and used for
    evaluation will be logged in `backend_config.json`. The backend may also log
    other intermediate results related to inference.
    """

    eval_kwargs: dict[str, Any] = field(default_factory=dict)
    """Additional keyword arguments to pass to the evaluation function.

    This allows for passing any evaluation-specific parameters that are not
    covered by other fields in TaskParams classes.
    """

    def get_evaluation_backend(self) -> EvaluationBackend:
        """Returns the evaluation backend as an Enum."""
        if not self.evaluation_backend:
            raise ValueError(
                "Missing `evaluation_backend`. When running evaluations, it is "
                "necessary to specify the evaluation backend to use for EACH task. "
                "The available backends can be found in the following enum: "
                "`oumi.core.configs.params.evaluation_params.EvaluationBackend`. "
                f"Current options: {EvaluationTaskParams.list_evaluation_backends()}."
            )