Repository: ray-project/ray
File: rllib/algorithms/algorithm.py (Python)

Code before:

    @override(Trainable)
    def step(self) -> ResultDict:
        """Implements the main `Algorithm.train()` logic.

        Takes n attempts to perform a single training step. Thereby
        catches RayErrors resulting from worker failures. After n attempts,
        fails gracefully.

        Override this method in your Algorithm sub-classes if you would like to
        handle worker failures yourself.
        Otherwise, override only `training_step()` to implement the core
        algorithm logic.

        Returns:
            The results dict with stats/infos on sampling, training,
            and - if required - evaluation.
        """
        # Ray metrics
        with TimerAndPrometheusLogger(self._metrics_step_time):
            # Do we have to run `self.evaluate()` this iteration?
            # `self.iteration` gets incremented after this function returns,
            # meaning that e.g. the first time this function is called,
            # self.iteration will be 0.
            evaluate_this_iter = (
                self.config.evaluation_interval is not None
                and (self.iteration + 1) % self.config.evaluation_interval == 0
            )

            evaluate_offline_this_iter = (
                self.config.offline_evaluation_interval
                and (self.iteration + 1) % self.config.offline_evaluation_interval == 0
            )

            # Results dict for training (and if appolicable: evaluation).
            eval_results: ResultDict = {}

            # Parallel eval + training: Kick off evaluation-loop and parallel train() call.
            if evaluate_this_iter and (
                self.config.evaluation_parallel_to_training
                or self.config.offline_evaluation_parallel_to_training
            ):
                (
                    train_results,
                    eval_results,
                    train_iter_ctx,
                ) = self._run_one_training_iteration_and_evaluation_in_parallel()

            # - No evaluation necessary, just run the next training iteration.
            # - We have to evaluate in this training iteration, but no parallelism ->
            #   evaluate after the training iteration is entirely done.

Review comment (bug):
Has nothing to do with this PR, just caught my eye because I spotted evaluate_this_iter being None while debugging.

Code after:

    @override(Trainable)
    def step(self) -> ResultDict:
        """Implements the main `Algorithm.train()` logic.

        Takes n attempts to perform a single training step. Thereby
        catches RayErrors resulting from worker failures. After n attempts,
        fails gracefully.

        Override this method in your Algorithm sub-classes if you would like to
        handle worker failures yourself.
        Otherwise, override only `training_step()` to implement the core
        algorithm logic.

        Returns:
            The results dict with stats/infos on sampling, training,
            and - if required - evaluation.
        """
        # Ray metrics
        with TimerAndPrometheusLogger(self._metrics_step_time):
            # Do we have to run `self.evaluate()` this iteration?
            # `self.iteration` gets incremented after this function returns,
            # meaning that e.g. the first time this function is called,
            # self.iteration will be 0.
            evaluate_this_iter = bool(
                self.config.evaluation_interval
                and (self.iteration + 1) % self.config.evaluation_interval == 0
            )

            evaluate_offline_this_iter = bool(
                self.config.offline_evaluation_interval
                and (self.iteration + 1) % self.config.offline_evaluation_interval == 0
            )

            # Results dict for training (and if appolicable: evaluation).
            eval_results: ResultDict = {}

            # Parallel eval + training: Kick off evaluation-loop and parallel train() call.
            if evaluate_this_iter and (
                self.config.evaluation_parallel_to_training
                or self.config.offline_evaluation_parallel_to_training
            ):
                (
                    train_results,
                    eval_results,
                    train_iter_ctx,
                ) = self._run_one_training_iteration_and_evaluation_in_parallel()

            # - No evaluation necessary, just run the next training iteration.
            # - We have to evaluate in this training iteration, but no parallelism ->
            #   evaluate after the training iteration is entirely done.