Deep RL Course documentation

Join the Hugging Face community

to get started

## Reducing variance with Actor-Critic methods

The solution to reducing the variance of the Reinforce algorithm and training our agent faster and better is to use a combination of Policy-Based and Value-Based methods: the Actor-Critic method.

To understand the Actor-Critic, imagine you’re playing a video game. You can play with a friend that will provide you with some feedback. You’re the Actor and your friend is the Critic.

You don’t know how to play at the beginning, so you try some actions randomly. The Critic observes your action and provides feedback.

Learning from this feedback, you’ll update your policy and be better at playing that game.

On the other hand, your friend (Critic) will also update their way to provide feedback so it can be better next time.

This is the idea behind Actor-Critic. We learn two function approximations:

• A policy that controls how our agent acts:$\pi_{\theta}(s)$

• A value function to assist the policy update by measuring how good the action taken is:$\hat{q}_{w}(s,a)$

## The Actor-Critic Process

Now that we have seen the Actor Critic’s big picture, let’s dive deeper to understand how the Actor and Critic improve together during the training.

As we saw, with Actor-Critic methods, there are two function approximations (two neural networks):

• Actor, a policy function parameterized by theta:$\pi_{\theta}(s)$
• Critic, a value function parameterized by w:$\hat{q}_{w}(s,a)$

Let’s see the training process to understand how the Actor and Critic are optimized:

• At each timestep, t, we get the current state$S_t$ from the environment and pass it as input through our Actor and Critic.

• Our Policy takes the state and outputs an action $A_t$.

• The Critic takes that action also as input and, using$S_t$ and$A_t$, computes the value of taking that action at that state: the Q-value.
• The action$A_t$ performed in the environment outputs a new state$S_{t+1}$ and a reward$R_{t+1}$ .
• The Actor updates its policy parameters using the Q value.
• Thanks to its updated parameters, the Actor produces the next action to take at$A_{t+1}$ given the new state$S_{t+1}$.

• The Critic then updates its value parameters.

The problem with implementing this advantage function is that it requires two value functions — $Q(s,a)$ and $V(s)$. Fortunately, we can use the TD error as a good estimator of the advantage function.