Actor-Critic Methods in Reinforcement Learning

Actor-Critic is an improvement of the Reinforce method. In Reinforce, the bottleneck is the reward-to-go \(G_{t}=\sum_{k=0}^{\infin} \gamma^{k}r_{t+k}\) in that

1. We must experience a full episode to sample one trajectory to obtain \(G_{t}\), meaning that we must update offline after the episode ends.
2. Since the trajectory is sampled in a stochastic environment and with a stochastic policy, \(G_{t}\) has a large variance, especially in a long episode with significant stochasticity.

The benefit is that Reinforce method is unbiased. To lower the variance, we have the variance reduction trick by adding an extra unbiased term. This gives birth to the A2C method.

The actor-critic can be regarded as Reinforce plus a learnable value function in place of the Monte-Carlo reward.

• The Actor is the policy \(\pi_{\theta}(a|s)\) that outputs an action (or a distribution of actions from which we sample an action) \(a_{t}\)
• The action \(a_{t}\) and the state \(s_{t}\) is processed by the environment and produces a reward \(r_{t}\) and the updated state \(s_{t+1}\).

• The Critic \(V_{\omega}(s)\) or \(Q_{\omega}(s,a)\) judges the policy. Critic usually produces a temporal difference error which tells how accurately the critic predicts the value, by compute the difference between new estimation and old estimation.

  \[ \delta_{t} = \underbrace{r_{t} + \gamma V_{\omega}(s_{t+1})}_{\text{new estimation of value}} - \underbrace{V_{\omega}(s_{t})}_{\text{old estimation of value}} \]

• Most importantly, after judgement, both actor and critic will be updated by the temporal difference error.

  \[\begin{aligned} \theta &\gets \theta + \alpha \cdot \delta_{t} \cdot \nabla \log \pi_{\theta} (a_{t}|s_{t}) \\ \omega &\gets \omega + \beta \cdot \delta_{t} \cdot \nabla V_{\omega} (s_{t}) \end{aligned} \]

  The \(V_{\omega}(s_{t})\) can also be replaced by \(Q_{\omega}(s_{t}, a_{t})\).