Policy Gradient Theorem
Policy Gradient Theorem is an important topic in RL as it envisions many popular policy-gradient RL methods, including REINFORCE, Actor-Critic, A2C, PPO, GRPO, etc.
Table of Contents
1. Background
In policy-gradient based methods, we want to maximize the expected reward over trajectories:
\[ J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}}[G_{t}(\tau)] \]
here, the \(G_{t}(\tau)\) represents the discounted cumulative rewards since step \(t\), the rewards come from trajectory \(\tau\).
Then, we wish to directly optimize the parameterized policy \(\theta\) to maximize this objective. Usually, it’s gradient ascent \(\theta \gets \theta + \alpha \nabla_{\theta}J(\theta)\). So the problem becomes: how to estimate \(J(\theta)\)?