Reinforce

value-based methods (q-learning and family) learn how good actions are and act by argmax. policy-gradient methods skip the middleman: parameterise the policy itself, \(\pi_\theta(a \mid s)\), and do gradient ascent on expected return. 𐃏 the entire family — reinforce, actor-critic, trpo, ppo, and by extension rlhf — rests on one identity, the policy gradient theorem, whose derivation is three lines of calculus and one very good idea. the standard reference is sutton & barto, free at http://incompleteideas.net/book/the-book-2nd.html.