Evolution Strategies
Gradient-free policy optimization — I used OpenAI-style ES with antithetic sampling when PPO could not assign credit through noisy rewards.
Evolution strategies (ES) optimize a model's parameters by evaluating many random perturbations and moving toward the ones that score best — no backpropagation through the environment is needed.
How I use it
In Real-time Data Systems, a Reinforcement Learning policy I trained with PPO stalled because the reward signal per step was far too noisy relative to the number of decisions in an episode. I retrained it with OpenAI-style ES: antithetic (mirrored) perturbations evaluated on the same recorded episodes, so shared noise cancels out, and a fitness that is the episode outcome itself. I kept the policy deliberately small — a few dozen parameters — to resist overfitting, and scored each candidate under both an optimistic and a pessimistic simulator so it could not win by exploiting one model's assumptions.
Why it matters
Choosing the optimizer that fits the signal, instead of the fashionable one, is a core ML engineering skill.