Dust: Pretraining Transformers without Backpropagation
It promotes itself as the first zeroth-order pretraining method that can compete with backpropagation.
Table of Contents
1. Motivation
2. Methodology
Dust adopts evolution strategy, but its search space moves from weight space to activation space.