Automatic Differentiation (AutoGrad)

AutoGrad is a method that can automatically compute the gradient/derivative of a program or function. It’s principle is that all numerical computation can be decomposed to basic operations such that these operations can be easily composed to compute the gradient with chain rule.

For example, let take the following function to see how autograd works, where \(x\) is input and \(w,b\) are weight and bias.

\[ f(x,w,b) = \frac{1}{\exp(-(wx+b)) + 1} \]

The computation of \(f\) can be decomposed to basic ops and construct a computation graph.

x --> (*) --> (+) --> (*) --> (exp) --> (+) --> (/) --> f(x,w,b)
       ↑       ↑       ↑                 ↑       ↑
       w       b       -1                1       1

Let’s denote the node from left to right as \(h_1,h_2,\dots,h_6\). By chain rule, we can compute the gradient w.r.t. \(x,w,b\)

\[ f_w= \frac{\partial f}{\partial h_6} \times \frac{\partial h_6}{\partial h_5} \times \frac{\partial h_5}{\partial h_4} \times \frac{\partial h_4}{\partial h_3} \times \frac{\partial h_3}{\partial h_2} \times \frac{\partial h_2}{\partial h_1} \times \frac{\partial h_1}{\partial w} \]