02a Optimisers | Omnia mea

Kitchen sink optimiser model

A typical loss function of a model is comprised of loss from data (E) and loss from weight or a regulariser (R)

l o s s = E (w_{t}) + γ R (w_{t})

gradient

g_{t} = \nabla E (w_{t}) + γ \nabla R (w_{t})

velocity

v_{t + 1} = f (β_{1} v_{t} + (1 - β_{1}) (g_{t})^{2})

momentum

m_{t + 1} = h (β_{2} m_{t} + (1 - β_{2}) g_{t})

weight update

w_{t + 1} = w_{t} - α (\frac{m_{t + 1}}{\sqrt{v_{t + 1}} + ϵ}) - λ w_{t}

The final term is weight decay. We can derive all the optimisers based on the value of constants and the nature of $f$ and $h$