Underflow & overflow<\/em><\/p>\n Underflow: where very small numbers –> zero, and overflow: big ones –> zero.<\/p>\n For instance,\u00a0for a softmax:<\/p>\n We have an issue if our x’s are very negative, because by exponentiating them we push them very close to zero, possibly resulting in underflow. Similarly, with huge x, we might get overflow. We can do a bit of\u00a0normalization<\/b> to overcome this: by subtracting max(x), we don’t change the computation, but we ensure that in the numerator everything is <0, and in the denominator, at least one of the entries is exp(0)=1, which means we’re not going to get underflow!<\/p>\n Poor conditioning<\/em><\/p>\n This is where we have a matrix -remember, we can usefully think of\u00a0matrices as transformations<\/a>– which responds very drastically to small changes in input – precisely the kind of small changes we might induce by digitizing an analogue variable.<\/p>\n Conditioning is defined by a condition number, which are calculated from the ratio of the maximum to the minimum eigenvalue. High condition numbers are problematic.<\/p>\n Gradient descent (vanilla)<\/strong><\/p>\n Gradient descent is the most obvious weapon in our arsenal for minimising some function, which we want to do a lot. Typically this function is a\u00a0cost<\/em> function of some kind – and by minimizing it, we maximise some measure of ‘goodness’ of our model.<\/p>\n In gradient descent, we work out the derivative of our function to be minimized, then take small steps in the direction\u00a0opposite<\/em> to the gradient.<\/p>\n Why do this? Well, if we’re in a region where the gradient is positive, this means that increasing X with increase Y. So we don’t want to do this (we’re minimising Y, remember). So we should decrease X.<\/p>\n Conversely, if dy\/dx is negative, then if we increase X, we’ll decrease Y. So that’s exactly what we should do: increase X.<\/p>\n Nicely illustrated by Figure 4.1 from the book: Gradient descent\u00a0(partial derivatives)<\/strong><\/p>\n What if we have multiple independent variables, and our cost is a function of lots of them:<\/p>\n Well, we can basically do the same trick, but use partial derivatives. What results is a gradient vector:<\/p>\n The gradient vector is denoted as :\u00a0 In both the single and multiple gradient case, how much we move in the direction of the gradient is a matter of choice: we define a\u00a0learning rate or step size<\/strong> which describes how much to jump at each step. Too small and everything will be super slow, and too large, and you risk missing the minimum.<\/p>\n Stationary points<\/strong><\/p>\n Stationary points are where the gradient = 0. They can be maxima, minima, or saddle points, best illustrated graphically:<\/p>\n Credit here<\/a><\/p>\n We can discriminate between these with the second derivative; positive second derivatives imply a minimum, negative second derivatives a maximum, and zeros a saddle point. This is straightforward in two dimensions, but for higher dimensional problems, we need to turn to the Jacobian.<\/p>\n Jacobians and Hessians<\/strong><\/p>\n Confusing terms, these. A Jacobian is a matrix filled with differentials. The gradient is itself the Jacobian of the original function.<\/p>\n A Hessian is a certain kind of a Jacobian: it’s the Jacobian of the gradient, when we have multiple variables and the gradient is a vector of partial derivatives.<\/p>\n So:<\/p>\n Why do we care about the Hessian?<\/em><\/p>\n Two reasons, so far as I can make out:<\/p>\n Newton’s Method<\/strong><\/p>\n Traditional gradient descent allows us to chug down our gradient, but it can be very slow.<\/p>\n Newton, widely held to be a pretty clever chap, described a neat method for doing zero-finding for a function – that is, finding the value of x\u00a0where f(x) = 0. This means that if we use f’ instead of f, and find the value of x where f'(x) =0, we’ve done minimisation<\/em>. To do this, we need the Hessian, because this provides us with the ‘gradient of the gradient’, so to speak.<\/p>\n Using Newton’s method for zero-finding<\/em><\/p>\n Newton’s method gives us an iterative way of finding better and better approximations to the roots of y(x).<\/p>\n Graphically, here’s how it works:<\/p>\n And repeat. This is illustrated below (from Wikipedia<\/a>):<\/p>\n<\/p>\n
<\/p>\n
<\/p>\n
![\"graddescent\"](\"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/06\/graddescent.jpg?resize=660%2C519\")
<\/a><\/p>\n
<\/p>\n
<\/p>\n
<\/p>\n
![\"function-min-max\"](\"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/06\/function-min-max.gif?resize=370%2C180\")
<\/a>s.<\/p>\n<\/p>\n
\u00a0<— Jacobian (
) = the\u00a0Hessian (incomplete; my Latex editor started playing up when I went to multiple lines.).<\/p>\n
\n
\n
\n
<\/li>\n
<\/li>\n<\/ol>\n
![\"300px-NewtonIteration_Ani\"](\"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/06\/300px-NewtonIteration_Ani.gif?resize=300%2C214\")
<\/a><\/p>\n