Fundamental Math Behind AI

Derivatives

Derivatives are used to measure the instantaneous rate of change of function f at point x w/ the notation f'(x). But why is this useful? To find f’s extreme values, the min and max. And in terms of neural networks we want to know whether we can increase or decrease a neuron to make it more accurate. And so f'(x) tells us how to change f(x)’s input to get the smallest and largest values.

We have things called critical points where f(c) - meaning the point on the graph is a local or extreme min or max. E.g. 0 increased to 100 decreased to 5, 0 will be the extreme min, 100 is the extreme max and 5 is the local min.

The second derivative of f measures the rate of change of the first derivative f′, so if c is a local max, then f''(c) < 0.

If the second derivative is negative, e.g. c = 0 then it is a local max because the rate of change from positive to negative occurs when reaching a peak and the opposite is true for a local min, descending, stopping, then ascending again after the lowest point.

To find the global min and max we must assess all locals and select the lowest and highest points of them all.