The A-Z Guide to Gradient Descent Algorithm and Its Variants
What is Gradient Descent? Gradient descent is an efficient first-order optimization algorithm for finding a differentiable function's global or local minimum. It estimates the values of parameters or coefficients that minimize a cost function. The gradient descent method has proved to be especially useful as it can be adopted in spaces of any number of dimensions. The gradient descent method can be used when parameters cannot be calculated analytically and is a good choice for the differentiable cost function. How does Gradient Descent work? Image Credit: Neural Networks and Deep Learning To get an intuitive idea of how Gradient Descent works, let us consider the entire range of values the parameters can take. Here the axes w and b represent the range of values the parameters w and b can take, respectively. In this case, these parameters express a simple linear unit. Hence, the curved surface shown represents the cost function J(w, b) would vary for different values of w and ...