The problem of least squares is widespread in applications amongst almost all natural sciences. The setting is a follows: We observe input data and some output data together with a model represented by a function which depends on a set of parameters . The task now is to find the parameter which minimizes
The case where is linear has been addressed first by Legendre although Gauss claimed that he have been had a solution earlier. The optimal parameter can be explicitly given in terms of the data.
In case that is non-linear, it usually is not possible to solve for directly. The first key concept of the method is to linearize the non-linear problem through a Taylor expansion, i.e.
One can then solve the linearized problem recursively, until some condition is fulfilled. In each step, the Levenberg-Maquard algorithm "interpolates" between the gradient-descend method and the Gauss-Newton algorithm.