Where did the linear approximation/linearization formula come from? I understand that it takes root in the point-slope form and slope intercept form of a linear equation, but I don't understand where they got the exact formula $$f(x) \approx f(x_0)+f'(x_0)(x-x_0)$$ Also, where did Newton's Method come from? Can someone please explain this, or rather the individual components of the equation, to me?
Thanks!
3 Answers
Consider the two points $(x_0, f(x_0))$ and $(x,f(x))$. Now suppose we wanted to find the slope between these two points. Using the slope formula, we know that this slope is: \begin{align*} m &= \frac{f(x) - f(x_0)}{x - x_0} \\ m(x - x_0) &= f(x) - f(x_0) \\ f(x_0) + m(x - x_0) &= f(x) \\ f(x) &= f(x_0) + m(x - x_0) \end{align*}
Now provided that $x$ and $x_0$ are "close enough", we can approximate the slope of the tangent line of $f$ at $x_0$ to be $m$. Hence, we obtain: $$ f(x) \approx f(x_0) + f'(x_0)(x - x_0) $$
Contrary to Sanath Devalapurkar's answer, this is not really an instance of Taylor series so much as Taylor series are a generalization of this. There are two parts to linear approximation: the formula for the line, and the fact that it is an approximation to the function.
The formula for the line is, as you say, just the point-slope formula for the line with slope $f'(x_0)$ passing through the point $(x_0, f(x_0))$; i.e. it is a line through a point of the graph of $f$ whose slope is the derivative of $f$ there.
The fact that this is actually an approximation to $f$ is the definition of the derivative. To see this, let $L(x) = f(x_0) + f'(x_0) (x - x_0)$ be the function for the tangent line. Then the definition of $f'(x_0)$ is $$\lim_{x \to x_0} \frac{f(x) - L(x)}{x - x_0} = \lim_{x \to x_0} \frac{f(x) - f(x_0)}{x - x_0} - f'(x_0) = 0,$$ which says, intuitively, that when $x$ is close enough to $x_0$ that their difference $x - x_0$ is tiny, the difference $f(x) - L(x)$ is tiny even when compared to that. This is the sense in which $L(x)$ is a good approximation to $f(x)$.
For Newton's method, see the Wikipedia article, which has a nice animation.
The Taylor series (see the link for a "technical" introduction) of $f(x)$ about $x_0$ approximates the value of $f(x)$ in some sufficiently small neighborhood of $x_0$: $$f(x)= \sum_{n=0} ^ {\infty} \frac {f^{(n)}(x_0)}{n!} \, (x-x_0)^{n}$$ Your above written formula is the first-order approximation. To see why Taylor series work, visit this page. Also, see this M.SE question.
