Fit a model y(x) to some data points $(y_i,\,x_i)$. There are $m$ total points.
$y(x)$ contains some parameters $\eta$ that appear linearly, so that $y(x)$ has the form
$$y(x;\eta)=f_1(x)\eta_1+f_2(x)\eta_2+\ldots f_n(x)\eta_n,$$where $f_1(x)$ etc. are some functions of $x$. We want the best set of parameters $\eta$, that is, the $\eta$ that will minimize the sum of the square error between the model and that data at each point.
The $\eta$ that minimizes the sum square error is given by solving the following for $\eta$:
$$\hat{A}\eta = \hat{b},$$where
$$\hat{A} = A^TA,$$$$\hat{b} = A^Tb,$$$$A = \left[ \begin{array}{c c c c} f_1(x_1) & f_2(x_1) & \ldots & f_n(x_1) \\ f_1(x_2) & f_2(x_2) & \ldots & f_n(x_2) \\ f_1(x_3) & f_2(x_3) & \ldots & f_n(x_3) \\ & \vdots & & \\ f_1(x_m) & f_2(x_m) & \ldots & f_n(x_m) \end{array} \right] $$$$b = \left( \begin{array}{c} y_1 \\ y_2 \\ y_3 \\ \vdots \\ y_m \end{array} \right) $$For example $y(x;\vec{\eta}) = y(x; \alpha,\beta,\gamma) = \alpha x^2 + \beta x + \gamma$.
The model can be evaluated at each data point: \begin{align} \alpha x_1^2 + \beta x_1 + \gamma &= y_1 \\ \alpha x_2^2 + \beta x_2 + \gamma &= y_2 \\ \alpha x_3^2 + \beta x_3 + \gamma &= y_3 \\ \vdots & \\ \alpha x_m^2 + \beta x_m + \gamma &= y_m \end{align}
This can be rewritten as $$A\eta = b,$$ or
$$\left[ \begin{array}{c c c} x_1^2 & x_1 & 1 \\ x_2^2 & x_2 & 1 \\ x_3^2 & x_3 & 1 \\ & \vdots & \\ x_m^2 & x_m & 1 \end{array} \right] \left( \begin{array}{c} \alpha \\ \beta \\ \gamma \end{array} \right) = \left( \begin{array}{c} y_1 \\ y_2 \\ y_3 \\ \vdots \\ y_m \end{array} \right) $$Note that there is no solution to this problem in general. That is, there is no single $\eta$ array that can fit all $m$ data points.
Instead, we find the $\eta$ that minimizes the error between $y(x_i;\eta)$ and $y_i$ in some global sense for all data points $i$.
The error in each point is
$$e_i = y(x_i;\eta)-y_i,$$or, in matrix form, the error vector is
$$e = A\eta-b.$$Minimize the square of the length of the error vector. This quantity is the sum square error between the model and the data points:
$$|e|^2 = |A\eta-b|^2 = (A\eta-b)^T(A\eta-b)$$Expand this (and use $(A\eta)^T=\eta^TA^T$):
$$|e|^2 = \eta^TA^TA\eta - \eta^TA^Tb - b^TA\eta + b^Tb$$The third term on the right is $-b^TA\eta = -(A\eta)^Tb = -\eta^TA^Tb$. Combining this with the second term on the right of the above equation gives
$$|e|^2 = \eta^TA^TA\eta - 2\eta^TA^Tb + b^Tb$$Now, $b^Tb$ is a scalar, so the $\eta$ that minimizes $|e|^2$ will also minimize $|e|^2-b^Tb$. Likewise, the $\eta$ that minimizes $|e|^2-b^Tb$ will also minimize $\frac{1}{2}(|e|^2-b^Tb)$. Let this equal $q$. Then minimize $q$:
$$q = \frac{1}{2}\eta^TA^TA\eta - \eta^TA^Tb.$$Let $\hat{A} = A^TA$ and $\hat{b} = A^Tb$. Then we have:
$$q = \frac{1}{2}\eta^T\hat{A}\eta - \eta^T\hat{b}.$$Consider the case where $\hat{A}$, $\hat{b}$, and $\eta$ are scalars. Then we would have $q = \frac{1}{2}\hat{A}\eta^2 - \hat{b}\eta$. Minimize $q$ by solving $\frac{dq}{d\eta}=\hat{A}\eta-\hat{b} = 0$. Then the $\eta$ that minimizes the square error is the solution to $\hat{A}\eta=\hat{b}$.
Show that solving $\hat{A}\eta=\hat{b}$ is the same as minimizing $q = \frac{1}{2}\eta^T\hat{A}\eta - \eta^T\hat{b}.$
For the given $(T_i,\,R_i)$ data given below, find the best fit values of $k$ and $E$ for the model $$R(T) = ke^{-E/T}.$$
using Plots
using PyFormattedStrings
T = [ 500.0, 650, 800, 950, 1100, 1250, 1400, 1550, 1700, 1850, 2000]
R = [5.06898064e-06, 9.49169460e-05, 5.91159348e-04, 2.78345192e-03,
4.88059454e-03, 1.62306334e-02, 3.57378985e-02, 2.30324885e-02,
8.40528066e-02, 1.07758145e-01, 6.39353585e-02];
##################### SOLUTION
m = length(T)
#--------- form y and x data
y = log.(R)
x = 1 ./T
#--------- form vector b and matrix A, along with bhat and Ahat
b = copy(y)
f_1 = ones(m)
f_2 = -x
A = [f_1 f_2]
bhat = A' * b
Ahat = A' * A
#--------- solve for η and recover parameters
η = Ahat \ bhat
lnk = η[1]
E = η[2]
k = exp(lnk)
println(f"Optimal parameters: k={k:.3f}, E={E:.0f}")
#-------- plot the model curve versus the data points
TT = LinRange(minimum(T), maximum(T), 1000) # lots of T
RR = k*exp.(-E./TT) # model R
scatter(T, R, label="data")
plot!(TT,RR, label="fitted model", lw=2)
plot!(xlabel="T", ylabel="R")
plot!(legend_foreground_color=nothing)
Optimal parameters: k=2.819, E=6659
Fit the $T$, $R$ data in the previous example to a quartic polynomial: $$R(T) = aT^4 + bT^3 + cT^2 + dT + e.$$
Here, $y$ is just $R$ and $x$ is just $T$. Also, $\vec{\eta}=(a,\,b,\,c,\,d,\,e)$.
x = T
y = R
#----------------
b = y
f_1 = x.^4
f_2 = x.^3
f_3 = x.^2
f_4 = copy(x)
f_5 = ones(length(x))
A = [f_1 f_2 f_3 f_4 f_5]
bhat = A' * b
Ahat = A' * A
#----------------
η = Ahat \ bhat
a,b,c,d,e = η
#----------------
TT = LinRange(minimum(T), maximum(T), 1000) # lots of T
RR = a*TT.^4 .+ b*TT.^3 .+ c*TT.^2 .+ d*TT .+ e
scatter(T, R, label="data")
plot!(TT,RR, label="fitted model", lw=2)
plot!(xlabel="T", ylabel="R")
plot!(legend_foreground_color=nothing)