ML

定义

约定：

$x_j^{(i)}$：训练数据中的第i列中的第j个特征值 value of feature j in the ith training example

$x^{(i)}$：训练数据中第i列 the input (features) of the ith training example

$m$：训练数据集条数 the number of training examples

$n$：特征数量 the number of features

Week1 - 机器学习基本概念

Cost Function损失函数

Squared error function/Mean squared function均方误差: 解析失败 (语法错误): {\displaystyle J(θ)=\frac{1}{2m}\sum_{i=1}^m(h_θ(x^{(i)})-y^{(i)})^2}
Cross entropy交叉熵: 解析失败 (语法错误): {\displaystyle J(θ)=-\frac{1}{m}\sum_{i=1}^m[y^{(i)}*logh_θ(x^{(i)})+(1-y^{(i)})*log(1-h_θ(x^{(i)}))]}

Gradient Descent梯度下降

解析失败 (语法错误): {\displaystyle θ_j:=θ_j-α\frac{∂}{∂θ_j}J(θ)}
对于线性回归模型，其损失函数为均方误差，故有：
解析失败 (语法错误): {\displaystyle \frac{∂}{∂θ_j}J(θ)= \frac{∂}{∂θ_j}(\frac{1}{2m}\sum_{i=1}^m(h_θ(x^{(i)})-y^{(i)})^2)}

解析失败 (语法错误): {\displaystyle = \frac{1}{2m}\frac{∂}{∂θ_j}(\sum_{i=1}^m(h_θ(x^{(i)})-y^{(i)})^2)}

解析失败 (语法错误): {\displaystyle = \frac{1}{2m}\sum_{i=1}^m( \frac{∂}{∂θ_j}(h_θ(x^{(i)})-y^{(i)})^2 )}

解析失败 (语法错误): {\displaystyle = \frac{1}{m}\sum_{i=1}^m( (h_θ(x^{(i)})-y^{(i)}) \frac{∂}{∂θ_j}h_θ(x^{(i)}) ) //链式求导法式}

解析失败 (语法错误): {\displaystyle = \frac{1}{m}\sum_{i=1}^m( (h_θ(x^{(i)})-y^{(i)}) \frac{∂}{∂θ_j}x^{(i)}θ ) }

解析失败 (语法错误): {\displaystyle = \frac{1}{m}\sum_{i=1}^m( (h_θ(x^{(i)})-y^{(i)}) \frac{∂}{∂θ_j}\sum_{k=0}^{n}x_k^{(i)}θ_k ) }

对于j>=1:

解析失败 (语法错误): {\displaystyle = \frac{1}{m}\sum_{i=1}^m( (h_θ(x^{(i)})-y^{(i)}) x_j^{(i)} ) }

解析失败 (语法错误): {\displaystyle = \frac{1}{m} (h_θ(x)-y) x_{j} }

Week2 - Multivariate Linear Regression

Multivariate Linear Regression模型的计算

解析失败 (语法错误): {\displaystyle h_θ(x) = θ_0x_0 + θ_1x_1 + θ_2x_2 + ... + θ_nx_n}

解析失败 (语法错误): {\displaystyle = [θ_0x_0^{(1)}, θ_0x_0^{(2)}, ..., θ_0x_0^{(m)}] + [θ_1x_1^{(1)}, θ_1x_1^{(2)}, ..., θ_1x_1^{(m)}] + ... + [θ_nx_n^{(1)}, θ_nx_n^{(2)}, ..., θ_nx_n^{(m)}] }

解析失败 (语法错误): {\displaystyle = [θ_0x_0^{(1)}+θ_1x_1^{(1)}+...+θ_nx_n^{(1)}, \ \ \ θ_0x_0^{(2)}+θ_1x_1^{(2)}+...+θ_nx_n^{(2)}, \ \ \ θ_0x_0^{(m)}+θ_1x_1^{(m)}+...+θ_nx_n^{(m)}] }

解析失败 (语法错误): {\displaystyle = θ^Tx}

其中，
解析失败 (语法错误): {\displaystyle x=\begin{vmatrix} x_0 \\ x_1 \\ x_2 \\ ... \\ x_n \end{vmatrix} = \begin{vmatrix} x_0^{(1)} & x_0^{(2)} & ... & x_0^{(m)} \\ x_1^{(1)} & x_1^{(2)} & ... & x_1^{(m)} \\ x_2^{(1)} & x_2^{(2)} & ... & x_2^{(m)} \\ ... & ... & ... & ...\\ x_n^{(1)} & x_n^{(2)} & ... & x_n^{(m)} \\ \end{vmatrix} , θ=\begin{vmatrix} θ_0 \\ θ_1\\ θ_2\\ ...\\ θ_n \end{vmatrix} }

m为训练数据组数，n为特征个数（通常，为了方便处理，会令解析失败 (语法错误): {\displaystyle x_0^{(i)}=1, i=1,2,...,m）} 。

数据归一化：Feature Scaling & Standard Normalization

解析失败 (语法错误): {\displaystyle x_i := \frac{x_i-μ_i}{s_i} }
其中，解析失败 (语法错误): {\displaystyle μ_i} 是第i个特征数据x_i的均值，而 $s_i$则要视情况而定：

• Feature Scaling:$s_i$为$x_i$中最大值与最小值的差（max-min）；
• Standard Normalization:$s_i$为$x_i$中数据标准差（standard deviation）。

特别注意，通过 Feature scaling训练出模型后，在进行预测时，同样需要对输入特征数据进行归一化。

Normal Equation标准工程

解析失败 (语法错误): {\displaystyle θ = (X^TX)^{-1}X^Ty}

Week3 - Logistic Regression & Overfitting

Logistic Regression

Sigmoid Function - S函数

解析失败 (语法错误): {\displaystyle h_θ(x)=g(θ^Tx)}
解析失败 (语法错误): {\displaystyle z = θ^Tx}
$g(z)=\frac{1}{1+e^{-z}}$

Cost Function

解析失败 (语法错误): {\displaystyle J(θ)=-\frac{1}{m}\sum_{i=1}^m[y^{(i)}*logh_θ(x^{(i)})+(1-y^{(i)})*log(1-h_θ(x^{(i)}))]}
向量化形式：
解析失败 (语法错误): {\displaystyle J(θ) = \frac{1}{m}( -y^Tlog(h) - (1-y)^Tlog(1-h) ) }

Gradient Descent

解析失败 (语法错误): {\displaystyle θ_j:=θ_j-α\frac{∂}{∂θ_j}J(θ)}

解析失败 (语法错误): {\displaystyle = θ_j-\frac{α}{m}\sum_{i=1}^m( (h_θ(x^{(i)})-y^{(i)}) x_j^{(i)} ) }

附推导过程如下：

解析失败 (语法错误): {\displaystyle \frac{∂}{∂θ_j}J(θ) = \frac{∂}{∂θ_j}\{-\frac{1}{m}\sum_{i=1}^m[y^{(i)}*logh_θ(x^{(i)})+(1-y^{(i)})*log(1-h_θ(x^{(i)}))]\}}

解析失败 (语法错误): {\displaystyle =-\frac{1}{m}\sum_{i=1}^m\frac{∂}{∂θ_j}[y^{(i)}*logh_θ(x^{(i)})+(1-y^{(i)})*log(1-h_θ(x^{(i)}))]} 解析失败 (语法错误): {\displaystyle ------式1)}

其中，

解析失败 (语法错误): {\displaystyle \frac{∂}{∂θ_j}[y^{(i)}*logh_θ(x^{(i)})] = y^{(i)}*\frac{∂}{∂θ_j}[logh_θ(x^{(i)})] = \frac{y^{(i)}}{h_θ(x^{(i)})*ln(e)}*\frac{∂}{∂θ_j}h_θ(x^{(i)})}

解析失败 (语法错误): {\displaystyle \frac{∂}{∂θ_j}[(1-y^{(i)})*log(1-h_θ(x^{(i)}))] = (1-y^{(i)})*\frac{∂}{∂θ_j}[log(1-h_θ(x^{(i)}))] = \frac{(1-y^{(i)})}{(1-h_θ(x^{(i)}))*ln(e)}*\frac{∂}{∂θ_j}(1-h_θ(x^{(i)}))}

由于解析失败 (语法错误): {\displaystyle \frac{∂}{∂θ_j}(1-h_θ(x^{(i)})) = -\frac{∂}{∂θ_j}h_θ(x^{(i)})} ，故有：

解析失败 (语法错误): {\displaystyle \frac{∂}{∂θ_j}[y^{(i)}*logh_θ(x^{(i)})+(1-y^{(i)})*log(1-h_θ(x^{(i)}))] = \frac{y^{(i)}}{h_θ(x^{(i)})*ln(e)}*\frac{∂}{∂θ_j}h_θ(x^{(i)}) + \frac{(1-y^{(i)})}{(1-h_θ(x^{(i)}))*ln(e)}*\frac{∂}{∂θ_j}(1-h_θ(x^{(i)}))}

解析失败 (语法错误): {\displaystyle = \frac{y^{(i)}}{h_θ(x^{(i)})*ln(e)}*\frac{∂}{∂θ_j}h_θ(x^{(i)}) - \frac{(1-y^{(i)})}{(1-h_θ(x^{(i)}))*ln(e)}*\frac{∂}{∂θ_j}h_θ(x^{(i)})}

解析失败 (语法错误): {\displaystyle = (\frac{y^{(i)}}{h_θ(x^{(i)})*ln(e)}- \frac{(1-y^{(i)})}{(1-h_θ(x^{(i)}))*ln(e)})*\frac{∂}{∂θ_j}h_θ(x^{(i)}) }

解析失败 (语法错误): {\displaystyle = \frac{y^{(i)}-h_θ(x^{(i)})}{h_θ(x^{(i)})*(1-h_θ(x^{(i)}))*ln(e)}*\frac{∂}{∂θ_j}h_θ(x^{(i)}) } //将 解析失败 (语法错误): {\displaystyle h_θ(x^{(i)})=g(z)=\frac{1}{1+e^{-z}}} 代入

解析失败 (语法错误): {\displaystyle = \frac{y^{(i)}*(1+e^{-z})^2-(1+e^{-z})}{e^{-z}*ln(e)} * \frac{∂}{∂θ_j}h_θ(x^{(i)}) }

解析失败 (语法错误): {\displaystyle = \frac{y^{(i)}*(1+e^{-z})^2-(1+e^{-z})}{e^{-z}} * \frac{∂}{∂θ_j}h_θ(x^{(i)}) } 解析失败 (语法错误): {\displaystyle ------式2)}

而解析失败 (语法错误): {\displaystyle \frac{∂}{∂θ_j}h_θ(x^{(i)}) = g'(z)*z'(θ^Tx^{(i)}) = (\frac{1}{1+e^{-z}})'*z'(θ^Tx^{(i)})}

解析失败 (语法错误): {\displaystyle = ((1+e^{-z})^{-1})'*z'(θ^Tx^{(i)})}

解析失败 (语法错误): {\displaystyle = \frac{e^{-z}}{(1+e^{-z})^{2}}*z'(θ^Tx^{(i)})}

解析失败 (语法错误): {\displaystyle = \frac{e^{-z}}{(1+e^{-z})^{2}}*\frac{∂}{∂θ_j}(θ^Tx^{(i)})}

解析失败 (语法错误): {\displaystyle = \frac{e^{-z}}{(1+e^{-z})^{2}}*\frac{∂}{∂θ_j}(θ_0*x_0^{(i)} + θ_1*x_1^{(i)} + θ_2*x_2^{(i)} +...+ θ_j*x_j^{(i)} +...+ θ_n*x_n^{(i)} )}

$=\frac{e^{-z}}{(1+e^{-z}{)}^2}*x_j^{(i)}$ 解析失败 (语法错误): {\displaystyle ------式3)}

将式3)代入式2)：

解析失败 (语法错误): {\displaystyle \frac{∂}{∂θ_j}[y^{(i)}*logh_θ(x^{(i)})+(1-y^{(i)})*log(1-h_θ(x^{(i)}))] = (y^{(i)} - \frac{1}{1+e^{-z}})*x_j^{(i)}}

解析失败 (语法错误): {\displaystyle = (y^{(i)} - h_θ(x^{(i)}))*x_j^{(i)}} 解析失败 (语法错误): {\displaystyle ------式4)}

将式4)代入式1)：

解析失败 (语法错误): {\displaystyle θ_j:= θ_j-\frac{α}{m}\sum_{i=1}^m( (h_θ(x^{(i)})-y^{(i)}) x_j^{(i)} ) }

向量化形式：
解析失败 (语法错误): {\displaystyle θ = θ - \frac{α}{m}X^T(g(Xθ) - \vec y) }

解决Overfitting

针对 hypothesis function，引入 Regularation parameter(解析失败 (语法错误): {\displaystyle λ} )到 Cost function中：
解析失败 (语法错误): {\displaystyle J(θ)=\frac{1}{2m}\sum_{i=1}^m(h_θ(x^{(i)})-y^{(i)})^2 + λ\sum_{j=1}^nθ_j^2}