06 - Strong Law of Large Numbers

Strong Law of Large Numbers

Let $X_1,X_2,\cdots$ be i.i.d with mean $\mu$ and

$$S_n=X_1+\cdots +X_n$$

Then

$$\frac{S_n}{n}\to \mu \text{ almost surely}\text{ as }n\to \infty$$

Proof - Non-Examinable

Assume additional condition $\mathbb{E}[X_n^4]<\infty$

Let $X_n$ be centred so define

$$W_n=X_n-\mu$$

Then

$$\mathbb{E}[W_n]=0$$

Hence

$$\mathbb{E}[X_n^4]<\infty ⟹\mathbb{E}[W_n^4]<\infty$$

Then

$$\begin{array}{rl}\mathbb{E}[(S_n-n\mu {)}^4]& =\mathbb{E}[(W_1+W_2+\cdots +W_n{)}^4]\\ & =\sum _{1\le i\le n}\mathbb{E}[W_i^4]+4\sum _{\begin{array}{c}1\le i,j\le n\\ i\ne j\end{array}}\mathbb{E}[W_i^3{W}_j]+3\sum _{\begin{array}{c}1\le i,j\le n\\ 1\ne j\end{array}}\mathbb{E}[W_i^2{W}_j^2]\\ & +6\sum _{\begin{array}{c}1\le i,j,k\le n\\ i,j,k\text{ distinct}\end{array}}\mathbb{E}[W_i^2{W}_j{W}_k]+\sum _{\begin{array}{c}1\le i,j,k,l\le n\\ i,j,k,l\text{ distinct}\end{array}}\mathbb{E}[W_i{W}_j{W}_k{W}_l]\end{array}$$

By independence and $\mathbb{E}[W_i]=0$ most of the terms vanish thus

$$\mathbb{E}[(S_n-n\mu {)}^4]=n\mathbb{E}[W_1^4]+3n(n-1)(\mathbb{E}[W_1^2]{)}^2\le 3n^2\mathbb{E}[W_1^4]$$

Hence

$$\begin{array}{rl}\mathbb{E}\left[\sum _{n=1}^{\infty }{\left(\frac{S_n}{n}-\mu \right)}^4\right]& =\sum _{n=1}^{\infty }\mathbb{E}\left[{\left(\frac{S_n}{n}-\mu \right)}^4\right]=\sum _{n=1}^{\infty }\frac{1}{n^4}\mathbb{E}[(S_n-n\mu {)}^4]\\ & \le \sum _{n=1}^{\infty }\frac{3\mathbb{E}[W_1^4]}{n^2}<\infty \end{array}$$

If $Z$ is a random variable with $\mathbb{E}[Z]<\infty$ then $\mathbb{P}(Z<\infty )=1$ hence with

$$Z={\left(\frac{S_n}{n}-\mu \right)}^5$$

Then

$$\mathbb{P}\left(\sum _{n=1}^{\infty }{\left(\frac{S_n}{n}-\mu \right)}^4<\infty \right)=1$$

As

$$\sum (a_n-\mu {)}^4\text{ is finite }\text{ then }{a}_n\to \mu \text{ as }n\to \infty$$

Hence

$$\mathbb{P}\left(\frac{S_n}{n}\to \mu \text{ as }n\to \infty \right)=1$$