02. Convergence of Random Variables, and Limit Theorems

2.1 Modes of Convergence for Random Variables

Almost Surely

Let $X_1,X_2,\cdots$ and $X$ be random variables

$X_n\to X$ almost surely (or with probability $1$) if

$$\mathbb{P}(X_n\to X\text{ as }n\to \infty )=1$$

Then we can abbreviate it as

$$X_n\to X\text{ a.s.}$$

Event Statement

$$\{X_n\to X\text{ as }n\to \infty \}\text{ is an event}$$

More formally it is

$$\{\omega \in \Omega :X_n(\omega )\to X(\omega )\text{ as }n\to \infty \}$$

Link to original

In Probability

Let $X_1,X_2,\cdots$ and $X$ be random variables

$X_n\to X$ in probability if

$$\forall ϵ>0:\mathbb{P}(|X_n-X|<ϵ)\to 1\text{ as }n\to \infty$$

Then we can abbreviate it as

$$X_n\stackrel{P}{\to }X\text{ as }n\to \infty$$

Link to original

In Distribution (Weakly)

Let $F_1,F_2,\cdots$ and $F$ be the distribution of $X_1,X_2,\cdots$ and $X$ respectively

$X_n\to X$ in distribution if
For every $x$ where $F$ is continuous at $x$ then

$$F_n(x)\to F(x)\text{ as }n\to \infty$$

Then we can abbreviate it as

$$X_n\stackrel{d}{\to }X\text{ as }n\to \infty$$

Link to original

---

2.2 Convergence in Distribution

Convergence in Distribution Does Not Require a Common Probability Space

Convergence in distribution depends only on

$$\text{distribution functions and not on any joint construction}$$

Link to original

---

2.3 Comparison of Different Modes of Convergence

Implications Between Modes of Convergence

Let $X_1,X_2,\cdots$ and $X$ be random variables then

$$X_n\to X\text{ almost surely }⟹X_n\to X\text{ in probability }⟹X_n\to X\text{ in distribution}$$

Proof

In Probability $⟹$ In Distribution

Let $F_n$ be the distribution function of $X_n$
Let $F$ be the distribution function of $X$

For any $x$ such that $F$ is continuous at $x$ then fix any $ϵ>0$ then
If $X_n\le x$ then either

$$X\le x+ϵ\text{ or }|X_n-X|>ϵ$$

Hence

$$\begin{array}{rl}{F}_n(x)& =\mathbb{P}(X_n\le x)\\ & \le \mathbb{P}(X\le x+ϵ\text{ or }|X_n-X|>ϵ)\\ & \le \mathbb{P}(X\le x+ϵ)+\mathbb{P}(|X_n-X|>ϵ)\\ & \to F(x+ϵ)\text{ as }n\to \infty \end{array}$$

So

$$F_n(x)<F(x+ϵ)+ϵ\text{ for all large enough }n$$

Similarly as $1-F_n(x)=\mathbb{P}(X_n>x)$ then

$$F_n(x)>F(x-ϵ)-ϵ\text{ for all large enough }n$$

As $ϵ>0$ and $F$ is continuous at $x$ then

$$F_n(x)\to F(x)\text{ as }n\to \infty$$

In Distribution $\text{centernot}⟹$ In Probability

Note that random variables do not need to be defined on the same probability space

Suppose $X$ and $Y$ are random variables with same distribution but

$$P(X=Y)<1$$

Then

$$X,X,X,\cdots \text{ converge to }Y\text{ in distribution}$$

However not in probability

Almost Surely $⟹$ In Probability

For $N\in \mathbb{N}$ and fix $ϵ>0$ then

Define event $A_n$ by

$$A_N=\{|X_n-X|<ϵ\text{ for all }n\ge N\}$$

Suppose $X_n\to X$ almost surely

If event $\{X_n\to X\}$ occurs then event $A_N$ occurs for some $N$ hence

$$\mathbb{P}\left(\bigcup A_N\right)=1$$

As $A_N$ is an increasing sequence of events
By Continuity of Probability from Below then

$$\underset{N\to \infty }{\lim }\mathbb{P}(A_N)=1$$

However

$$A_N⟹|X_N-X|<ϵ$$

So

$$\mathbb{P}(|X_N-X|<ϵ)\to 1$$

Since $ϵ$ is arbitrary then

$$X_n\to X\text{ in probability}$$

In Probability $\text{centernot}⟹$ Almost Surely

Consider sequence of independent random variables $X_n$ where

$$\mathbb{P}(X_n=1)=\frac{1}{n}\text{ and }\mathbb{P}(X_n=0)=\frac{n-1}{n}$$

with

$$X_n\to 0\text{ in probability}$$

as for any $ϵ>0$

$$\mathbb{P}(|X_n-0|<ϵ)\ge \mathbb{P}(X=0)\to 1$$

As $X_n$ only takes values $0$ and $1$ then event

$$\{X_n\to 0\}\text{ is equivalent to }\{X_n=0\text{eventually}\}$$

This is

$$\bigcup _{N\ge 1}{B}_N\text{ where }{B}_N=\{X_n=0\text{ for all }n\ge N\}$$

But for any $N$ and $K$ then

$$\begin{array}{rl}\mathbb{P}(B_N)& \le \mathbb{P}(X_n=0\text{ for all }n=N,\cdots ,N+K)\\ & =\frac{N-1}{N}\frac{N}{N+1}\frac{N+1}{N+2}\cdots \frac{N+K-1}{N+K}\\ & =\frac{N-1}{N+K}\end{array}$$

As $K\ge 1$ then

$$\mathbb{P}(B_N)=0$$

Hence by Continuity of Probability from Below then

$$\mathbb{P}\left(\bigcup _{N\ge 1}{B}_N\right)=0$$

Thus

$$\mathbb{P}(X_n\to 0)=0$$

Hence $X_n$ does not converge to $0$ almost surely

Link to original

Continuity of Probability from Below lemma

Let $A_n$ for $n\ge 1$ be an increasing sequence of events with

$$A_1\subseteq A_2\subseteq A_3\subseteq \cdots$$

Then

$$\underset{n\to \infty }{\lim }\mathbb{P}(A_n)=\mathbb{P}\left(\bigcup _{n\ge 1}{A}_n\right)$$

Proof

As sequence $A_n$ is increasing then

$$\begin{array}{rl}\mathbb{P}\left(\bigcup _{n\ge 1}{A}_n\right)& =\mathbb{P}\left(A_1\cup \bigcup _{n\ge 1}(A_{n+1}\setminus A_n)\right)\\ & =\mathbb{P}(A_1)+\sum _{i=1}^{\infty }\mathbb{P}(A_{i+1}\setminus A_i)\text{(using countable additivity)}\\ & =\mathbb{P}(A_1)+\underset{n\to \infty }{\lim }\sum _{i=1}^{n-1}\mathbb{P}(A_{i+1}\setminus A_i)\\ & =\underset{n\to \infty }{\lim }\left(\mathbb{P}(A_1)+\sum _{i=1}^{n-1}\mathbb{P}(A_{i+1}\setminus A_i)\right)\\ & =\underset{n\to \infty }{\lim }\mathbb{P}\left(A_1\cup \bigcup _{1\le i\le n-1}(A_{i+1}\setminus A_i)\right)\\ & =\underset{n\to \infty }{\lim }\mathbb{P}(A_n)\end{array}$$

Link to original

Convergence to a Constant in Distribution and Probability

Let $X_1,X_2,\cdots$ be a sequence of random variables defined on same probability space

If $X_n\to c$ in distribution for some constant $c$ then

$$X_n\to c\text{ in probability}$$

Link to original

---

2.4 Review: Weak Law of Large Numbers

Weak Law of Large Numbers

Let $X_1,X_2,\cdots$ be i.i.d random variables with finite mean $\mu$ and

$$S_n=X_1+X_2+\cdots +X_n$$

Then

$$\frac{S_n}{n}\stackrel{P}{\to }\mu \text{ as }n\to \infty$$

So for all $ϵ>0$ then

$$\mathbb{P}\left(\left|\frac{S_n}{n}-\mu \right|<ϵ\right)\to 1\text{ as }n\to \infty$$

Proof

Let $X_i$ be i.i.d with mean $\mu$ and variance ${\sigma }^2$ then

$$\mathbb{E}\left[\frac{S_n}{n}\right]=\mu$$

and

$$\begin{array}{rl}\text{var}\left(\frac{S_n}{n}\right)& =\frac{\text{var}(S_n)}{n^2}\\ & =\frac{\text{var}(X_1)+\text{var}(X_2)+\cdots +\text{var}(X_n)}{n^2}\\ & =\frac{n{\sigma }^2}{n^2}\\ & =\frac{{\sigma }^2}{n}\end{array}$$

Fix any $ϵ>0$ then by Chebyshev’s Inequality then

$$\begin{array}{rl}\mathbb{P}\left(\left|\frac{S_n}{n}-\mu \right|\ge ϵ\right)& \le \frac{\text{var}\left(\frac{S_n}{n}\right)}{{ϵ}^2}\\ & =\frac{{\sigma }^2}{n{ϵ}^2}\\ & \to 0\text{ as }n\to \infty \end{array}$$

Link to original

Markov's Inequality

Let $X$ be random variable taking non-negative values so $\mathbb{P}(X\ge 0)=1$ then
For any $z>0$

$$\mathbb{P}(X\ge z)\le \frac{\mathbb{E}[x]}{z}$$

Proof

Consider random variable

$$X_z=z{1}_{\{X\ge z\}}$$

Hence

1. $X_z$ takes value $0$ whenever $X$ in $[0,z)$
2. $X_z$ takes value $z$ whenever $X$ in $z,\infty )$

Thus

$$X\ge X_z$$

Hence

$$\mathbb{E}[X]\ge \mathbb{E}[X_z]=z\mathbb{E}[1_{\{X\ge z\}}]=z\mathbb{P}(X\ge z)$$

So result follows

Link to original

Chebyshev's Inequality

Let $Y$ be a random variable with finite mean and variance then

For any $ϵ>0$

$$\mathbb{P}(|Y-\mathbb{E}[Y]|\ge ϵ)\le \frac{\text{var}(Y)}{{ϵ}^2}$$

Proof

$$\begin{array}{rl}\mathbb{P}(|Y-\mathbb{E}[Y]|\ge ϵ)& =\mathbb{P}([Y-\mathbb{E}[Y]{]}^2\ge {ϵ}^2)\\ & \le \frac{\mathbb{E}([Y-\mathbb{E}[Y]{]}^2)}{{ϵ}^2}\\ & =\frac{\text{var}(Y)}{{ϵ}^2}\end{array}$$

Link to original

---

2.5 Strong Law of Large Numbers

Strong Law of Large Numbers

Let $X_1,X_2,\cdots$ be i.i.d with mean $\mu$ and

$$S_n=X_1+\cdots +X_n$$

Then

$$\frac{S_n}{n}\to \mu \text{ almost surely}\text{ as }n\to \infty$$

Proof - Non-Examinable

Assume additional condition $\mathbb{E}[X_n^4]<\infty$

Let $X_n$ be centred so define

$$W_n=X_n-\mu$$

Then

$$\mathbb{E}[W_n]=0$$

Hence

$$\mathbb{E}[X_n^4]<\infty ⟹\mathbb{E}[W_n^4]<\infty$$

Then

$$\begin{array}{rl}\mathbb{E}[(S_n-n\mu {)}^4]& =\mathbb{E}[(W_1+W_2+\cdots +W_n{)}^4]\\ & =\sum _{1\le i\le n}\mathbb{E}[W_i^4]+4\sum _{\begin{array}{c}1\le i,j\le n\\ i\ne j\end{array}}\mathbb{E}[W_i^3{W}_j]+3\sum _{\begin{array}{c}1\le i,j\le n\\ 1\ne j\end{array}}\mathbb{E}[W_i^2{W}_j^2]\\ & +6\sum _{\begin{array}{c}1\le i,j,k\le n\\ i,j,k\text{ distinct}\end{array}}\mathbb{E}[W_i^2{W}_j{W}_k]+\sum _{\begin{array}{c}1\le i,j,k,l\le n\\ i,j,k,l\text{ distinct}\end{array}}\mathbb{E}[W_i{W}_j{W}_k{W}_l]\end{array}$$

By independence and $\mathbb{E}[W_i]=0$ most of the terms vanish thus

$$\mathbb{E}[(S_n-n\mu {)}^4]=n\mathbb{E}[W_1^4]+3n(n-1)(\mathbb{E}[W_1^2]{)}^2\le 3n^2\mathbb{E}[W_1^4]$$

Hence

$$\begin{array}{rl}\mathbb{E}\left[\sum _{n=1}^{\infty }{\left(\frac{S_n}{n}-\mu \right)}^4\right]& =\sum _{n=1}^{\infty }\mathbb{E}\left[{\left(\frac{S_n}{n}-\mu \right)}^4\right]=\sum _{n=1}^{\infty }\frac{1}{n^4}\mathbb{E}[(S_n-n\mu {)}^4]\\ & \le \sum _{n=1}^{\infty }\frac{3\mathbb{E}[W_1^4]}{n^2}<\infty \end{array}$$

If $Z$ is a random variable with $\mathbb{E}[Z]<\infty$ then $\mathbb{P}(Z<\infty )=1$ hence with

$$Z={\left(\frac{S_n}{n}-\mu \right)}^5$$

Then

$$\mathbb{P}\left(\sum _{n=1}^{\infty }{\left(\frac{S_n}{n}-\mu \right)}^4<\infty \right)=1$$

As

$$\sum (a_n-\mu {)}^4\text{ is finite }\text{ then }{a}_n\to \mu \text{ as }n\to \infty$$

Hence

$$\mathbb{P}\left(\frac{S_n}{n}\to \mu \text{ as }n\to \infty \right)=1$$

Link to original

---

2.5.1 Proof of Strong Law of Large Numbers (Non-Examinable)

Proof is Part of Theorem Above

---

2.6 Central Limit Theorem: Statement and Examples

2.6.1 Central Limit Theorem

Central Limit Theorem

Let $X_1,X_2,\cdots$ be i.i.d random variables with mean $\mu$ and variance ${\sigma }^2\in (0,\infty )$ and

$$S_n=X_1+X_2+\cdots +X_n$$

Then

$$\frac{S_n-n\mu }{\sigma \sqrt{n}}\stackrel{d}{\to }N(0,1)\text{ as }n\to \infty$$

Properties of CLT

1. Distribution of $S_n$ concentrates around $n\mu$

2. Fluctuations of $S_n$ around $n\mu$ are of order $\sqrt{n}$

3. Asymptotic Distribution of these fluctuations are normal

Alternate Forms of CLT

$$\mathbb{P}\left(a\le \frac{S_n-n\mu }{\sigma \sqrt{n}}\le b\right)\to \mathbb{P}(a\le Z\le b)\text{ as }n\to \infty \text{ for }Z\sim N(0,1)\forall a\le b$$

$$\frac{S_n-n\mu }{\sqrt{n}}\stackrel{d}{\to }N(0,{\sigma }^2)\text{ as }n\to \infty$$

Link to original

---

2.6.2 Example: Tail Probabilities

Refer to Lecture Notes for Examples