04. Joint Distribution of Continuous Random Variables

4.1 Review of Jointly Continuous Random Variables

Joint Cumulative Distribution Function

Let $X$ and $Y$ be two random variables then

Joint Cumulative Distribution Function is defined by

$$F_{X,Y}(x,y)=\mathbb{P}(X\le x,Y\le y)$$

Link to original

Jointly Continuous

Let $X$ and $Y$ be two random variables then

$X$ and $Y$ are jointly continuous if joint cdf can be written as

$$F_{X,Y}(x,y)={\int }_{u=-\infty }^x{\int }_{v=-\infty }^{y}f(u,v)dudv$$

for some $f_{X,Y}$

Link to original

Joint PDF

Let $X$ and $Y$ be two random variables then

Joint PDF is defined by

$$f_{X,Y}(x,y)=\frac{{\partial }^2}{\partial x\partial y}{F}_{X,Y}(x,y)$$

with probability function

$$\mathbb{P}((X,Y)\in A)={\iint }_A{f}_{X,Y}(x,y)dxdy\text{ for general sets }A\subset {\mathbb{R}}^2$$

Link to original

Marginal Distribution

Let $X$ and $Y$ be two random variables then

Marginal Distribution of $X$ (similar for $Y$) is defined by

$$f_X(x)={\int }_{y=-\infty }^{\infty }{f}_{X,Y}(x,y)dy$$

Link to original

Expectation of Two Variables

Let $X$ and $Y$ be two random variables then

$$\mathbb{E}[h(X,Y)]={\int }_{x=-\infty }^{\infty }{\int }_{y=-\infty }^{\infty }h(x,y)f_{X,Y}(x,y)dxdy$$

Link to original

Independent of Two Variables

Let $X$ and $Y$ be two random variables then

$X$ and $Y$ are independent if

$$F_{X,Y}(x,y)=F_X(x)F_Y(y)\text{ for all }x,y$$

or

$$f_{X,Y}(x,y)=f_X(x)f_Y(y)$$

Link to original

---

4.2 Transformation Formula for Bivariate Probability Density Functions

Change of Variables Formula for Joint Densities

Suppose $T:(x,y)\mapsto (u,v)$ is a one-to-one mapping from $D\subseteq {\mathbb{R}}^2$ onto range $R\subseteq {\mathbb{R}}^2$

Jacobian $J$ as function of $(u,v)$ is defined by

$$J=\det \left(\begin{array}{cc}\frac{\partial x}{\partial u}& \frac{\partial x}{\partial v}\\ \frac{\partial y}{\partial u}& \frac{\partial y}{\partial v}\end{array}\right)=\frac{\partial x}{\partial u}\frac{\partial y}{\partial v}-\frac{\partial x}{\partial v}\frac{\partial y}{\partial u}$$

assuming that partial derivatives exist and are continuous

If $X,Y$ have joint probability density function $f_{X,Y}$ then

Random variables $U,V$ defined by

$$(U,V)=T(X,Y)$$

are jointly continuous with joint probability density function $f_{U,V}$ defined by

$$f_{U,V}(u,v)=\{\begin{array}{ll}{f}_{X,Y}(x(u,v),y(u,v))|J(u,v)|& \text{ if }(u,v)\in R\\ & \\ 0& \text{ otherwise}\end{array}$$

Proof

Suppose $A\subseteq D$ and $T(A)=B$ then as $T$ is one-to-one

$$\begin{array}{rl}\mathbb{P}((U,V)\in B)& =\mathbb{P}((X,Y)\in A)\\ & ={\iint }_A{f}_{X,Y}(x,y)dxdy\\ & ={\iint }_B{f}_{X,Y}(x(u,v),y(u,v))|J(u,v)|dudv\end{array}$$

Hence final integrand is joint pdf of $(U,V)$

Link to original

---

4.2.1 Examples of Transformed Bivariate PDFs

Convolution of Sum of Independent Continuous Variables

Let $X$ and $Y$ be two independent random variables then

$$f_{X+Y}(u)={\int }_{-\infty }^{\infty }{f}_X(v)f_Y(u-v)dv$$

Link to original

---

4.3 Multivariate Normal Distribution

Joint Density of i.i.d. Standard Normals

Let $Z_1,Z_2,\cdots ,Z_n$ be i.i.d standard normal random variables then

Joint Density Function is defined as

$$\begin{array}{rl}{f}_{\mathbf{Z}}(\mathbf{z})& =\prod _{i=1}^n\frac{1}{\sqrt{2\pi }}\exp \left(-\frac{z_i^2}{2}\right)\\ & =\frac{1}{(2\pi {)}^{\frac{n}{2}}}\exp \left(-\frac{1}{2}{\mathbf{z}}^T\mathbf{z}\right)\end{array}$$

Link to original

Density of an Affine Transformation of Standard Normals

Let $Z_1,Z_2,\cdots ,Z_n$ be i.i.d standard normal random variables

Define $W_1,\cdots ,W_n$ by

$$\left(\begin{array}{c}{W}_1\\ W_2\\ ⋮\\ W_n\end{array}\right)=A\left(\begin{array}{c}{Z}_1\\ Z_1\\ ⋮\\ Z_n\end{array}\right)+\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ⋮\\ {\mu }_n\end{array}\right)$$

where $A$ is some $n\times n$ matrix (assume invertible)

By Change of Variables (with Jacobian Constant as Linear Transformation) then

$$f_{\mathbf{W}}(\mathbf{w})=\frac{1}{(2\pi {)}^{\frac{n}{2}}|\det A|}\exp \left(-\frac{1}{2}(\mathbf{w}-\mu {)}^T(A{A}^T{)}^{-1}(\mathbf{w}-\mu {)}_{}\right)$$

Link to original

Covariance Matrix

Let $Z_1,Z_2,\cdots ,Z_n$ be i.i.d standard normal random variables

Define $W_1,\cdots ,W_n$ by

$$\left(\begin{array}{c}{W}_1\\ W_2\\ ⋮\\ W_n\end{array}\right)=A\left(\begin{array}{c}{Z}_1\\ Z_1\\ ⋮\\ Z_n\end{array}\right)+\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ⋮\\ {\mu }_n\end{array}\right)$$

where $A$ is some $n\times n$ matrix (assume invertible)

Covariance Matrix is defined as

$$\Sigma :=A{A}^T$$

so that

$$\text{cov}(W_i,W_j)=(A{A}^T{)}_{ij}={\Sigma }_{ij}$$

Link to original

Multivariate Normal Distribution

Let $Z_1,Z_2,\cdots ,Z_n$ be i.i.d standard normal random variables

Define $W_1,\cdots ,W_n$ by

$$\left(\begin{array}{c}{W}_1\\ W_2\\ ⋮\\ W_n\end{array}\right)=A\left(\begin{array}{c}{Z}_1\\ Z_1\\ ⋮\\ Z_n\end{array}\right)+\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ⋮\\ {\mu }_n\end{array}\right)$$

where $A$ is some $n\times n$ matrix (assume invertible)

$W_1,\cdots ,W_n$ have multivariate normal distribution with mean $\mu$ and covariance matrix $\Sigma$

Link to original

---

4.4 Conditional Densities

4.4.1 Conditioning on Events of Positive Probability

Conditional Probability

Let $A$ and $B$ be events with $\mathbb{P}(A)>0$ then

Conditional Probability of $B$ given $A$ is

$$\mathbb{P}(B\mid A):=\frac{\mathbb{P}(A\cap B)}{\mathbb{P}(A)}$$

Link to original

Conditional Distributions

Let $A$ be an event with $\mathbb{P}(A)>0$
Let $X$ be a random variable then

$$F_{X\mid A}(x):=\mathbb{P}(X\le x\mid A)=\frac{\mathbb{P}(\{X\le x\}\cap A)}{\mathbb{P}(A)}$$

Link to original

Conditional Probability Mass Function

Let $A$ be an event with $\mathbb{P}(A)>0$
Let $X$ be a discrete random variable then

$$p_{X\mid A}(x)=\mathbb{P}(X=x\mid A)$$

Link to original

Conditional Density Function

Let $A$ be an event with $\mathbb{P}(A)>0$
Let $X$ be a continuous random variable and for set $C$ then

$$\mathbb{P}(X\in C\mid A)={\int }_{x\in C}{f}_{X\mid A}(x)dx$$

Link to original

Conditional Expectation

Let $A$ be an event with $\mathbb{P}(A)>0$
Let $X$ be a random variable

Conditional Expectation of $X$ given $A$ is defined by

1. Discrete

$$\mathbb{E}[X\mid A]=\sum _{x}x{p}_{X\mid A}(x)$$

2. Continuous

$$\mathbb{E}[X\mid A]=\int x{f}_{X\mid A}(x)dx$$

Link to original

---

4.4.2 Definition of Conditional Probability Density Function, and an Example

Conditional Density Function Given

Let $X$ and $Y$ be continuous variables then

Conditional Density Function of $Y$ given $X=x$ is defined by

$$f_{Y\mid X=x}(y)=\frac{f_{X,Y}(x,y)}{f_X(x)}$$

Link to original

---

4.4.3 Example: Bivariate Normal Distribution

Conditional Distribution of the Bivariate Normal

Let $(X,Y)$ be bivariate normal with means ${\mu }_1,{\mu }_2$, variances ${\sigma }_1^2,{\sigma }_2^2$, and correlation $\rho$ then

$$Y\mid (X=x)\sim N\left({\mu }_2+\rho \frac{{\sigma }_2}{{\sigma }_1}(x-{\mu }_1),(1-{\rho }^2){\sigma }_2^2\right)$$

Link to original

---

4.5 Cautionary Tale

Refer to Lecture Notes page 38