06 - Inner Product Spaces

Norms

Let $V$ be a vector space over field $F$ then

Mapping $||\cdot ||:V\to \mathbb{F}$ is a norm on $V$ if
1)

$$||f||\ge 0\text{ for all }f\in V\text{ with }||f||=0⟺f=0\in V$$

$$||\lambda f||=|l|||f||\text{ for all }\lambda \in \mathbb{F}\text{ and }f\in V$$

3. Triangle Inequality

$$||f+g||\le ||f||+||g||\text{ for all }f,g\in V$$

Types of Norms

Vectors in ${\mathbb{R}}^n$

For $x\in {\mathbb{R}}^n$ where $x=(x_1,x_2,\cdots ,x_n{)}^T$ then

$$||x||\equiv ||x|{|}_2=(x_1^2+x_2^2+\cdots +x_n^2{)}^{\frac{1}{2}}=\sqrt{x^{T}x}$$

is the ${\ell }^2$- or vector two-norm

Continuous Function

For continuous functions on $[a,b]$

$$||f||\equiv ||f|{|}_{\infty }=\underset{x\in [a,b]}{\max }|f(x)|$$

Integrable Functions

For functions in $V$ where

$$V=L_w^2(a,b)\equiv \left\{f:[a,b]\to \mathbb{R}:{\int }_a^{b}w(x)[f(x){]}^{2}dx<\infty \right\}$$

for some weight function $w(x)>0$ then

$$||f||=||f|{|}_2={\left({\int }_a^{b}w(x)[f(x){]}^{2}dx\right)}^{\frac{1}{2}}$$

is the $L^2$- or two-norm

If $||f||=0$ then $f=0$ almost everywhere on $[a,b]$

Note that space $L^2(a,b)$ is a common abbreviation for $L_w^2(a,b)$ when $w(x)\equiv 1$

Real Inner Product Space

Real Inner Product Space is a vector space $V$ over $\mathbb{R}$ with inner product mapping

$$⟨\cdot ,\cdot ⟩:V\times V\to \mathbb{R}$$

such that

$$|v,v|\ge 0\text{ for all }v\in V\text{ and }⟨v,v⟩=0⟺v=0$$

$$⟨u,v⟩=⟨v,u⟩\text{ for all }u,v\in V$$

$$⟨\alpha u+\beta v,z⟩=\alpha ⟨u,z⟩+\beta ⟨v,z⟩\text{ for all }u,v,z\in V\text{ and all }\alpha ,\beta \in \mathbb{R}$$

Properties of Inner Product Spaces

Let $V$ be an inner product space with inner product $⟨\cdot ,\cdot ⟩$

Norm on $V$

$$⟨v,v{⟩}^{\frac{1}{2}}$$

defines a norm on $V$

Orthogonality

For $u,v\in V$ then

$$⟨u,v⟩=0⟺u\text{ and }v\text{ are orthogonal}$$

Pythagoras Theorem:

For $u,v\in V$ such that $⟨u,v⟩=0$ then

$$||u\pm v|{|}^2=||u|{|}^2+||v|{|}^2$$

Proof

$$\begin{array}{rl}||u\pm v|{|}^2& =⟨u\pm v,u\pm v⟩=⟨u,u\pm v⟩\pm ⟨v,u\pm v⟩\\ & =⟨u,u\pm v⟩\pm ⟨u\pm v,u⟩\\ & =⟨u,u⟩\pm ⟨u,v⟩\pm ⟨u,v⟩+⟨v,v⟩\\ & =||u|{|}^2+||v|{|}^2\end{array}$$

Cauchy-Schwarz Inequality

For any $u,v\in V$

$$|⟨u,v⟩|\le ||u||||v||$$

Proof

For $\lambda \in \mathbb{R}$ then

$$0\le ⟨u-\lambda v,u-\lambda v⟩=||u|{|}^2-2\lambda ⟨u,v⟩+{\lambda }^2||v|{|}^2=\varphi (\lambda )$$

which is a quadratic in $\lambda$

$\varphi (\lambda )$ has a minimiser at ${\lambda }_{\ast }=\frac{⟨u,v⟩}{||v|{|}^2}$
So as $\varphi ({\lambda }_{\ast })\ge 0$ then

$$||u|{|}^2-\frac{⟨u,v{⟩}^2}{||v|{|}^2}\ge 0$$

Hence we get inequality

Triangle Inequality

For $u,v\in V$ then

$$||u+v||\le ||u||+||v||$$

Proof

$$||u+v|{|}^2=⟨u+v,u+v⟩=||u|{|}^2+2⟨u,v⟩+||v|{|}^2$$

By Cauchy-Schwarz Inequality then

$$||u+v|{|}^2\le ||u|{|}^2+2||u||||v||+||v|{|}^2=(||u||+||v||{)}^2$$

Hence

$$||u+v||\le ||u||+||v||$$

Gram-Schmidt Orthogonalisation Procedure

Consider normal equation $A\alpha =\phi$

Let $\{{\varphi }_0,{\varphi }_1,\cdots ,{\varphi }_n\}$ be a basis for ${\Pi }_n$ then

$$p_n(x)=\sum _{k=0}^n{\beta }_k{\varphi }_k(x)$$

So normal equations for $A\alpha =\varphi$ are

$${\int }_a^{b}w(x)\left(f(x)-\sum _{k=0}^n{\beta }_k{\varphi }_k(x)\right){\varphi }_i(x)dx=0\text{ for }i=0,1,\cdots ,n$$

So in other words

$$\sum _{k=0}^n\left({\int }_a^{b}w(x){\varphi }_k(x){\varphi }_i(x)dx\right){\beta }_k={\int }_a^{b}w(x)f(x){\varphi }_i(x)dx\text{ for }i=0,1,\cdots ,n$$

Hence

$$A\beta =\phi$$

where $\beta =({\beta }_0,{\beta }_1,\cdots ,{\beta }_n{)}^T$ and $\phi =(f_1,f_2,\cdots ,f_n{)}^T$ and

$$a_{i,k}={\int }_a^{b}w(x){\varphi }_k(x){\varphi }_i(x)dx\text{ and }{f}_i={\int }_a^{b}w(x)f(x){\varphi }_i(x)dx$$

Hence $A$ is diagonal if

$$⟨{\varphi }_i,{\varphi }_k⟩={\int }_a^{b}w(x){\varphi }_i(x){\varphi }_k(x)dx\{\begin{array}{ll}=0& \text{ if }i\ne k\\ \ne 0& \text{ if }i=k\end{array}$$

where set of orthogonal polynomials can be produced below

Construction of Orthogonal Polynomials via Gram–Schmidt lemma

Let ${\varphi }_0,\cdots ,{\varphi }_k$ with ${\varphi }_i\in {\Pi }_i$ for each $i$ are orthogonal with respect to inner product

$$⟨f,g⟩={\int }_a^{b}w(x)f(x)g(x)dx$$

Then

$${\varphi }_{k+1}(x)=x^{k+1}-\sum _{i=0}^k{\lambda }_i{\varphi }_i(x)$$

satisfies

$$⟨{\varphi }_{k+1},{\varphi }_j⟩={\int }_a^{b}w(x){\varphi }_{k+1}(x){\varphi }_j(x)dx=0\text{ for }j=0,1,\cdots ,k$$

with

$${\lambda }_j=\frac{⟨x^{k+1},{\varphi }_j⟩}{⟨{\varphi }_j,{\varphi }_j⟩}\text{ for }j=0,1,\cdots ,k$$

Proof

For any $j$ and $0\le j\le k$ then

$$\begin{array}{rl}⟨{\varphi }_{k+1},{\varphi }_j⟩& =⟨x^{k+1},{\varphi }_j⟩-\sum _{i=0}^k{\lambda }_i⟨{\varphi }_i,{\varphi }_j⟩\\ & =⟨x^{k+1},{\varphi }_j⟩-{\lambda }_j⟨{\varphi }_j,{\varphi }_j⟩\\ & \text{ by orthogonality of }{\varphi }_i\text{ and }{\varphi }_j\text{ for }i\ne j\\ & =0\text{ by definition of }{\lambda }_j\end{array}$$

Types of Orthogonal Polynomials

Legendre Polynomials

Inner Product defined by

$$⟨f,g⟩={\int }_{-1}^{1}f(x)g(x)dx$$

which gives orthogonal polynomials

$${\varphi }_0(x)\equiv 1,{\varphi }_1(x)=x,{\varphi }_2(x)=x^2-\frac{1}{3},{\varphi }_3(x)=x^3-\frac{3}{5}x,\cdots$$

Chebyshev Polynomials

Inner Product defined by

$$⟨f,g⟩={\int }_{-1}^1\frac{f(x)g(x)}{\sqrt{1-x^2}}dx$$

which gives orthogonal polynomials

$${\varphi }_0(x)\equiv 1,{\varphi }_1(x)=x,{\varphi }_2(x)=2x^2-1,{\varphi }_3(x)=4x^3-3x,\cdots$$

Laguerre Polynomials

Inner Product defined by

$$⟨f,g⟩={\int }_{-1}^1{e}^{-x}f(x)g(x)dx$$

which gives orthogonal polynomials

$${\varphi }_0(x)\equiv 1,{\varphi }_1(x)=1-x,{\varphi }_2(x)=2-4x+x^2,{\varphi }_3(x)=6-18x+9x^2-x^3,\cdots$$

Orthogonal Polynomials Annihilate Lower-Degree Polynomials lemma

Let $\{{\varphi }_0,{\varphi }_1,\cdots ,{\varphi }_k,\cdots \}$ be orthogonal polynomials for inner product $⟨\cdot ,\cdot ⟩$

Then

$$⟨{\varphi }_k,q⟩=0\text{ for all }q\in {\Pi }_{k-1}$$

Proof

If $q\in {\Pi }_{k-1}$ then

$$q(x)=\sum _{i=0}^{k-1}{\sigma }_i{\varphi }_i(x)\text{ for }{\sigma }_i\in \mathbb{R},i=0,1,\cdots ,k-1$$

Hence

$$⟨{\varphi }_k,q⟩=\sum _{i=0}^{k-1}{\sigma }_i⟨{\varphi }_k,{\varphi }_i⟩=0$$