Convergence of Random Variables
Preamble
Recall that for $X_1, X_2, …, X_n$ an IID sample, with $Var(X_i) = \sigma^2$,
\[\begin{aligned} Var(\bar X_n) &= Var(n^{-1} \sum_i^n X_i ) \\ &= n^{-2} Var(\sum_i^n X_i) \\ &= n^{-2} \sum_i^n Var(X_i) \\ &= n^{-2} (n \sigma^2) \\ &= \sigma^2 / n. \end{aligned}\]Law of Large Numbers
Statement: Let $X_1, X_2, …, X_n$ be an IID sample. Let $\mu = E[X_i]$ and $\sigma^2 = Var(X_i)$. Then
\[\bar X_n \to \mu\]as $ n \to \infty $, in probability.
Proof: For some $\epsilon > 0$,
\[P( \mid \bar X_n - \mu \mid > \epsilon) \leq \frac{Var(\bar X_n)}{\epsilon^2} = \frac{\sigma^2}{n \epsilon^2},\]by Chebyshev’s inequality, which tends to $0$ as $n \to \infty$. $\square$
Central Limit Theorem
Statement: Let $X_1, X_2, …, X_n$ be IID with mean $\mu$ and variance $\sigma^2$. Let $\bar X = n^{-1} \sum_i^n X_i.$ Then
\[\bar X \to N(\mu, \sigma^2 / n)\]as $n \to \infty$. This is remarkable because nothing was assumed about the form of the distrbution of $X$.