Moments and Moment Generating Functions

The kth moment of a random variable X is given by E[X^k]. The kth central moment of a random variable X is given by E[(X-E[X])^k].

The moment generating function of X is given by:

$$M(\theta) = E[e^{X\theta}] = \int_{-\infty}^{\infty} e^{x\theta} f(x) dx.$$ (9)

If X is non-negative, we can define its Laplace transform:

$$L(s) = M(-s) = \int_{0}{\infty} e^{-sx}f(x)dx.$$ (10)

Taking the power series expansion of $e^{X\theta}$ yields:

$$e^{X \theta} = 1+X \theta + \frac{X^{2} \theta^{2}}{2!} + \cdots$$ (11)

Taking the expectation yields:

$$E[e^{X \theta}] = 1+E[X] \theta + \frac{E[X^{2}] \theta^{2}}{2!} + \cdots$$ (12)

We can then find the kth moment of X by taking the kth derivative of the moment generating function and setting $\theta=0$.

$$E[X^{k}] = \frac{d^{k}M(\theta)}{d\theta^{k}}\left\vert _{\theta=0} \right.$$ (13)

For the Laplace transform, the moments can be found using:

$$E[X^{k}] = (-1)^{k} \frac{d^{k}L(s)}{ds^{k}}\left\vert _{s=0} \right.$$ (14)

Example:

$$f(x)=\lambda e^{-\lambda x} \vspace{0.3in} \mbox{for $x>0$ }$$ (15)

$$M(\theta) \int_{0}^{\infty} e^{x\theta} \lambda e^{-\lambda x} dx$$ = (16)

$$\lambda \int_{0}^{\infty} e^{(\theta-\lambda)x} dx$$ = (17)

$$\frac{\lambda}{\theta - \lambda} e^{(\theta-\lambda)x} \vert _{0}^{\infty}$$ = (18)

$$\frac{\lambda}{\lambda - \theta} \vspace{0.3in} \mbox{for $\theta <\lambda$ }$$ = (19)

$$E[X] = \frac{dM(\theta)}{d\theta}\left\vert _{\theta=0} \righ... ...da}{(\lambda-\theta)^{2}}\vert _{\theta=0} = \frac{1}{\lambda}$$ (20)

$$E[X^{2}] = \frac{d^{2}M(\theta)}{d\theta^{2}}\left\vert _{\th... ... = \frac{2 \lambda^{2}}{\lambda^{4}} = \frac{2}{\lambda^{2}}$$ (21)

$$Var(X)= \frac{2}{\lambda^{2}} - \frac{1}{\lambda^{2}} = \frac{1}{\lambda^{2}}$$ (22)

For X non-negative, integer-valued, and discrete, we can define the z-transform:

$$G(z)=E[z^{X}] = \sum_{i=0}^{\infty} p(i) z^{i}$$ (23)

The first and second moments can be found as follows:

$$E[X]=\frac{dG(z)}{dz}\vert _{z=1}$$ (24)

$$E[X^{2}]=\frac{d^{2}G(z)}{dz^{2}}\vert _{z=1} + \frac{dG(z)}{dz}\vert _{z=1}$$ (25)

A property of transforms, known as the convolution theorem is stated as follows: Let $X_{1},X_{2}, \ldots, X_{n}$ be mutually independent random variables. Let $Y=\sum_{i=1}{n} X_{i}$. If $M_{X_{i}}(\theta)$ exists for all i, then $M_{Y}(\theta)$ exists, and:

$$M_{Y}(\theta)= \prod_{i=1}^{n} M_{X_{i}}(\theta).$$ (26)

Example: Let X₁ and X₂ be independent exponentially distributed random variables with parameters $\lambda_{1}$ and $\lambda_{2}$respectively. Let Y = X₁+X₂. Find the distribution of Y.

The Laplace transforms for X₁ and X₂ are:

$$L_{X_{1}}(s) = \frac{\lambda_{1}}{\lambda_{1}+s}$$ (27)

$$L_{X_{2}}(s) = \frac{\lambda_{2}}{\lambda_{2}+s}$$ (28)

By the convolution theorem:

$$L_{Y}(s) = \frac{\lambda_{1} \lambda{2}}{(\lambda_{1}+s)(\lambda_{2}+s)}$$ (29)

Expanding this into partial fractions:

$$L_{Y}(s) = \frac{a_{1}\lambda_{1}}{(\lambda_{1}+s)} + \frac{a_{2}\lambda_{2}}{(\lambda_{2}+s)}$$ (30)

where:

$$a_{1} = \frac{\lambda_{2}}{\lambda_{2}-\lambda_{1}}$$ (31)

$$a_{2} = \frac{\lambda_{1}}{\lambda_{1}-\lambda_{2}}$$ (32)

Taking the inverse Laplace transform yields:

$$f(y) = a_{1}\lambda_{1}e^{\lambda_{1}x} + a_{2}\lambda_{2}e^{\lambda_{2}x}.$$ (33)