Discrete Random Variables

The following derivation relies on the formula for the expectation of a function of a discrete random variable, \(g(X)\), which is given by \(E(g(X)) = \sum g(x_i)P(X=x_i)\).
Let the function be \(g(X) = aX + b\).$$\begin{aligned}E(aX+b) &= \sum_{i} (ax_i + b) P(X=x_i) && \text{(by the formula for } E(g(X))\text{)} \\ &= \sum_{i} (ax_i P(X=x_i) + b P(X=x_i)) && \text{(distribute the probability)} \\ &= \sum_{i} ax_i P(X=x_i) + \sum_{i} b P(X=x_i) && \text{(split the summation)} \\ &= a \sum_{i} x_i P(X=x_i) + b \sum_{i} P(X=x_i) && \text{(factor out constants } a \text{ and } b\text{)} \\ &= a E(X) + b(1) && \text{(using } E(X) \text{ definition and } \sum P(X=x_i)=1\text{)} \\ &= aE(X) + b\end{aligned}$$

Let \(\mu = E(X)\).$$\begin{aligned}V(X) &= E[(X - \mu)^2] \\ &= E[X^2 - 2\mu X + \mu^2] \\ &= E(X^2) - E(2\mu X) + E(\mu^2) && \text{(by linearity of expectation)} \\ &= E(X^2) - 2\mu E(X) + \mu^2 && \text{(since } \mu \text{ and } \mu^2 \text{ are constants)} \\ &= E(X^2) - 2\mu(\mu) + \mu^2 \\ &= E(X^2) - 2\mu^2 + \mu^2 \\ &= E(X^2) - \mu^2 \\ &= E(X^2) - [E(X)]^2\end{aligned}$$

Proof of the Expected Value \(E(X)\):For a uniform distribution on \(\{1, 2, \dots, n\}\), the probability of any outcome is \(P(X=i) = \frac{1}{n}\). $$ \begin{aligned} E(X) &= \sum_{i=1}^n i \cdot P(X=i) \\ &= \sum_{i=1}^n i \cdot \frac{1}{n} \\ &= \frac{1}{n} \sum_{i=1}^n i \quad \text{(factoring out the constant } 1/n\text{)} \\ &= \frac{1}{n} \left( \frac{n(n+1)}{2} \right) \quad \text{(using the formula for the sum of integers)} \\ &= \frac{n+1}{2} \end{aligned} $$

• \(\begin{aligned}[t]E(X)&=0\times P(X=0)+1\times P(X=1)\\&=0\times(1-p)+1\times p\\&=p\end{aligned}\)
• \(\begin{aligned}[t] V(X) &= (0-p)^2(1-p) + (1-p)^2 p \\ &= p^2(1-p) + p(1-p)^2 \\ &= p(1-p) [p + (1-p)] \\ &= p(1-p) \\ \end{aligned} \)

Consider the case where \(n = 3\). Let \(X_1\), \(X_2\), and \(X_3\) be three independent Bernoulli random variables, each with a probability of success \(p\). Define \(X = X_1 + X_2 + X_3\), which represents a binomial random variable.

• The possible values of \(X\) are \(0, 1, 2, 3\).
• Probability calculations:
  • \(\begin{aligned}[t] P(X = 0) &= P(X_1 = 0 \text{ and } X_2 = 0 \text{ and } X_3 = 0) \\ &= P(X_1 = 0) P(X_2 = 0) P(X_3 = 0) \quad \text{(since \)X_1, X_2, X_3\( are independent)} \\ &= (1-p)^3 \\ &= \binom{3}{0} p^0 (1-p)^3 \end{aligned}\)
  • \(\begin{aligned}[t] P(X = 1) &= P(X_1 = 1 \text{ and } X_2 = 0 \text{ and } X_3 = 0) + P(X_1 = 0 \text{ and } X_2 = 1 \text{ and } X_3 = 0) \\ &\quad + P(X_1 = 0 \text{ and } X_2 = 0 \text{ and } X_3 = 1) \\ &= p (1-p)^2 + p (1-p)^2 + p (1-p)^2 \\ &= 3 p (1-p)^2 \\ &= \binom{3}{1} p^1 (1-p)^2 \end{aligned}\)
  • \(\begin{aligned}[t] P(X = 2) &= P(X_1 = 1 \text{ and } X_2 = 1 \text{ and } X_3 = 0) + P(X_1 = 1 \text{ and } X_2 = 0 \text{ and } X_3 = 1) \\ &\quad + P(X_1 = 0 \text{ and } X_2 = 1 \text{ and } X_3 = 1) \\ &= p^2 (1-p) + p^2 (1-p) + p^2 (1-p) \\ &= 3 p^2 (1-p) \\ &= \binom{3}{2} p^2 (1-p)^1 \end{aligned}\)
  • \(\begin{aligned}[t] P(X = 3) &= P(X_1 = 1 \text{ and } X_2 = 1 \text{ and } X_3 = 1) \\ &= p^3 \\ &= \binom{3}{3} p^3 (1-p)^0 \end{aligned}\)

Thus, \(P(X = x) = \binom{3}{x} p^x (1-p)^{3-x}\) for \(x = 0, 1, 2, 3\), matching the binomial distribution form.
The logic generalizes for any \(n\). To obtain exactly \(x\) successes, we must choose \(x\) of the \(n\) trials to be successes, which can be done in \(\binom{n}{x}\) ways. Each specific arrangement of \(x\) successes and \(n-x\) failures has a probability of \(p^x (1-p)^{n-x}\). By the addition rule, the total probability is the sum over all these arrangements, resulting in \(P(X=x) = \binom{n}{x}p^x (1-p)^{n-x}\).