Random Variables - HTML Version

Random Variables

As with the other refresher courses, you will possibly have seen the material below. If you have not, don’t worry, every student’s journey is different. This material will all be taught again once you arrive

What is a Random Variable?

In the background material, we described a football match between Manchester United and Manchester City as an experiment, with the result of the match expressed as outcomes. Instead of representing the outcomes as Win, Lose or Draw, we could represent as the number of points United received for each outcome, namely 3 for Win, 1 for Draw and 0 for Lose. We could also represent the result as the final score, and use a positive or negative number for the difference in scores; e.g. -1 would mean City scored one more goal than United, 0 would be a draw, etc. The difference in score tells us who won, and thus how many points they got, while also providing information on performance.

Outcomes are more useful to us mathematically. If we represent each outcome as the points received, then summing all outcomes over the season gives the final points earned. If we sum the goal difference in each game, we get the goal difference over the entire season.

A student answers 10 questions in an exam. We could represent the final result as a sequence of correct ( $C$) or wrong ( $W$) answers, e.g. CCWWCWCCWW. Instead, let $C = 1$ and $W = 0$ and write the sequence as 1100101100. Now, summing the sequence tells how many correct answers, $5$, were given, summarising the 10 digit sequence and the final result in just one integer.

The overall performance and outlook of a company depends on several factors, often interacting with each other in complex and random ways. However, its share (or stock) price says how much outside investors are willing to pay for the company, giving a summary of both past performance and future outlook. One price represents all of the complex underlying factors.

Not every experiment results in a number, with the above 3 examples demonstrating how useful it is to translate outcomes into numbers. A random variable makes this transformation.

The random variable now represents the experiment mathematically, with any outcome being a realisation. We define the probability of an outcome happening as $P(X = x)$.

I think this definition of a random variable is rather weird and difficult to conceptualise but is one you will see in University. I simplify this to a random variable being the outcomes of an experiment written as numbers, ignoring the “function" aspect.

Types of Random Variables

Random variables are functions and so we use appropriate terminology. The sample space of the experiment $\Omega$ is the domain of $X$. The range of $X$, Range( $X$), is the set of numbers that represent the outcomes. For example, if $\Omega = \{W, D, L\}$ for the football match, we might let Range( $X$) = $\{3, 1, 0\}$, the corresponding points received. Here the range has a finite number of outcomes. If $X$ corresponds to the number of correct answers out of 10, then Range $(X) = \{0,1,\dots,10\}$, again finite.

If for the football match $X$ is the goals difference instead of points, the range is then both positive and negative integers. In theory, there is no upper (or lower) bound to the goal difference, and so we could let Range $(X) = \mathbb{Z}$. The set of integers is infinite but it is discrete. Sets like $\mathbb{N}$ and $\mathbb{Q}$ are also discrete; you will learn why but the following definition is enough for now.

Recall that $\mathbb{Z}$ and $\mathbb{N}$ are subsets of $\mathbb{Q}$. One intuition for what makes a set discrete is if there are “gaps"; we can mark each number down distinctly. Another intuition is that we can”list" all the numbers.

However, for the share price example, the range is all positive real numbers $\mathbb{R}^+$, not just $\mathbb{N}$. Imagine in the football game we measured the distance all players ran; this would also not be discrete. We call such random variables continuous.

A continuous range does not need to go as far as $\infty$, a common example is the interval $(a,b)$; all numbers between $a$ and $b$. This is not discrete as there are no gaps; if we choose any two numbers, we can always find a number between them that is also in the interval. Despite not being infinitely long, there are infinitely many numbers within.

You will see a more detailed explanation later this year.

Measuring the exact time an exam takes or the heights of students in a class are further examples of continuous random variables. However, we often measure these things in rounded units; using seconds or centimetres but can always get more and more accurate.

Key Takeaways:

• A random variable is a mathematical formulation of an experiment

• The elements of the range of a random variable are the numbers representing the sample space of the experiment

• A discrete random variable has a range that is finite or infinite with “gaps"

• A continuous random variable has a continuous range with no “gaps".

Probability Distributions

It’s helpful to translate experiments into numbers, but we also want to have some idea how often these occur; their probability.

In football, we want to know how often goals are scored or the probability a team wins a certain number of games. When it comes to share prices, we want to know the probability of the price increasing by a certain amount, or at what price we should sell. When measuring the heights of students in a class, we want to know what proportion of students should be below a certain height. For a random variable, we describe the probabilities of specific realisations through a probability distribution.

Probability Mass Function

We have already seen an example of a probability distribution in the refresher. When we roll a fair die, there are six equally likely outcomes. We give each of these outcomes a probability of $\tfrac{1}{6}$ and put them in a table as below (now with random variable notation).

The roll of a die has a finite range and so we write each $P(X=x)$ explicitly in a table. When the range is something like $\mathbb{Z}$, we need to be a bit more general (which we will see later).

Note: The domain of $p(x)$ is $\mathbb{R}$ rather than just $\text{Range}(X)$. For any real numbers not in $\text{Range}(X)$ we define their probability to be 0 and don’t need to list them.

In order to be a pmf, $p(x)$ must satisfy:

• $0 \leq p(x) \leq 1, \forall x$: each probability must be between 0 and 1

• $\sum_{x}p(x) = 1$: the sum of all probabilities must be 1.

We can use the pmf to calculate probabilities on subsets of the range by summing as before.

Cumulative Distribution Function

For a continuous random variable, it is impossible to list each probability explicitly. In fact, each outcome must have a probability of 0; $P(X = x) = 0 \ \forall x$. Why? There are too many options to think any are actually possible to happen. This is one of the trickier concepts when thinking of infinity and so don’t worry yet if you don’t understand.

The probability a share price rises to exactly B#280 is 0; there are too many possibilities. However, we can find the probability the price is between B#279.99 and B#280.01. So, rather than $P(X = x)$, think of $P(X \in (a,b))$ where $(a,b) \subseteq \text{Range}(X)$.

The cdf is the probability of $X$ taking any value up to and including $x$; the cumulative probability up to $x$. Writing outcomes as numbers allows us to use set notation here.

We use the cdf to find the probability of $X$ being in any interval as: $$\begin{aligned} P(a \leq X \leq b) &= P(X \leq b)-P(X\leq a)\\ &=F(b)-F(a) \end{aligned}$$

Note: The set $\{X \leq x\}$ can be written as $\{X<x\}\cup \{x\}$. For a continuous random variable, knowing $P(X =x) = 0$ implies $P(X\leq x) = P(X<x)$.

The same definition of the cdf works for discrete random variables, and can actually be calculated explicitly as: $$\begin{aligned} F(x) = P(X \leq x) = \sum_{y: y \leq x}p(y) \end{aligned}$$

We see the stepwise nature of the cdf for a discrete random variable; at each possible realisation there is an increase after adding on the corresponding probability. We sum the bars of the histogram for the pmf as we go along.

Just like with the pmf, we write the cdf as a table or create a general formula. See below for a table corresponding to the die roll.

A key difference from the continuous case is that because individual outcomes may have nonzero probabilities in the discrete case, $P(X\leq x) \neq P(X<x)$ for discrete $X$.

If we are asked to find $P(X > x)$, we use the fact that probabilities sum to 1 to get: $$\begin{aligned} P(X > x) &= 1 - P(X \leq x)\\ &= 1-F(x) \end{aligned}$$ However, if asked for $P(X \geq x)$, the discrete case has to add on $P(X=x)$.

The cdf of the largest outcome should always be 1, as everything must be less than or equal to it. Also, note how the cdf is a non-decreasing function; it either stays the same or increases at every jump.

A cdf table gives you the pmf by inverting the process.

Probability Density Function

So for discrete random variables, we can find the probability, or “mass", of both points and intervals. For continuous random variables, only the mass of intervals is possible; individual points have mass 0. So, in the continuous case, we instead think of”density" of points. The mass of the interval is then a function of the density of the points within and the width or “volume" of the interval.

To get from the pmf to cdf we summed. Integration is basically summing over an interval.

Note: We use $a$ in the integral as a “dummy" variable because it would not make sense to have $x$ both as the variable being integrated and in the limits of the integral.

Conversely, we have: $$\begin{aligned} f(x) = \tfrac{dF(x)}{dx} \end{aligned}$$

So, we can get the pdf from the cdf or vice versa.

The pdf will usually be presented in the following way: $$\begin{aligned} f(x) = \begin{cases} \text{some function of }x, \text{if } x \in \text{Range($X$)}\\ 0, \text{otherwise} \end{cases} \end{aligned}$$

See that $f$ is a function of the whole real line; any value not within the range is given density zero.

As with the pmf, we need the “sum", namely integral, to be 1:

• $\int_{-\infty}^{\infty}f(x)dx = 1$

However, as we are dealing with density not mass, the pdf is not restricted to being between $0$ and $1$ like the pmf is.

Note that our definition for $f$ allowing for any real number also allows us to define integration from $-\infty$ to $\infty$. If we were to plot the pdf $f(x)$, these requirements corresponds to a function that will always be nonnegative and with an area under the curve of 1.

The following is Question 1 in the November 2020 Further Maths statistics exam.

The following was Question 8 in the 2023 Further Maths Statistics paper and has a video solution accompanying it.

We refer to the pmf or pdf of a random variable as its probability distribution. We often use $f$ rather than $p$ to refer to the function in general.
Key Takeaways:

• A discrete random variable has a pmf which gives the probability of any realisation

• A continuous random variable has a pdf which gives the density of any realisation, and probabilities exist only over intervals.

• The cdf sums the pmf or integrates the pdf to get probabilities of sets

• A pmf/pdf must always sum/integrate to 1

• A pmf must always have values between 0 and 1

Expectation and Variance

A probability distribution tells us how likely it is that United score 3 goals against City. It tells us what proportion of students should be taller than 180cm. It tells us the probability a share price falls below B#200.

However, it’s often more interesting to know what happens on average. What is the average number of goals United score? What is the average height in the class? What is the average result I get when I roll a die? What is the average share price?

When we know these average values, it’s also interesting to know how variable the data is around this average. If United score 2 goals a game on average, are they scoring 2 goals consistently or do they sometimes score 0 and sometimes 4? Does my class have some very tall and very short students, or is everyone roughly the same? Is the share price very volatile, or is it a safe investment?

We explore ways you may have seen before of summarising the properties of probability distributions and random variables. If you have not seen these concepts in such detail, don’t worry, it will be taught once you arrive.

Expectation

For a random variable $X$, we are interested in the average result, or what we expect to happen. Think about calculating the mean from a frequency distribution table, except with probability rather than frequency.

For a discrete random variable: $$\begin{aligned} E[X] = \sum_{x}x\cdot p(x) \end{aligned}$$ $E[X]$ is a generalisation of the weighted average: we take each outcome $x$ and weight it by its corresponding probability. We then sum all the weighted outcomes.

So, when calculating the expect number of goals United score, we weight the amount of goals by the probability they score that many goals and sum everything.

E $[X]$ extends to continuous random variables as we might expect: $$\begin{aligned} E[X] = \int_{-\infty}^{\infty}x\cdot f(x)dx \end{aligned}$$ For outcomes that lie outside the range, their probability and thus weight is 0. From this point onwards, we will use the pdf/integral definition for convenience unless otherwise stated; for discrete use a sum instead of integral.

For some function $g(x)$, we can also define the expectation as: $$\begin{aligned} E[g(X)] = \int_{-\infty}^{\infty}g(x)f(x)dx \end{aligned}$$ This is is often called the law of the unconcious statistician, or LOTUS (although you probably have not seen it called this). For example, if $g(x) = x^2$, then: $$\begin{aligned} E[X^2] = \int_{-\infty}^{\infty}x^2f(x)dx \end{aligned}$$

Note: We only replace the first $x$ with $g(x)$ in the formula, we do not change the $f(x)$. We change the value but not the weight.

Using the formula for $E[g(X)]$ (feel free to try prove the following yourself), for some $a, b \in \mathbb{R}$ we get: $$\begin{aligned} \label{eqn:sum_exp} E[aX + b] = aE[X]+b \end{aligned}$$ We call this linearity of expectation. If $b = 0$, this shows $E[cX] = cE[X]$; we can take the constant factor outside of the expectation.

Think about if we subtracted 5cm from the height of every student in the class. The new expected value would just be the previous minus 5cm. If we doubled their heights, we would double the average.

Suppose we have two random variables $X$ and $Y$. Then: $$\begin{aligned} E[X+Y] = E[X]+E[Y] \end{aligned}$$

Combining all our rules together we have: $$\begin{aligned} E[aX+bY] = aE[X]+bE[Y] \end{aligned}$$

Intuition - Games

In the background material, we mentioned how games (and gambling) were often a driving force for the development of probability. Games also provide some intuition for expectations.

In a game, think of $E[X]$ as the average score. So, for a 6-sided die, I expect to get 3.5 each time I roll, despite the fact I can never actually roll 3.5. This is important when the score relates to some monetary payout; if I get B# $x$ when I roll an $x$, then I make B#3.50 on average each time I play the game.

$E[X]$ gives a fair price for the game. If I have to pay B#3 to play this game then I should always play; I make B#0.50 on average every time I play. If it’s B#4 pound, I should find something else to do as I lose B#0.50 each time (all casino games that are pure chance are similar). B#3.50 would be a fair price; playing and not playing end up the same on average and it’s up to you how much risk you want to take. This line of thinking will be covered in greater detail once you start studying game theory and financial mathematics.

Instead, suppose I roll the die and the amount I receive is the result squared. Now, my average payout will be $E[X^2]$; I calculate how much I would get each time ( $x^2$) and weight it by the probability.

Now suppose the game consists of rolling a 6-sided die and a 12-sided die. I add both numbers together and receive the total. If I let $X$ be the 6-sided and $Y$ be the 12-sided, my return is $X+Y$. My average return will then be $E[X+Y]$ and so I can calculate both expectations separately and add them together.

Median and Mode

When looking at the heights of students in a class, the expected value is just one measure of central tendency. Rather than thinking of the average height, instead we can think of the most common height. Alternatively, we might be interested in the height which is right in the middle; 50% of students are taller, 50% shorter. You may have seen these concepts before when analysing data sets, but they extend to probability distributions.

We begin with the most common outcome, the mode.

The $\arg \max$ stands for “argument max", the value of $x$ that maximises $f(x)$; you have probably not seen this notation before. If we just had $\max$, it would be the actual maximum probability rather than the $x$ value.

If we plot a probability distribution, the mode will be the $x$ value corresponding to the highest point in the graph. To calculate the mode, we use techniques from calculus to find maximum points. In a game, the mode is the most likely outcome.

The median of a dataset is the value right in the middle, with half the possible outcomes on either side.

So, the median is the value with a probability of 0.5 of being lower (and thus higher) than it. In a game, it would be the score that half the time you beat, half the time you don’t.

In a plot of the probability distribution, the median will be the $x$ which has half the area under the graph on either side of it. To find the median, we solve the algebraic equation $P(X \leq Median(X)) = 0.5$.

Here we see a skewed distribution with the mean, median and mode labelled. The mode is the highest point in the graph. The median is moved slightly to the right, allowing for half the area under the curve to be either side of it. The mean is even further to the right than the median, as it depends not just on the probabilities but also the values; bigger values will increase the mean but not the median.

Variance

Knowing what happens on average can only tell us so much. Knowing how far the outcomes can be from the average is also valuable.

Suppose I record the heights of two (very large) classes of students and they both have the same mean. In Class 1, most students are pretty close to the expectation. In Class 2, there is a large spread to the heights; most students are in the middle but some are short and some are tall. Notice below how the increased spread decreases the height (of the curve, not the students).

Suppose instead I am deciding between two stocks to invest in, both with the same price. Stock 1 has major fluctuations in value while stock 2 is pretty consistent. Some way of measuring the volatility and comparing the two is very useful.

We want some way to describe the spread of the data. Well, think about the average distance from the mean; a large average distance would mean a large spread. So, we calculate $X - E[X]$, the deviation from the mean. However, we generally square distances when calculating, and so we take the squared deviation from the mean, $(X-E[X])^2$. Then, we want the weighted average so take expectation. This gives the variance.

Note: $E[X]$ is a constant in the above formula, so $g(x) = (x-E[X])^2$ for LOTUS.

The larger the variance, the more likely outcomes far from $E[X]$ are to occur. A share price with high variance is risky, with a chance of making a big return but also a big loss. A share price with low variance is more predictable and probably give a steady but unspectacular return.

Similar qualities to linearity of expectation exist for variance. For $a, b \in \mathbb{R}$: $$\begin{aligned} Var(aX + b) = a^2Var(X) \end{aligned}$$

As the variance of a constant is zero, adding it to $X$ has no effect on variance. Think about subtracting 5cm from the height of every student to account for the machine used; it won’t change the actual variability, just the values.

However, multiplying $X$ by $a$ will increase the variance by a factor of $a^2$; this makes sense given the power of 2 in $Var(X)$.

We often use a different formula to calculate the variance: $$\begin{aligned} Var(X) = E[X^2]-(E[X])^2 \end{aligned}$$

Generally, we know $E[X]$ and it is easier to calculate $E[X^2]$ than $E[(X-E[X])^2]$.

Suppose we have two random variables $X$ and $Y$. Then, if they are independent, we have: $$\begin{aligned} Var(X+Y) = Var(X)+Var(Y) \end{aligned}$$ If they are dependent in some way, there is an extra covariance term which we will not go into.

This gives us a general formula for $X$ and $Y$ independent and $a, b \in \mathbb{R}$: $$\begin{aligned} Var(aX + bY) = a^2Var(X) + b^2Var(Y) \end{aligned}$$

Note: Even if we are looking at $X-Y$ (so $a =1, \ b = -1$), the variance will increase as $b^2$ is positive. If we include a new random variable, it can only get more variable.

Now try the following quick question; the first statistics question from 2019 Further Mathematics A Level.

The following is a modified version of Q4 in the 2020 Further Maths paper.

Key Takeaways:

• The expected value $E[X]$ of a random variable is a weighted average

• The mode is the value with highest probability

• The median is the value that has a probability of 0.5 either side of it

• The variance measures the spread of the distribution

• $E[aX + b] = aE[X]+b$, $Var(aX+b) = a^2Var(X)$

• $E[X+Y] = E[X]+E[Y]$, $Var(X+Y) = Var(X)+Var(Y)$

Specific Distributions

So far, we have seen examples of experiments and random variables, and talked about distributions. However, we have not discussed specific distributions for those examples.

In this section I introduce several such distributions. If you studied Further Maths for A-levels you may have seen them all. If you studied Maths, you may have seen just the first two. Nonetheless, no matter what your mathematical journey until now, it is ok to not have seen (or remember) some of these. I will recap them now as they are necessary to solve some of the sample exam questions but they will be covered again in detail once you get to Warwick.

Binomial Distribution

Many experiments have two possible outcomes; success and failure. This could be answering a question on an exam, playing a tennis match, buying a car at an auction. Any experiment with two possible outcomes is called a Bernoulli Trial, where $p$ is the probability of success.

Flipping a coin is the classic example of a Bernoulli trial. Let $X$ be the random variable where $H = 1$, a success, and $T=0$, a failure. For a fair coin, $p=\tfrac{1}{2}$, and $X$ is a Bernoulli random variable.

However, often we are more interested in a series of Bernoulli trials. Rather than flipping a coin once, we flip it ten times and count the heads. Rather than answering one question we answer a whole exam and check the final score. Rather than playing one tennis match we play a whole tournament. If each of these is a Bernoulli random variable and $1$ is a success, we add all values of the random variable together to count the number of successes. Then, we can find the probability of a particular number of success (5 heads out of 10, 40% final score, 10 wins out of 12 matches).

The sum of Bernoulli random variables is called a Binomial random variable, under two conditions. First, the number of trials, $n$, must be fixed. We need to know how many coins will be flipped or how many questions are on the exam. The tennis fans among you will know that usually tournaments are knockout; you play until you lose. In this case, $n$ is not fixed and a Binomial is not a suitable distribution.

Second, the probability $p$ is fixed between trials. In other words, each trial is independent of the previous. If we were to draw a tree diagram, the edge probabilities would always be the same. In the coin flip example, flipping a heads should have no effect on the rest of the flips. In the exam, however, maybe the questions get more difficult as you go through, and so $p$ actually decreases with each trial; the Binomial would not be appropriate.

The Binomial is a discrete random variable, as Range( $X$) $= \{0,1,\dots,n\}$. The probability of recording $k$ successes in $n$ trials is given by the pmf: $$\begin{aligned} P(X = k) = {n \choose k} p^n (1-p)^{n-k}, \ k \in \{0,1,\dots,n\} \end{aligned}$$ where ${n \choose k} = \tfrac{n!}{k!(n-k)!}$ is the Binomial coefficient.

Note: We don’t use the $p(x)$ notation for the pmf here so as not to get confused with the probability of success $p$.

For a $B(n,p)$ distribution, we have:

• $E[X] = np$

• $Var(X) = np(1-p)$

The expectation is simply the number of trials multiplied by probability of success. The variance is maximised (check yourself) when $p=\tfrac{1}{2}$; success and failure being equally likely gives most uncertainty.

As mentioned in the Tree Diagrams refresher, any discrete random variable can be transformed into a Bernoulli by picking a set of outcomes as success and the rest as failure. When rolling a die, let even numbers be success and odd be failure. When spinning a roulette wheel, let our chosen numbers be success and the rest be failure. In a football match, let winning be a success and losing or drawing be a failure. This make the Binomial distribution extremely flexible once the conditions on $n$ and $p$ are satisfied.

The following is a modified version of Q15 on the 2024 A level mathematics paper 3.

Normal Distribution

The Binomial is the most common discrete random variable you may have encountered up until now. When dealing with continuous data, the Normal distribution is most common.

When measuring heights of students in a class, a reasonable first assumption to make on the distribution is symmetry; a student is as likely to be tall as they are short. Similarly, assume a share price is just as likely to increase as decrease. For a symmetric distribution, the mean and median are equal.

A second assumption we might make is unimodality; there is only one mode, a single peak in the curve. This assumes we have only one height in the class that is most common, or one share price that is most likely. For a symmetric distribution, the mean and mode (and thus median) are equal.

The cdf of the Normal Distribution is not available in a nice, easy form, and so is instead written as an integral of the pdf. The graph of the pdf for multiple values of $\mu$ and $\sigma^2$ is below; $\mu$ dictates location and $\sigma$ dictates the width/spread of the graph.

The shape of the distribution is a bell curve. Often, the phrases “Normal" and”bell curve" are used interchangeably. However, other distributions have bell curve shaped graphs too, so use Normal or Gaussian to be specific.

The mean $\mu$ tells us where the center of the bell curve is, while the standard deviation $\sigma$ describes its width. A large $\sigma$ makes it more likely to observe extreme values and thus less likely to observe values close to $\mu$. We see how a bigger $\sigma$ (the green curve) leads to a more “squashed" graph. As the area under the curve is fixed to 1, increasing the width decreases the height.

The Normal is continuous with $\text{Range}(X) = \mathbb{R}$; technically, any real number is possible. However, as we get further from the expectation, the probability decreases to the point where it is essentially impossible. The larger the variance, the larger this range of reasonable values.

The standard deviation leads to the following empirical rule:

• c.68% of the distribution is within one standard deviation of the mean

• c.95% of the distribution is within two standard deviations of the mean

• c.99.7% of the distribution is within three standard deviations of the mean

This is why hypothesis tests often use a 95% significance level; it is two standard deviations either side.

We can assume $X$ is a Normal random variable, but until we record real data it is impossible to know if this is appropriate. A histogram of the data checks whether the requirements are met. Nonetheless, the Normal is a good default choice for a distribution due to its nice properties (which we will see) and ubiquity in nature. I often think of dropping marbles to form Pascal’s triangle; most will settle into the middle but some will spread out evenly.

The symmetry of the Normal distribution gives us the following property: $$\begin{aligned} P(X \leq -a) = P(X \geq a) = 1-P(X\leq a) \end{aligned}$$ Thus, complex probability statements can be generated from simply knowing $P(X \leq a)$.

The Standard Normal Distribution

The ability to break down complex statements is important because we cannot directly integrate the pdf. Instead, we use probabilities already given to us (such as in log tables). However, it is impossible to have these probabilities available for each possible $\mu$ and $\sigma$, so we use a reference.

Rather than thinking of the heights of students, think of their difference from the average height. Then negative values will indicate shorter students and so on. For a share price, think of the change from some starting price instead.

Note: We usually represent the Standard Normal with a $Z$ but $Z$ can also be used for any Normal. Sometimes, $\phi$ is used for the Standard Normal and $\Phi$ for its cdf.

To get from any Normal to the Standard Normal, we standardise.

So, to standardise any Normal random variable $X$ we:

• Shift by subtracting the expected value

• Scale by dividing by the standard deviation

Using the previous properties to help intuition: $$\begin{aligned} E[\tfrac{X - \mu}{\sigma}] &= \tfrac{1}{\sigma}(E[X]-\mu) = 0\\ Var(\tfrac{X - \mu}{\sigma}) &= \tfrac{1}{\sigma^2} Var(X - \mu) = \tfrac{1}{\sigma^2} Var(X) = 1 \end{aligned}$$

Standardising the Normal is very useful because we only need the specific probabilities for this one distribution; everything else can be derived from them.

The Normal Approximation

The Normal distribution is widely studied not just because of its simplicity and ubiquity, but also because it has a very useful property for large sample sizes.

Suppose $X$ counts how many twos you get in $1000$ rolls of a fair 6-sided die, such that $X \sim Bin(1000,\tfrac{1}{6})$. We can easily find $P(X = 50)$. However, getting $P(20 \leq X \leq 100)$ would require individually calculating the probability of all 81 numbers in the set. There is a quicker way.

By approximating the number of twos with a Normal distribution, we find $P(20 \leq X \leq 100)$ through two calculations, $P(X \leq 100)$ and $P(X \leq 20)$, rather than 81.

Note: It is often recommended to perform a continuity correction when approximating a discrete random variable with the Normal. This entails adding a small number, usually $0.5$, to the value of interest to account for the starting point of the interval. I will not do this in my solutions for simplicity.

The Binomial is the most common distribution to approximate. It becomes more symmetric as $n$ grows and it is tedious to sum probabilities. Thus, when $n$ is large, the Normal approximation of the Binomial is appropriate.

Note: It is common to make statements about “large" $n$, but often what constitutes large is not given or justified. What it actually means is that it is true for the limit as $n \rightarrow \infty$, and so we choose an $n$ along the way to get close. The larger $n$ is, the better the approximation.

The reasons for the Normal approximation are rooted in the Central Limit Theorem and Law of Large Numbers; you will explore these in great detail but these are core properties as to why the Normal distribution is so popular.

The following is a modified version of Q17 from the 2024 A level Maths Paper 3.

Note: Part (d) references 95% but this is not 2 standard deviations away like the empirical rule. There is a difference between the number with 95% above it, as in (d), and the 2 numbers with 95% between them. Those two numbers have 2.5% either side.

A Quick Note

The following three distributions were not covered in A level Maths but may have been in Further Maths. They will be used to solve questions from Further Maths or STEP later but don’t panic if this is your first time seeing them.

Uniform Distribution

The simplest form of any probability distribution is to assume each outcome is equally likely. When we roll a fair die or flip a fair coin, we have no reason to believe any outcome is more likely than another; we assume the probabilities are uniform.

For a discrete random variable with a finite range, we get the pmf by assigning each outcome the same probability of 1 divided by the number of outcomes. An example would be flipping a fair coin or rolling a fair die; the word fair suggests uniformity. However, we cannot define the pmf this way for a range such as $\mathbb{Z}$; think about how you could prove this.

Nonetheless, we still define the uniform distribution for a continuous random variable. Suppose Range( $X$) = $[a,b]$. Then, we assume each number in the interval is equally likely and we get the pdf to be: $$\begin{aligned} f(x) = \begin{cases} \tfrac{1}{b-a}, \ x \in [a,b],\\ 0, \text{otherwise} \end{cases} \end{aligned}$$ The cdf is then defined as: $$\begin{aligned} F(x) = \begin{cases} 0, \ x < a,\\ \frac{x-a}{b-a}, \ x \in [a,b],\\ 1, x > b \end{cases} \end{aligned}$$

We write $X \sim U[a,b]$, with:

• $E[X] = \tfrac{a+b}{2}$,

• $Var(X) = \tfrac{(b-a)^{2}}{12}$

As we might expect, the expectation is the average of the two end points, so right in the middle.

The Uniform distribution is often called the rectangular distribution because of how the graph of the pdf looks

We see how it is a horizontal line between $a$ and $b$, and then zero elsewhere. The area under the curve being 1 is also clear: the base is $b-a$ and the height is $\tfrac{1}{b-a}$

A random number generator is a common example of a continuous uniform distribution. The following was Q6 in the 2023 Further Maths paper.

Question 4 was originally presented in the form of a discrete uniform distribution rather than as an n-sided die; they are equivalent.

Poisson Distribution

Often, we don’t have a fixed number of trials but instead a fixed interval of time, and we wish to count the number of events that happen in that interval. The Poisson distribution is applicable in these cases.

Rather than considering whether United win or lose a match, suppose we want to predict how many goals they score in a game. There is no fixed number of trials; the Binomial can model how many times in 10 games they score more than 2 goals, for example, but not how likely a number of goals is in one game.

Instead, we consider the average number of goals they score in a game and use that to find the probability. This is the Poisson and requires the following assumptions.

As there is no fixed number of trials, there is no upper limit to the amount of events that can happen. In theory, United could score any amount of goals, although physical (and tactical) limitations prevent that in reality.

The average rate of goals should be assumed consistent within the interval. If United score 1.5 goals on average. this rate should be the same no matter what point in the game we are at. This is often a simplification; it is more common to score goals near the end of a game than the start.

Finally, the time taken for a goal should be independent of when the last goal was scored. It doesn’t matter if the last goal was 5 minutes or 50 minutes ago, the probability I score now is the same.

Essentially, the Poisson is applicable when we have an average rate of occurrence, each occurrence has no effect on the others, and there is no maximum that can happen.

For a $Poi(\lambda)$ distribution, we have:

• $E[X] = \lambda$

• $Var(X) = \lambda$

So for a Poisson distribution the mean and variance are equal.

Note: $X$ is not just a count, but a count over a fixed interval. It is the average number of goals scored over a 90 minute game. If the rate is given for one interval but we are interested in another, we can scale the rate $\lambda$. For example, we can divide $\lambda$ by $90$ to get the goals per minute and go from there. It is important to make sure our rate matches the interval.

Note: The interval of interest for the Poisson is generally time, but it could be something like length (such as accidents over a stretch of road) or area (such as trees in a forest). In linear modeling courses you will encounter such cases.

There is a link between the Poisson and Binomial.

In fact, when $n$ is large and $p$ is small (so $1-p$ is close to 1 and $np(1-p) \approx np$), a $Poi(np)$ distribution is a good approximation of the Binomial distribution. I like to think of the Poisson as the Binomial with an infinite number of trials; in each tiny interval of time you are checking whether the event happens or not. So when $n$ is large they are similar.

So when $n$ is large, the Poisson and Normal are both good approximations of the Binomial. As you might expect, this means the Normal is a good approximation of the Poisson as well when $\lambda = np$ is large.

Exponential Distribution

We saw the Poisson distribution as a way of counting how many events occur in a fixed interval. An alternative but equivalent way of thinking about this problem is as the time between events.

The number of customers entering a queue is often modeled with a Poisson. Suppose we know in one hour period that 2 people entered. Instead, I could say that 15 minutes in, one person came. Then, 35 minutes after that, another person came. 20 minutes after that a third person arrived. This third person came after the hour and so we know only two people came in the first hour.

Note: Just as with the Poisson, the rate parameter is the rate per unit time. It is common to transition between the Exponential and Poisson distributions and you should be careful with what $\lambda$ represents.

The Exponential distribution has a property similar to the Poisson.

That is, it doesn’t matter how long has passed to the current point, we only need to look from now onwards.

We have for $T \sim \exp(\lambda)$:

• $E[T] = \tfrac{1}{\lambda}$

• $Var(T) = \tfrac{1}{\lambda^2}$

The result for expectation should make sense when we think of the link to the Poisson; if we expect $\lambda$ events to occur in one unit of time and the rate is constant, then we should expect the average time between each event to be $\tfrac{1}{\lambda}$.

The Exponential distribution is very important when measuring times between events, such as in risk analysis for insurance companies.
Key Takeaways:

• The Binomial distribution is for counting the number of successes in $n$ independent Bernoulli trials

• The Normal distribution is common, simple and a good approximation of some discrete distributions

• The Uniform distribution assumes each outcome is equally likely

• The Poisson distribution counts the number of events in a fixed interval of time

• The Exponential distribution measures the time between events, rather than counting them over time like the Poisson

Worked Questions

The following was Q8 in 2019 Further Maths Paper 3.

The following is Q11 in STEP II 2023.