Errors in hypothesis tests and confidence intervals AI HL

A hypothesis test can go wrong in two ways: rejecting a null hypothesis that is true (a Type I error) or keeping one that is false (a Type II error), and you must be able to describe each in context and find its probability. A confidence interval gives a range of plausible values for a mean, found with the calculator from raw data or from summary values, using the normal distribution when the standard deviation is known and the t-distribution when it is not. The first section deals with the errors and the second with intervals, and each has a worked example before the questions.

Formulas at a glance

The formula booklet has no result for either topic. The error probabilities, the decision rule and the two interval formulas are not in it, so learn them; the calculator builds the interval for you.

Probability of a Type I error

P(Type I)=αP(\text{Type I})=\alpha, the significance level

Not in the booklet

Probability of a Type II error, and power

β=P(do not reject H0∣H0 false)\beta=P(\text{do not reject }H_0\mid H_0\text{ false}), power =1−β=1-\beta

Not in the booklet

Decision rule with a p-value

Reject H0H_0 if p<αp<\alpha

Not in the booklet

Mean of a sample, normal data with known standard deviation

X¯∼N(μ,σ2n)\bar X\sim N\!\left(\mu,\dfrac{\sigma^2}{n}\right)

Not in the booklet

Confidence interval for the mean, standard deviation σ\sigma known

x¯±z∗σn\bar x\pm z^{*}\dfrac{\sigma}{\sqrt n}

Not in the booklet

Confidence interval for the mean, standard deviation σ\sigma unknown

x¯±t∗sn\bar x\pm t^{*}\dfrac{s}{\sqrt n}, with n−1n-1 degrees of freedom

Not in the booklet

Which method to use

Decide first whether the question is about a decision and its errors or about estimating a mean. For an error question, say which hypothesis is true in reality before choosing the distribution to use.

The question hasUseNotes
A description of a Type I or a Type II errorType I: reject H0H_0 when it is true. Type II: do not reject H0H_0 when it is falseWrite it in the words of the question, not only with the symbol H0H_0
A p-value and a significance levelReject H0H_0 if p<αp<\alpha, otherwise do not rejectThe probability of a Type I error is α\alpha, so a larger level makes it more likely
The probability of a Type I error for a stated critical regionAssume H0H_0 is true and find the probability that the test statistic falls in the critical regionFor a discrete test this can be less than the stated significance level
The probability of a Type II error when the true mean is givenUse the distribution with the true mean and find the probability of landing outside the critical regionPower is 1−β1-\beta; a larger sample lowers β\beta
A confidence interval, and the standard deviation σ\sigma is givenThe zz-interval x¯±z∗σn\bar x\pm z^{*}\dfrac{\sigma}{\sqrt n}, or the calculator's Z intervalUse the standard deviation given, whatever the size of the sample
A confidence interval, and σ\sigma is not given, so ss comes from the dataThe calculator's T interval with the data list, or with x¯\bar x, ss and nnUse a tt interval whatever the size of the sample
An interval given, and the mean or the margin of error wantedThe mean is the midpoint and the margin of error is half the widthThe two ends are symmetrical about x¯\bar x
A request to say what an interval means, or to compare two levelsA sentence in context, and a higher level gives a wider intervalAbout 95% of intervals built this way contain the true mean; it does not say where individual values lie

Section 1 of 2 AHL4.18

Type I and Type II errors

A Type I error is rejecting H0H_0 when it is true, and a Type II error is failing to reject H0H_0 when it is false. The probability of a Type I error is the significance level α\alpha, and the probability of a Type II error is written β\beta. Questions ask you to describe these errors in context and to say how changing the test affects them. To find the probability of a Type I error, assume H0H_0 is true and find the probability that the test statistic falls in the critical region. To find the probability of a Type II error, use the true value of the parameter instead, and find the probability that the test statistic falls outside the critical region. Given a p-value, reject H0H_0 if p<αp<\alpha and do not reject it otherwise. For a discrete variable, such as a binomial or Poisson count, the critical region is the largest one whose probability under H0H_0 is less than the stated significance level, so the probability of a Type I error can be less than α\alpha.

Key terms

What is the significance level?

The significance level α\alpha is the probability of rejecting H0H_0 when it is true. It is chosen before the test and gives the probability of a Type I error.

e.g. A test at the 2% level has α=0.02\alpha=0.02, so the probability of rejecting a true H0H_0 is 0.020.02.

How do you describe an error in context?

Say what was concluded and what was actually true, using the words of the question rather than only the symbol H0H_0.

e.g. For H0:μ=35H_0:\mu=35, a Type I error is concluding that the mean is not 3535 when it is in fact 3535.

What is the critical region?

The critical region is the set of sample results for which H0H_0 is rejected. It is chosen so that, when H0H_0 is true, the probability of landing in it is no more than α\alpha.

e.g. At the 2% level, the critical region has a probability of at most 0.020.02 when H0H_0 is true.

What is the power of a test?

The power is 1−β1-\beta, the probability of correctly rejecting H0H_0 when it is false. A higher power means the test is better at detecting a real difference.

e.g. If β=0.2\beta=0.2, the power is 1−0.2=0.81-0.2=0.8.

Worked example, step by step

The question Calculator needed [5 marks]

The lengths of metal rods made by a machine are normally distributed with standard deviation 0.6 cm. The mean length is claimed to be 20 cm. A manager suspects that the mean has increased, and tests H0:μ=20H_0:\mu=20 against H1:μ>20H_1:\mu>20, where μ\mu cm is the mean length. A random sample of 9 rods is taken, and H0H_0 is rejected if the sample mean X¯\bar X is greater than 20.4 cm.

(a) Find the probability of a Type I error.

(b) The mean length has in fact increased to 20.5 cm. Find the probability of a Type II error.

(c) Explain what a Type II error means in this context.

Worked solution: worked example

Answer: (a) 0.02280.0228 (b) 0.3090.309 (c) concluding the mean length has not increased when it has

Step 1 - Find the distribution of the sample mean when H0H_0 is true.

A Type I error is rejecting H0H_0 when it is true, so assume μ=20\mu=20. The standard deviation of X¯\bar X is 0.69=0.2\dfrac{0.6}{\sqrt9}=0.2, so X¯∼N(20,0.22),\bar X\sim N(20,\,0.2^2), and the probability needed is P(X¯>20.4)P(\bar X>20.4). M1 - Use X¯∼N(20,0.22)\bar X\sim N(20,\,0.2^2) with P(X¯>20.4)P(\bar X>20.4)

Why? The error probability is found assuming the true situation that makes the error possible. For a Type I error that is H0H_0, which is why the mean is 20.

Step 2 - Use the calculator.

Use the normal cumulative distribution function with lower bound 20.4, upper bound 109910^{99} (a very large number), mean 20 and standard deviation 0.2. The calculator shows P(Type I error)=0.0228.P(\text{Type I error})=0.0228. A1 - 0.02280.0228

Why? This is the significance level of the test as it is set up. A critical value of 20.4 is two standard deviations above the mean, which gives roughly 2.3% in the upper tail.

Step 3 - Find the distribution when the true mean is 20.5.

A Type II error is not rejecting H0H_0 when it is false, so now μ=20.5\mu=20.5 and X¯∼N(20.5,0.22).\bar X\sim N(20.5,\,0.2^2). Not rejecting means X¯≤20.4\bar X\le20.4, so the probability needed is P(X¯≤20.4)P(\bar X\le20.4). M1 - Use X¯∼N(20.5,0.22)\bar X\sim N(20.5,\,0.2^2) with P(X¯≤20.4)P(\bar X\le20.4)

Why? The standard deviation of X¯\bar X stays the same, because it depends only on σ\sigma and nn. Only the centre of the distribution moves.

Step 4 - Use the calculator.

Use the normal cumulative distribution function with lower bound −1099-10^{99}, upper bound 20.4, mean 20.5 and standard deviation 0.2. The calculator shows P(Type II error)=0.309.P(\text{Type II error})=0.309. A1 - 0.3090.309

Why? The value 20.4 is half a standard deviation below the true mean, so about 31% of samples fall in the "do not reject" region even though the mean has increased.

Step 5 - Describe the error in context.

A Type II error means concluding that there is no evidence the mean length has increased, when the mean length has in fact increased. R1 - Concluding the mean length has not increased when it has

Why? The answer must say what was concluded and what was actually true, in the words of the question, not only "failing to reject H0H_0 when it is false".

Marks (5):
M1 Use X¯∼N(20,0.22)\bar X\sim N(20,\,0.2^2) with P(X¯>20.4)P(\bar X>20.4)
A1 0.02280.0228
M1 Use X¯∼N(20.5,0.22)\bar X\sim N(20.5,\,0.2^2) with P(X¯≤20.4)P(\bar X\le20.4)
A1 0.3090.309
R1 Concluding the mean length has not increased when it has

Now you try: enter your answers to be marked

These are real practice questions. Type your answer in each box and press Mark it. The worked solution and mark scheme appear straight away, and your first attempt is the one that counts. Use the keyboard for powers, roots and trig, and the working box for your notes.

1
Easy
No calc
[2 marks]
In a hypothesis test at significance level α=0.05\alpha = 0.05,
(a) Define a Type I error.
[1]
(b) State the probability of a Type I error.
[1]
Fractions & powers
Other

Worked solution: practice question 1

2
Medium
No calc
[2 marks]
A two-tailed test is conducted at the 10% significance level.
(a)(i) State the probability of a Type I error (as a percentage).
[1]
(a)(ii) State the percentage in each tail.
[1]
Fractions & powers
Other

Worked solution: practice question 2

3
Medium
No calc
[3 marks]
A test gives a p-value of 0.03.
(a) State the conclusion at the 5% level.
[1]
(b) State the conclusion at the 1% level.
[1]
(c) At which level is a Type I error more likely?
[1]
Fractions & powers
Other

Worked solution: practice question 3

Your score so far: 0 out of 7 marks

Common mistakes

  • Defining the errors in the abstract instead of in context. "Rejecting a true null hypothesis" alone often loses the mark - the answer should say what that means for the actual scenario, e.g. "concluding the mean has changed when it hasn't."
  • Confusing which error the significance level controls. α\alpha is set before the data is collected, and for a normal test statistic it is the probability of a Type I error. For a binomial or Poisson count, the probability of a Type I error is the probability of the critical region, which can be less than α\alpha.
  • Assuming lowering α\alpha always improves the test. A smaller significance level reduces the Type I error rate but increases the Type II error rate (lower power) - the two trade off against each other unless the sample size also increases.

Try some questions on type I and Type II errors

Section 2 of 2 AHL4.16

Confidence intervals

A confidence interval for the mean is a range, calculated from a sample, that is likely to contain the true mean μ\mu. It has the form x¯±margin of error\bar{x}\pm\text{margin of error}, built with the normal distribution when σ\sigma is known and with the tt-distribution when it is not. The result must be interpreted in context.

Choosing your interval

Choose the interval from what the question gives you, then interpret the result in context.

z-interval vs t-interval

x¯±z∗σn\bar{x} \pm z^* \dfrac{\sigma}{\sqrt{n}}  or  x¯±t∗sn\bar{x} \pm t^* \dfrac{s}{\sqrt{n}}

These aren't given as algebra in the formula booklet - your calculator builds them directly from raw or summary data, so the real skill is spotting which routine to use. If the question gives the standard deviation σ\sigma, use a zz-interval; if the standard deviation has to be found from the data, as ss, use a tt-interval, however large nn is.

Confidence level

90% → 95% → 99%

A higher confidence level needs a larger critical value, which widens the interval.

Key terms

What is the margin of error?

The margin of error is the distance from the sample mean x¯\bar{x} to each end of the interval, so the interval is x¯\bar{x} plus or minus this amount.

e.g. An interval (46.2,53.8)(46.2,\,53.8) has centre 46.2+53.82=50\dfrac{46.2+53.8}{2}=50 and margin of error 53.8−50=3.853.8-50=3.8.

What does 95% confidence mean?

If many samples were taken and an interval built from each, about 95% of those intervals would contain the true mean. It does not describe where individual data values lie.

e.g. Out of 100100 intervals built from 100100 samples, about 9595 would be expected to contain μ\mu.

How does sample size affect the interval?

A larger sample size nn reduces the spread of the sample mean, so the interval is narrower and the estimate more precise.

e.g. With σ\sigma fixed, the margin of error is proportional to 1n\dfrac{1}{\sqrt{n}}, so increasing nn from 2525 to 100100 halves it.

What is the critical value?

The critical value is the multiplier of σn\dfrac{\sigma}{\sqrt{n}} (or the equivalent for the tt-distribution) needed for the chosen confidence level. For a 95% interval using the normal distribution it is about 1.961.96.

e.g. With σ=12\sigma=12 and n=36n=36, the margin of error is 1.96×1236=1.96×2=3.921.96\times\dfrac{12}{\sqrt{36}}=1.96\times2=3.92.

Worked example, step by step

The question Calculator needed [4 marks]

The times, in minutes, taken by 8 volunteers to complete a puzzle were

12.4, 15.1, 13.8, 14.6, 12.9, 15.5, 13.2, 14.0.

The times may be assumed to come from a normal distribution with unknown standard deviation.

(a) Find a 95% confidence interval for the mean time to complete the puzzle.

(b) Interpret the interval in context.

Worked solution: worked example

Answer: (a) (13.0, 14.8)(13.0,\ 14.8) minutes (b) 95% confident that the mean time lies in this interval

Step 1 - Choose the interval.

The standard deviation of the times is not given, so it is estimated from the data and a tt-interval is used. With 8 values there are 8−1=78-1=7 degrees of freedom.

Why? A zz-interval needs a known standard deviation. When the standard deviation is unknown the tt-distribution is used whatever the size of the sample.

Step 2 - Enter the data and run the interval.

Enter the 8 times in a list. Choose the tt-interval routine with the data option, the list just entered and confidence level 0.95. The calculator finds the mean x¯=13.9375\bar x=13.9375 and the standard deviation s=1.085…s=1.085\ldots itself. M1 - Use a tt-interval with the data and 95% confidence

Why? Entering the raw data avoids rounding the mean and standard deviation before they are used. The calculator uses the divisor n−1n-1 for ss, which is the value a tt-interval needs.

Step 3 - Read the interval.

The calculator shows the interval from 13.03 to 14.84, so to three significant figures (13.0, 14.8) minutes.(13.0,\ 14.8)\text{ minutes}. A1 - Lower end 13.013.0 A1 - Upper end 14.814.8

Why? The interval is x¯±t∗sn=13.94±2.36×1.0858\bar x\pm t^{*}\dfrac{s}{\sqrt n}=13.94\pm2.36\times\dfrac{1.085}{\sqrt8}, a margin of error of about 0.907 either side of the mean.

Step 4 - Interpret the interval.

We are 95% confident that the mean time to complete the puzzle lies between 13.0 minutes and 14.8 minutes. If many samples of 8 volunteers were taken and an interval found from each, about 95% of the intervals would contain the true mean time. R1 - 95% confident the mean time lies between 13.0 and 14.8 minutes

Why? The interval is about the mean, not about individual volunteers: it does not say that 95% of volunteers finish within these times.

Marks (4):
M1 Use a tt-interval with the data and 95% confidence
A1 Lower end 13.013.0
A1 Upper end 14.814.8
R1 95% confident the mean time lies between 13.0 and 14.8 minutes

Now you try: enter your answers to be marked

1
Medium
Calculator
[3 marks]
A sample of n=25n=25 has mean x¯=50\bar{x}=50 and standard deviation sn−1=4.s_{n-1}=4. Find a 95% confidence interval for the population mean.
[3]
Fractions & powers
Other

Worked solution: practice question 1

2
Medium
Calculator
[5 marks]
The values are known to have standard deviation σ=10\sigma = 10. A sample of n=64n = 64 gives x¯=75\bar{x} = 75. Construct a 95% confidence interval using the zz-interval formula x¯±z∗σn\bar{x} \pm z^* \dfrac{\sigma}{\sqrt{n}}, where z∗=1.96.z^* = 1.96.
[5]
Fractions & powers
Other

Worked solution: practice question 2

3
Medium
Calculator
[3 marks]
The masses (g) of 6 items are 21, 23, 20, 24, 22, 22.

Find a 95% confidence interval for the mean mass.
[3]
Fractions & powers
Other

Worked solution: practice question 3

Your score so far: 0 out of 11 marks

Common mistakes

  • Reaching for zz by default. Unless the question explicitly gives you the standard deviation σ\sigma, you should be using a tt-interval with ss, found from the data - it does not matter how large nn is.
  • Assuming a higher confidence level is always "better". A 99% interval is wider, not more accurate - it trades precision for a greater chance of containing the true mean, which is worth stating explicitly if a question asks you to compare levels.
  • Losing the units or context in the final answer. A confidence interval is only fully answered once you've stated it in context - "the mean height lies between 171.2 cm and 174.8 cm" - not just as a bare pair of numbers.

Try some questions on confidence intervals

Using your GDC

The exact key presses for the TI-84 Plus CE, Casio fx-CG50, Casio fx-CG100 and TI-Nspire CX, with what the screen shows. Pick your calculator.

Interpret p-values, significance levels and errors

Used in: Type I and Type II errors

How to read a p-value on a TI-84 Plus CE (chi-squared goodness-of-fit test)

Run the test, then compare p with the significance level

Press stat and choose 1:Edit.... Type the observed counts in L1 and the expected counts in L2, pressing enter after each.

L1    L2
 28   20
 18   20
 14   20

Press stat, press ▶ twice to reach TESTS and choose D:χ²GOF-Test.... With the Stat Wizards on, set Observed to L1 (2nd 1), Expected to L2 (2nd 2) and df to 2, then choose Calculate.

χ²GOF-Test
Observed:L1
Expected:L2
df:2
Calculate

The screen shows χ², p and df. The value after p= is the p-value. Compare it with the significance level: if p is smaller, reject H0H_0; if it is larger, do not.

χ2=5.2\chi^2 = 5.2 and p=0.0743p = 0.0743 to 3 significant figures. Since 0.0743>0.050.0743 > 0.05, do not reject H0H_0.

χ²GOF-Test
χ²=5.2
p=0.0742736
df=2
CNTRB={3.2,0.2,1.8}

The calculator never says reject or do not reject. It gives only p. You compare p with the significance level yourself and write the conclusion.

The full guide: Interpret p-values, significance levels and errors

Normal distribution probabilities

Used in: Type I and Type II errors

How to find a normal probability on a TI-84 Plus CE

normalcdf (between two bounds)

Press 2nd vars (DISTR) and choose 2:normalcdf(. With the Stat Wizards on, a screen asks for each value in turn: type it and press enter each time.

Enter the lower bound 140, the upper bound 165, the mean 150 and the standard deviation 12, then press enter on Paste.

normalcdf(
lower:140
upper:165
μ:150
σ:12

Press enter to run it. If the Stat Wizards are off, type normalcdf(140,165,150,12) with commas between the four values and press enter.

P(140<X<165)=0.692P(140 < X < 165) = 0.692 to 3 significant figures.

normalcdf(140,165,150,12)
0.6920218

For more than 170, there is no upper limit, so use a very large number as the upper bound: type 1 2nd , (EE) 9 9 for 1E99.

P(X>170)=0.0478P(X > 170) = 0.0478 to 3 significant figures.

normalcdf(170,1E99,150,12)
0.0477904

For less than a value, use −1E99 as the lower bound. If you give only the two bounds, the calculator uses mean 0 and standard deviation 1.

The full guide: Normal distribution probabilities

Confidence interval for a mean (t-interval)

Used in: Confidence intervals

How to find a confidence interval for a mean on a TI-84 Plus CE

TInterval (from a list of data)

Press stat and choose 1:Edit. Empty L1 if it holds old numbers (cursor on the name L1, then clear enter), then type the data into L1, pressing enter after each value: 23.1, 25.4, 22.8, 24.9, 26.2, 23.7, 25.0, 24.3, 22.5, 26.1.

L1
23.1
25.4
22.8
24.9
26.2
23.7
25.0
24.3
22.5
26.1

Press stat, then ▶ ▶ to move past CALC to TESTS, and choose 8:TInterval. On the Inpt line choose Data.

TInterval
Inpt:Data Stats
List:L1
Freq:1
C-Level:.95
Calculate

Set List to L1 (2nd 1), leave Freq as 1, and type the confidence level as a decimal: . 9 5 for 95%. Move down to Calculate and press enter.

The first line is the interval as (lower, upper). Below it are the sample mean x̄, the standard deviation Sx and the sample size n.

The 95% confidence interval is (23.4, 25.4)(23.4,\ 25.4), each end to 3 significant figures.

TInterval
(23.44321,25.35679)
x̄=24.4
Sx=1.337494
n=10
TInterval (from summary statistics)

Use this when the question gives the mean, the standard deviation s and the sample size, for example a sample of 10 with mean 24 and standard deviation 3. Press stat, ▶ ▶, choose 8:TInterval, and on the Inpt line choose Stats. Enter x̄ as 24, Sx as 3, n as 10 and C-Level as . 9 5, then choose Calculate and press enter.

The 95% confidence interval is (21.9, 26.1)(21.9,\ 26.1), each end to 3 significant figures.

TInterval
Inpt:Data Stats
x̄:24
Sx:3
n:10
C-Level:.95
Calculate
ZInterval (σ given)

Use a z interval when the question gives the standard deviation σ of the population, for example σ = 4 with a sample of 16 values and mean 62.5. Press stat, ▶ ▶, choose 7:ZInterval, and on the Inpt line choose Stats. Enter σ as 4, x̄ as 62.5, n as 16 and C-Level as . 9 5, then choose Calculate and press enter.

The 95% confidence interval is (60.5, 64.5), each end to 3 significant figures.

ZInterval
Inpt:Data Stats
σ:4
x̄:62.5
n:16
C-Level:.95
Calculate

On the Stats screen, Sx must be the n - 1 standard deviation s. The σx from 1-Var Stats is a different, smaller number and gives an interval that is too narrow.

The full guide: Confidence interval for a mean (t-interval)

See the full list of GDC guides for every calculator task.

Syllabus content covered

CodeSyllabus content
AHL4.16Confidence intervals for the mean of a normal population. Use of the normal distribution when σ\sigma is known and the tt-distribution when σ\sigma is unknown, regardless of sample size. Students should be able to interpret the meaning of their results in context.
AHL4.18Critical values and critical regions. Type I and II errors including calculations of their probabilities. Applied to normal with known variance, Poisson and binomial distributions. For discrete random variables, hypothesis tests and critical regions will only be required for one-tailed tests. The critical region will maximize the probability of a Type I error while keeping it less than the stated significance level.

Official IB source: Mathematics: applications and interpretation subject brief (PDF, ibo.org).

Quick questions

What is a Type I error?

A Type I error is rejecting H0H_0 when H0H_0 is actually true. For a normal test statistic its probability equals the significance level α\alpha; for a binomial or Poisson count it is the probability of the critical region, which can be less than α\alpha.

When do I use a z-interval instead of a t-interval for the mean?

Use a zz-interval only when the standard deviation σ\sigma is given directly. Use a tt-interval whenever the standard deviation has to be found from the data, as ss, regardless of how large nn is.

How can the chance of a Type II error be reduced?

Increasing the sample size reduces β\beta without changing α\alpha. Raising the significance level also reduces β\beta, but at the cost of a larger chance of a Type I error.

Why does a 99% confidence interval come out wider than a 95% one?

A higher confidence level needs a larger critical value, which widens the margin of error and therefore the interval - you are trading precision for a higher chance of capturing the true mean.

Why can't both errors be made impossible?

A decision based on a random sample always carries some risk. For a fixed sample size, lowering α\alpha shrinks the critical region, which makes β\beta larger.

How do I write a confidence interval as an answer?

Give both end points, as (a,b)(a,\,b) or a<μ<ba<\mu<b, to the accuracy asked, with units. Add a sentence in context saying what the interval means.

How do I find the probability of a Type I error for a given critical region?

Find the probability, assuming H0H_0 is true, that the test statistic falls in the critical region. For a discrete test this can be less than the stated significance level.

How do I find the sample mean from a confidence interval?

Add the two end points and halve: x¯=a+b2\bar{x}=\dfrac{a+b}{2}. The margin of error is half the width, b−a2\dfrac{b-a}{2}.

Why is a t-interval wider than a z-interval for similar data?

The t-distribution has heavier tails to allow for the extra uncertainty from estimating the standard deviation. Its critical value is larger, so the interval is wider.

Ready to practise properly?

Practice questions on every section, marked instantly like the real exam.