In short
Sample size depends on the size of difference you care about and the risk of error you'll accept. Most perfumers' tests are small, and that's fine when reported honestly. A small test can reveal large differences, but it often misses moderate ones, and it can't prove that two things are the same. The arithmetic for a triangle or paired test can be done with a calculator, and this lesson shows you how. You used the triangle test earlier, and now you'll learn how to read it.
What you will learn
- Calculate the chance probability of a triangle or paired result
- Explain power and why a null result proves little
- Report a small sensory result with its uncertainty
Before this lesson: Controls, Randomization and Blinding
Chance in a triangle test
Each person gets three samples, two the same, and must pick the odd one. Guessing gives a one-in-three chance. The probability of exactly k correct out of n by chance comes from the binomial formula:
P(k) = C(n, k) × (1/3)k × (2/3)n−k
The p-value is the chance of k or more correct. For the critical values below, we calculated the probabilities exactly. You can also find such tables in sensory texts and in ISO 4120.
| Assessors | Correct needed | Exact p | Power at 50% correct | Power at 60% | Power at 80% |
|---|---|---|---|---|---|
| 6 | 5 | 0.018 | 0.11 | 0.23 | 0.66 |
| 12 | 8 | 0.019 | 0.19 | 0.44 | 0.93 |
| 18 | 10 | 0.043 | 0.41 | 0.74 | Above 0.99 |
| 30 | 15 | 0.043 | 0.57 | 0.90 | Above 0.99 |
| 60 | 27 | 0.040 | 0.82 | 0.99 | Above 0.99 |
Read the "50%" column: if half of the responses would be correct, a real and modest difference, then a 12-person test finds it significant less than one time in five. Notice, too, that power doesn't always rise smoothly as you add people, since these are whole-number counts. By our calculation, 18 assessors needing 10 correct have more power at 50% (0.41) than 21 assessors needing 12 (0.33).
Paired tests
If people choose between two samples, chance is one in two, and for a preference you'd normally use a two-sided test.
| People | Needed for one side |
|---|---|
| 10 | 9 |
| 20 | 15 |
| 30 | 21 |
| 50 | 33 |
So 13 of 20 preferring A has a two-sided p of about 0.26, which is the example from the consumer research lesson. It's weak evidence at best.
What a p-value isn't
- It isn't the probability that your hypothesis is true.
- It isn't a measure of how large or important a difference is.
- A p of 0.049 and a p of 0.051 are nearly the same evidence.
- Non-significant doesn't mean "no difference". Absence of evidence isn't evidence of absence.
- Testing many things means something will appear significant by chance. Twenty independent tests with no real differences will, on average, give about one result under 0.05.
Testing whether two things are the same
A failure to find a difference doesn't show that a replacement is equivalent. To support a claim of similarity, you'd plan a test with enough power to detect the smallest difference you care about, and specify that in advance. For a small maker, an honest sentence might be: "Three of nine correct, which doesn't show a difference. With nine people we'd have only a one-in-four chance of detecting a moderate difference." See your reformulation work.
Independence and repeats
The formulas assume independent judgments. Ten trials by one person tell you about that person, and don't count as ten people. Repeating tests with the same people needs a different analysis. Say what you did, and don't report repeated trials as a larger panel.
Reporting
Invented example: In a triangle test on strips at 2 hours, with 12 adults from a local group, 8 chose the odd sample correctly (one-sided binomial p = 0.019). This suggests a detectable difference for this group, under these conditions. It doesn't indicate size, preference or market relevance.
Practical exercise: Plan the numbers for your study
- Calculate by hand or with a spreadsheet: the chance of exactly 2 of 3 correct in a triangle test. (Answer: 3 × (1/9) × (2/3) = 2/9, about 0.222.)
- Use a spreadsheet's binomial function to confirm one line of the triangle table above.
- For your own study, state your test, sample size, critical value and the size of effect you'd want to detect, using the tables.
- Write what you'll conclude under each outcome, including no significant difference.
- Ask someone to read your reporting sentence, and to tell you what they'd conclude.
You have done this well if your numbers were fixed in advance, and your plan for a null result doesn't claim that the samples are the same.
Common mistakes
- Stopping when it becomes significant. Set the sample size first.
- Counting repeats as people. They aren't independent.
- Calling a null result proof of a match. Consider the power.
Key terms
- Null hypothesis
- The assumption that there's no real difference, so results arise from chance alone. In a triangle test, it implies one in three correct.
- Statistical power
- The chance that a test will detect a real effect of a stated size. Small tests often have low power.
- P-value
- The probability of a result at least this extreme, if the null hypothesis were true. It isn't the probability that the hypothesis is true.
Knowledge check
Written and reviewed by the RUŌOD Lab team. This is general education, not legal, regulatory, medical or safety advice. Example dilutions and exercises are for learning, are not validated products, and have not been safety-assessed. Completing a lesson confers no qualification. How we check these lessons.