Discrimination Tests and Thurstonian d′

Before you start

You need sample size and power, blinding and randomization and comparative evaluation.

The binomial test

Under the null hypothesis of no perceptible difference, each of n assessors picks the odd sample with probability p₀ = 1/3. The probability of at least k correct is

P(X ≥ k) = Σ_i=kⁿ C(n, i) p₀ⁱ (1 − p₀)ⁿ⁻ⁱ

For n = 36: P(X ≥ 17) ≈ 0.058 and P(X ≥ 18) ≈ 0.028, so 18 correct is the smallest significant count at α = 0.05, matching ISO 4120's table. A result of 20 correct gives P ≈ 0.005.

The guessing model and its weakness

The proportion of discriminators assumes each assessor either truly perceives the difference (and is always right) or guesses:

pd = (p_c − p₀) / (1 − p₀)

With 20/36 correct, p_c = 0.556 and pd = 0.333: "a third of the population discriminates". This is intuitive and widely reported, but its picture of perception — everyone either perceives perfectly or not at all — is not how sensation works, and pd for the same products changes with the method.

The Thurstonian model

Each presentation of a product gives a sensation drawn from a normal distribution; two products differ by d′ standard deviations. A method is a decision rule applied to these noisy sensations, and its proportion correct is a known function of d′. For the 2-AFC (which is stronger?) it is simply

p_c = Φ(d′ / √2)

where Φ is the standard normal distribution function. For the triangle test the function is an integral with no closed form, usually read from tables; integrated numerically, d′ = 1 gives p_c ≈ 0.42 and d′ = 2 gives ≈ 0.61.

The triangle test is a hard cognitive task — finding the odd one of three — so even a large d′ yields a modest proportion correct. A directional 2-AFC, where the attribute is named, uses the same sensory difference far more efficiently. That is the main practical consequence: for a known attribute (is the reformulation less animalic?), a directional test needs far fewer assessors for the same d′.

Planning with d′

State the smallest difference that matters as a d′ — say, a reformulation must not reach d′ = 1 — then choose the method and calculate the sample size that would detect that d′ with your chosen power. Ennis's 1993 paper gives such tables; later work notes that its normal approximation overstated power slightly, so check with exact binomial calculations.

Where it stops

• Equal-variance normal sensations are an assumption. Fragrance evaluation adds adaptation and carryover, which the basic model ignores.
• Failing to reject is not proving similarity. A non-significant triangle test with twelve people does not show the products are the same; a similarity test is designed differently, around a maximum tolerable d′.
• Assessors are not interchangeable. Trained evaluators and consumers have different d′ for the same products.

Open questions

How large a d′ do consumers notice in use, as opposed to in a lab test? Linking laboratory d′ to in-use detection for fragrance is still a research question, and it decides how strict a reformulation's target should be.

Common mistakes

• Comparing pc across methods. Compare d′.
• Calling a non-significant result "no difference". Check power first.
• Testing too few people and concluding anything strong. Small panels can only detect large d′.

Key terms

Thurstonian model

A model of discrimination in which each sample produces a sensation that varies randomly from moment to moment. Difference is measured as the distance between the two distributions in standard-deviation units, d′.

d′ (d-prime)

The standardized distance between the sensation distributions of two products. It describes the products' difference independently of the test method used to measure it.

Proportion of discriminators

pd = (pc − p0)/(1 − p0), where pc is the proportion correct and p0 the guessing probability. It assumes each assessor either discriminates or guesses, which the Thurstonian model does not.

2-AFC test

Two-alternative forced choice: two samples, and the assessor names the one that is stronger in a specified attribute. Guessing gives 1/2.