Level 8 · Performance Engineering and Evaluation

Evaluation Bias: How You Fool Yourself

Expectation, order, fatigue, color and the wish to succeed all change what you smell. Learn the main biases in evaluation and simple ways to reduce them.

In short

Your judgment of a fragrance is shaped by things other than the fragrance: what you expect, what you smelled just before, the color of the sample, who made it, and how much you want it to work. The most useful defenses are blind coding, balanced orders and writing your prediction first. You cannot remove bias entirely, but you can see it in your results.

What you will learn

  • Name the main biases in fragrance evaluation
  • Build controls into your routine
  • Measure one bias in your own judgment

Before this lesson: Comparative Evaluation: Choosing the Right Test

Common biases

Biases and practical controls
BiasWhat happensControl
ExpectationYou smell what the label or your plan predictsCode samples; have a helper prepare them
InvestmentYou prefer the version you worked on mostBlind comparison with earlier versions
Order and contrastA sample seems stronger or weaker after anotherRandomize orders; rest between samples
FatigueLater samples seem weakerFew samples; breaks; balanced sequences
Visual cuesColor or strip wetness changes your judgmentIdentical containers; prepare out of sight
LanguageA word you read shapes what you noticeWrite your own words before you read a description
SocialOthers echo the first opinion, or try to please youSeparate assessors; do not reveal your hopes
PrestigeA famous name or high price changes the ratingHide brand and price

Everyone is affected, and experience does not provide immunity. Professional panels work under these same controls for this reason.

Practical routine

  1. Write the prediction first. It shows you later what you expected.
  2. Code everything you can. Random numbers, not A and B, since people favor A or lower numbers.
  3. Balance orders. With two samples, use both AB and BA across your trials.
  4. Keep the first answer. Do not change it after you see the key.
  5. Check again later. A judgment that survives a week is worth more than a late-night one.

Self-blinding when you work alone

A helper is best. Without one, prepare identical vials with hidden labels on the bottom. Shuffle them with your eyes closed, then write random codes on the tops. Keep the key sealed. It is imperfect, because you may recognize the samples by smell, which is a real discrimination rather than bias, or by small visual clues, which is not.

Limits

Blinding does not protect you from every problem. It does not solve anosmia, fatigue or small numbers of trials. A blind test by one person shows what that person can do on that day. Treat it accordingly, and get other noses for anything that matters. The research track covers randomization and controls in more depth.

Practical exercise: Catch your own bias

About 30 minutes · You will have: a labeled-versus-blind comparison showing how a label changed your own ratings

  1. Prepare one material or fragrance in two identical strips. Have a helper write the labels, or do it yourself: label one "premium grade" and one "budget grade", although both are the same.
  2. Rate each for strength, quality and character. Write your notes.
  3. Some days later, repeat with both strips coded and blind, in balanced order, using the same scales. Include one real alternative if you have it.
  4. Compare the labeled and blind ratings. Did the label move your ratings? In which direction?
  5. Write one change you will make to your routine.

You have done this well if you recorded the labeled ratings honestly before you did the blind test, and report any effect you saw, even if it was none. You knew about the trick, so any effect probably understates what happens when you do not.

Common mistakes

  • Believing you are immune. People who think so tend not to check.
  • Coding with A and B. Letters and order carry associations. Use random numbers.
  • Discussing samples before all answers are in. The first opinion spreads.

Key terms

Expectation error
A judgment shaped by what you expect to find rather than by what is present.
Order effect
A change in a judgment caused by the position of a sample in a sequence.

Knowledge check

1. Which control best addresses expectation?
Show the answer

Blind coding of samples. If you do not know which sample is which, your expectation has less to work with.

2. You always smell version B after version A. What should you do?
Show the answer

Randomize or balance the order. A fixed order confounds the sample with its position.

3. A darker sample is rated stronger. Which bias might this be?
Show the answer

A visual cue affecting a smell judgment. Use identical containers and coded strips, and hide color differences.

Written and reviewed by the RUŌOD Lab team. This is general education, not legal, regulatory, medical or safety advice. Example dilutions and exercises are for learning, are not validated products, and have not been safety-assessed. Completing a lesson confers no qualification. How we check these lessons.