RUŌOD Lab

Mastery course · Software and Computational Perfumery

Validating Odor-Prediction Models

How published models predict odor from structure, how they are validated, and the leakage, label-noise and domain errors that flatter a model.

Mastery 55 min study By the RUŌOD Lab team Updated

In short

Odor-prediction models learn a mapping from molecular structure to odor descriptors, and the best published ones are now prospectively validated: Lee and colleagues reported in 2023 that, for 400 molecules outside their training data, their model's predicted profiles were on average closer to the panel's mean than the median panelist's were. Such a result is meaningful only within the model's domain — single molecules, the descriptors it was trained on — and only if the validation avoided leakage. Knowing how to check that is the skill.

What you will learn

  • Describe what Keller and colleagues (2017) and Lee and colleagues (2023) predicted and how they validated it
  • Explain train/test splits, prospective validation and data leakage in this field
  • Identify label noise, applicability domain and baseline comparison as questions to ask of any model
  • Write a structured critique of a published or commercial odor-prediction claim

Before this lesson: Molecular Descriptors for Odorants, Predicting Odor from Structure, AI in Perfume Research and Formulation: Uses and Limits, Reading Fragrance Research Critically

Before you start

This builds on molecular descriptors, predicting odor from structure, AI-assisted research and formulation and reading fragrance science papers. The RUŌOD Lab app contains no AI and makes no predictions; this lesson is about evaluating other people's models.

Two landmark studies

Keller and colleagues (2017) ran a crowd-sourced challenge: teams predicted panel ratings of intensity, pleasantness and descriptors for molecules from chemical features, and their models were scored on held-out molecules. It showed that some perceptual qualities can be predicted from structure, better for some descriptors than others, with individual perception harder than the population average.

Lee and colleagues (2023) trained a graph neural network on an industry dataset of odorants and their descriptors and built a "principal odor map". Their key test was prospective: new molecules, rated by a trained panel, compared with the model's predictions. The claim to remember is the one in the abstract, about reliability comparable to a human panelist for single molecules. Press coverage gave further figures; check any number in the paper itself before quoting it.

Questions to ask of any model

  1. What exactly is predicted? Which descriptors, from which panel, on which scale, for single molecules or mixtures?
  2. How was the test set separated? A random split of molecules can put near-duplicates (enantiomers, homologs, salts) in both sets. A split by structural scaffold, or a prospective test, is harder and more honest.
  3. Is there leakage? Were descriptors chosen, hyperparameters tuned or outliers removed after looking at test results?
  4. How noisy are the labels? If two panelists agree only modestly, no model can exceed that agreement by much; compare the model with human-to-panel agreement, as Lee and colleagues did.
  5. What is the baseline? Does it beat nearest-neighbor similarity, or a simple descriptor regression?
  6. What is the applicability domain? Is your molecule like the training molecules? Is your question (a mixture, a trace impurity, a concentration effect) one the model was trained on?

Where these models stop, today

  • Single molecules, not perfumes. Mixture prediction is far less developed, and perfumes are mixtures with interactions.
  • Quality, not dose. Most models predict descriptors at a fixed concentration; intensity curves and thresholds are another problem.
  • Proprietary data. Many training sets are industry data, which limits independent replication.
  • Safety is a separate science. No odor model says whether a molecule is safe or permitted.

Open questions

Whether structure-based models can predict how a material will behave in an accord — suppression, blending, emergent notes — is open. So is how well models trained on one population's descriptors transfer to another's vocabulary and genetics, given the receptor variation the Molecular Perfumery track discussed.

Practical exercise: Critique one odor-prediction claim

About 150 minutes · You will have: a two-page structured critique of one published paper, preprint or product claim, answering the validation questions in this lesson

Choose one claim: a published paper or preprint on odor prediction, or a commercial tool's statement that it predicts odor or generates formulas.

  1. State the claim in one sentence, quoting the source.
  2. Answer each of the six questions above, citing where in the source the answer is, or saying that it is not given.
  3. Name the single most important missing piece of evidence, and design the test that would supply it.
  4. Say what, if anything, the claim would change in your own practice.

You have done this well if every answer points to a specific place in the source or says plainly that the information is absent, you separate what was shown from what was claimed, your proposed test could be carried out, and you neither dismiss nor accept the claim beyond what the evidence supports.

Common mistakes

  • Quoting press figures as findings. Use the paper.
  • Treating a model as a panel. It predicts an average of the panel it learned from.
  • Extending a single-molecule result to formulas.

Key terms

Data leakage
When information about the test set reaches the model during training, for example the same molecule appearing in both sets under different names. It inflates reported performance.
Prospective validation
Testing a model on molecules evaluated after the model was built, by people who did not see its predictions. The strongest form of validation.
Applicability domain
The region of chemical space where a model's training data give it grounds to predict. Outside it, predictions are extrapolations.
Label noise
Disagreement or error in the training targets. Odor descriptors vary between people and panels, so the labels themselves are uncertain.

Knowledge check

1. What did Lee and colleagues (2023) report for their prospective set of 400 odorants?
Show the answer

The model's predicted profiles were, on average, closer to the panel mean than the median panelist was. That is the abstract's claim: reliability comparable to a human panelist for single molecules, on molecules outside the training data.

2. A model scores very well, but the test set contains enantiomers and salts of training molecules. What is the problem?
Show the answer

Leakage: near-duplicates make the test easier than real use. Near-identical structures in both sets measure memory rather than generalization.

3. Why compare a model with a simple baseline?
Show the answer

A sophisticated model is only useful if it beats something simple, such as nearest-neighbor similarity. Without a baseline you cannot tell whether the complexity bought anything.

4. A model trained on single molecules is used to predict a perfume's smell. What is the main concern?
Show the answer

Mixtures are outside its applicability domain. Mixture perception involves suppression and blending that single-molecule training data do not contain.

Sources

  • Keller A, et al. Predicting human olfactory perception from chemical features of odor molecules. Science 355(6327):820–826, 2017 — A crowd-sourced challenge predicting perceptual ratings from chemical features
  • Lee BK, et al. A principal odor map unifies diverse tasks in olfactory perception. Science 381:999–1006, 2023 — A graph neural network map of odor, prospectively validated on 400 out-of-sample odorants
  • Snitz K, et al. Predicting odor perceptual similarity from odor structure. PLOS Computational Biology, 2013 — Similarity prediction, including for mixtures

Further reading and materials

Written and reviewed by the RUŌOD Lab team. This is general education, not legal, regulatory, medical or safety advice. Example dilutions and exercises are for learning, are not validated products, and have not been safety-assessed. Completing a lesson confers no qualification. How we check these lessons.