Request a Demo
← Back to Blog

Foundations 05Research NoteGaitAI Research5 min read

When Fusion Looks Better Than It Is

Five questions to ask before trusting a multimodal AI model

More modalities and more elaborate fusion do not automatically mean better evidence. Five questions that separate a genuinely convincing multimodal result from a well-decorated benchmark number.

Before you scroll: five questions to put to any multimodal result, including our own.

Interactive heroRemove a stream. Then corrupt one.Conceptual demonstration

RGBappearance · scene

Posegeometry · symmetry

IMUstep timing · dynamics

Audiofootfall rhythm

Complete. All four streams present and sound. The best case — and the only case most benchmarks test. No accuracy or confidence is shown because none is published; coverage and honesty are drawn qualitatively.
Text description

Four input streams — RGB, pose, IMU and audio — converge on a fusion stage that produces one result. Each stream can be set to healthy, missing or corrupted. Healthy: all four bars in the result are filled and the result reads "complete". Missing: the stream's line becomes a dashed gap, its bar in the result is empty and labelled "gap", and the result reads "known gap — the system can say what it cannot answer". Corrupted: the stream's line keeps arriving but is visibly noisy, its bar is still filled but jagged, and the result keeps its full layout while a label says "looks complete — the damage is not announced". The lesson: a missing input is a routing problem the system can see; a silently corrupted one is a detection problem it may not.

Short on time? Read the 2-minute version
  1. If the benchmark is already easy, an extra modality can raise the score without adding capability.
  2. The honest test is what happens when one modality disappears at inference time.
  3. Silent corruption is worse than absence: a stream that is present but wrong is believed by default.
  4. An explanation is only useful if it reflects the model that actually made the decision.
  5. An improvement has to be both statistically and operationally meaningful to matter to anyone.

The full article follows, with the diagrams and the caveats.

Multimodal AI has an intuitive promise.

More sensors
More signals
More sophisticated fusion
Better intelligence
The intuition — persuasive, and sometimes correct.

Sometimes that is exactly what happens.

But a higher benchmark score does not, by itself, demonstrate that the fusion mechanism is responsible for the gain, that it is robust, or that it is useful outside the evaluation. Extra modalities bring extra parameters, extra tuning and extra opportunities for a number to move for reasons unrelated to the idea being tested.

Before trusting a multimodal result — including our own — these are the questions we think it should survive.

Question 1

Is the benchmark already too easy?

Some benchmarks are effectively solved. When a classical, well-tuned baseline already sits close to ceiling, the remaining headroom is too small to distinguish an architectural contribution from ordinary tuning variance.

Saturated

Little to learn

  • A simple baseline is already near-perfect
  • Remaining headroom sits inside run-to-run noise
  • A small gain establishes almost nothing about the architecture

Informative

Room to discriminate

  • Baselines leave substantial headroom
  • Improvements exceed seed variance
  • The task exercises the mechanism being claimed

This is the strongest argument for reporting simple baselines prominently rather than burying them. A well-implemented classical method is not an embarrassment in a results table — it is the reference that gives every other row its meaning.

Question 2

What happens when one modality disappears?

Full-input evaluation describes the best case. Deployments rarely operate in the best case: a camera is obstructed, a wearable is not worn, a stream drops.

A multimodal claim should therefore be evaluated under systematic input removal, not only with everything present:

  • RGB + Audio + Pose + Trajectory — the complete set
  • RGB + Audio + Pose
  • RGB + Pose
  • Audio + Pose
  • Pose only
  • …and every other subset the deployment could realistically encounter

What that reveals is whether the model has learned to combine evidence or merely to depend on it. A system that collapses when one stream is absent has not fused those modalities so much as chained them.

Robustness to missing inputs is not an edge case. In deployment, it is most of the cases.

Question 3

Is the modality missing — or silently corrupted?

These are routinely conflated, and they are not the same technical problem at all.

Known missing

Absence the system can see

  • The sensor is unavailable
  • The system knows the stream is absent
  • A mask or availability signal is present
  • The model can route around the gap

Silent corruption

Presence the system trusts

  • The sensor reports as available
  • The evidence it supplies is damaged
  • Nothing marks the stream as unreliable
  • The model may weight it as if it were sound

Known absence is a routing problem: the architecture needs a defined behaviour for an input that is not there. is a detection problem: nothing in the input announces itself as wrong, so the system must estimate the reliability of evidence it has already accepted.

Question 4

Does the explanation reflect the actual model?

Attribution methods produce explanations that are easy to read and easy to believe. Plausibility, however, is not faithfulness. An explanation can look entirely sensible to a domain expert while bearing little relationship to what the model actually computed.

Useful interrogations of an attribution claim:

  • Does the explanation change when the modality set changes? An attribution indifferent to its inputs is describing something other than this model.
  • Is the ranking stable? Across seeds, across runs, across comparable configurations.
  • Does it depend on learned behaviour? Or would an untrained network produce a similar-looking map?
  • Does it survive sanity checks? Parameter-randomization and data-randomization tests exist precisely for this.

A plausible explanation is not necessarily a faithful one.

Question 5

Is the improvement statistically AND operationally meaningful?

The last question is the least glamorous and the most frequently skipped. A single-run improvement on a single split is a starting point for an investigation, not a result.

  • Does the gain hold across random seeds?
  • Does it hold against simple, well-tuned baselines?
  • Does it hold under matched compute and comparable parameter budgets?
  • Does it hold under missing inputs?
  • Does it hold under corrupted inputs?
  • Is it computed over the correct independent evaluation unit — subject or session, rather than individual frames from the same recording?
  • Is it reported with confidence intervals, not as a bare point estimate?

The evaluation-unit question deserves particular attention in movement research. Frames drawn from one walking sequence are not independent observations; treating them as if they were inflates apparent sample size and shrinks apparent uncertainty, often dramatically.

The bigger lesson

None of these questions is hostile to multimodal work. We build multimodal systems; the questions exist because we would rather find the weakness ourselves than ship it.

The goal is not to make fusion look impressive. The goal is to determine which evidence actually supports the model.

That standard is what the research programme is organized around, and what the peer-reviewed record in the publications library is meant to expose to scrutiny.

Interactive checklistFive questions to put to any multimodal resultConceptual demonstration
Every diagram is a conceptual demonstration with no values on any axis. Use them to feel the shape of each question, then read the section that asks it.
Text description

Five questions, each with a small interaction. 01 Is the benchmark too easy? A control raises the baseline's strength; the headroom left for any new method shrinks until a gain sits inside noise. 02 What happens if a modality disappears? Removing an input shows whether the result degrades gracefully (fused) or collapses (chained). 03 What happens if a modality silently corrupts? Corrupting an input leaves the result the same size, with nothing announcing the damage. 04 Are the explanations faithful? Swapping the input set changes a faithful attribution and leaves a merely plausible one unchanged. 05 Does the improvement matter operationally? Adding seeds draws an interval around a point estimate; evaluating per subject instead of per frame widens it further.

The question is not whether fusion can produce a better number.

The question is whether the evidence justifies believing the number.

Explore GaitAI Research →
Multimodal AISensor FusionModel EvaluationResearch

Written by GaitAI Research. GaitAI produces movement measurements and decision-support outputs for research, clinical and operational teams — not diagnoses. Published research is listed in the publications library.

GaitAI Foundations · 5 of 5

You now know how to tell a convincing multimodal result from a well-decorated benchmark number.

The path completes

You have followed one walk from pixels to evidence. Where does the reading path begin again?

GaitAI FoundationsWalk the whole path againFive stories, from a walking video to an audited claimStart here