Engineering GaitAI 02Engineering NoteGaitAI Research8 min read
A Good Model Can Still Be a Bad System
Everything between the camera and the person who acts
A movement model can be right on every frame and the system built around it can still fail the person it was meant to help. Cameras drop frames, inference queues fill, networks stall, alerts arrive late or twice, dashboards hide the one that matters, and a tired operator dismisses the tenth false alarm of the shift. Model quality is one link in that chain. This is about the others.
Before you scroll: a chain from camera to operator that you can break at any link — and a model that stays right the whole time.
Text description
A chain of six components drawn left to right — camera, inference, network, alert, interface, operator — joined by arrows. Each can be marked as failed. The inference link carries a permanent note that the model's outputs are correct. Breaking a link answers two questions beneath the chain. What the model sees: for a frozen camera, a stationary person inferred correctly from a frozen frame; for a saturated inference queue, correct results produced late; for every downstream failure, nothing — its outputs are correct and go nowhere useful. What reaches the person: the outcome named in words — alert in time, silence mistaken for calm, alert after the window, alert never arrives, alert doubled or swallowed, alert not seen, alert dismissed unread. The upstream-most failure decides the outcome. Conceptual: kinds of failure and their consequences, not rates, uptimes or latencies.
Short on time? Read the 2-minute version
- A deployed movement system is a chain: capture, inference, transport, alerting, interface, human review. The model is one link.
- Each link fails in its own way — a frozen camera, a saturated inference queue, a network partition, an alert that arrives twice or never, a dashboard that buries it, an operator who has learned to ignore it.
- Most of these failures are invisible to the model. It keeps producing correct outputs for the frames it receives; the frames, or the outputs, simply stop going anywhere useful.
- Late is a failure mode of its own. A correct alert that arrives after the moment it was for has the cost of a missed one and the cost of a false one.
- Reliability has to be designed end to end: heartbeats, known-unknown states, idempotent alerts, honest interfaces, and a human workload the system does not exhaust.
- The measure of a system is not the model's score. It is whether the right person saw the right thing in time — and knew when they could not.
The full article follows, with the diagrams and the caveats.
Most of the writing about movement AI, including a good deal of this journal, is about the model: what it sees, where it fails, what it should and should not claim. That is right, and it is not enough. A model does not help anyone. A system does — and the system is everything between the lens and the person who decides what to do.
This story follows that chain link by link. It is not a catalogue of things that go wrong for its own sake; it is an argument that reliability is a property of the whole, that most of the whole is not the model, and that a system has to be honest about its own state in exactly the way the earlier stories asked a model to be honest about its measurements.
The hero figure is a conceptual demonstration. It shows kinds of failure and their consequences as states, not as rates; no uptime, latency or accuracy figure appears anywhere in this article, and none should be inferred.
The chain
Every deployed movement system has some version of this chain. A camera or sensor captures; a model, on a device or a server, infers; a network carries the result; an alerting layer decides what is worth someone's attention; an interface shows it; a person reads it and acts, or does not. The names vary, the order sometimes does, but the shape is constant.
The model's quality — everything the first Foundation and the AI Under Stress stories are about — decides how good the second link is. It says nothing about the other five. A system is only as reliable as the weakest of the six, and the weakest is rarely the one with the research literature behind it.
How each link fails, and what the model sees
The hero figure lets you break each link in turn. The right-hand column is the important one: what the model experiences while the system fails around it. In most cases, nothing.
In five of the six failures, the model's outputs were correct. In all six, the person who needed them did not get them in time.
Late is its own failure
Model evaluation asks whether an output is right. Systems have a second axis: when. A movement event — a fall, a stop, a change in gait that a clinician wanted to know about — has a window in which knowing about it is useful. An alert that arrives after the window is not a late success. It is a miss that also costs someone's attention.
In time
The alert does its job
- Arrives inside the window it was for
- Costs one look
- Builds trust when it is right
Late
The alert costs twice
- The event has already resolved, well or badly
- Costs a look and a dismissal
- Teaches the operator that alerts arrive after the fact
Latency accumulates along the chain: capture, inference, transport, queueing, rendering, noticing. Each stage adds a little; several add a lot under load, which is exactly when the events being watched for are most likely. A system that reports its own end-to-end delay, and knows when it has exceeded the window, is doing something the model cannot do for it.
The system has to know when it does not know
The fifth Foundation drew a line between a missing input and a silently corrupted one: the first is a routing problem the system can see, the second a detection problem it may not. The same line runs through the whole chain. A camera that reports "no frames for thirty seconds" is a known unknown. A camera that keeps sending the same frozen frame is a silent one — and the model will infer a perfectly stationary person from it, correctly.
- Heartbeats, not silence. Every link should say "I am here and this is my state" on a schedule, so that silence means failure rather than calm.
- Staleness is a state. A result older than its window should be shown as stale, not as current. Age is data.
- Duplicates are failures too. An alert delivered twice is not twice as safe; it is a sign the transport does not know what it delivered, and it costs attention.
- Degraded is a mode. A system that has lost a camera should say so and keep running on what remains, rather than either stopping or pretending.
The last link is a person
Everything upstream exists so that a person can decide something. That person is a component with its own failure modes, and they are not moral failings: attention is finite, trust is earned and lost by experience, and a screen full of alerts that are usually wrong trains anyone to stop reading it. This is the oldest finding in alarm design, and movement AI is not exempt from it.
It follows that the number of alerts a system emits is a reliability decision, not just a sensitivity setting. A model tuned to miss nothing will, through the operator, end up missing a great deal. The framing the Foundations insisted on is partly about this: the output is offered to a person to review, so the system has to be designed around what a person can actually review.
What reliability means here
Reliability for a movement system is not the model's score and not the servers' uptime. It is a chain of properties, each of which has to hold:
- The capture is present and the system knows when it is not.
- The inference is current and the system knows when it is stale.
- The transport delivers once and the system knows what it delivered.
- The alert is in time and the system knows the window.
- The interface shows the right thing first.
- The person has a workload they can carry, and reasons to trust what they see.
A good model is the part of this that gets published. A good system is the part that gets deployed. GaitAI's own position is that the second is the product — and that it should be described, tested and reported as a chain, because that is what fails.
A model that is right on every frame is necessary and not remotely sufficient. The frames have to arrive, the results have to travel, the alert has to land in time on a screen someone is looking at, and that someone has to have been given reason to believe it. Every one of those is engineering, and every one of them fails quietly.
Engineering GaitAI continues with the hardest version of this chain — the one that has to run in real time, where every link's latency is a link in its own right.
See how GaitAI deploys →Written by GaitAI Research. GaitAI produces movement measurements and decision-support outputs for research, clinical and operational teams — not diagnoses. Published research is listed in the publications library.
Engineering GaitAI · 2 of 2
You now know that a movement system is a chain, that the model is one link, and that most of the chain fails without the model noticing.
The series so far
What does the chain look like when every link has to run in real time?
Engineering GaitAIEvery story in Engineering GaitAI2 stories so far — more followSee the series