Evidence-Linked Video Analysis Over Time
Scene Models That Connect Every Claim to a Frame and Test What Happens Next
Why This Matters
Ask a conventional video AI what it sees and it will tell you: “Pedestrians crossing an intersection.”
That is accurate, and you cannot do anything with it.
It does not tell you which pedestrian is on a path that intersects a turning vehicle. It does not tell you that a cyclist entered the frame four seconds ago and has not yet cleared. It does not tell you which of its statements it is confident about and which it guessed. And if it is wrong, there is no way to find out where the error entered — the output is a sentence, not a record.
For anything with real consequences attached — a robot deciding whether to move, an operator watching twelve camera feeds, an incident review six months later — a caption is the wrong output. You need to know what is about to happen, what evidence supports that, and how wrong the system might be.
That is what this system produces instead.
(For the underlying idea in plain English, with one photograph and no jargon, see What a Skateboard Reveals About the Framework →.)
Executive Overview
Evidence-Linked Video Analysis builds an organized model of a changing scene, moving beyond surface-level captions and generated summaries.
The system produces a checkable 3x3 scene model that records:
- What is present (Objects & Entities)
- What is happening (Behaviors & Dynamics)
- What mechanisms explain it (Underlying Physical & Algorithmic Controls)
- What emerges from their interaction (Macro System Properties)
- What must happen next (Refutation-Tested Predictions)
Every observation records whether it was seen, inferred, or predicted, along with its confidence score, image region, and exact video frame timestamp.
What Makes This Different
Three properties separate this from typical video-to-text systems.
1. Every claim points back at pixels. Each observation carries the frame index and image region it came from. You can ask any statement “where did you get that?” and receive coordinates rather than a generated explanation. Typical video summaries do not retain this direct link back to the pixels, which makes well-supported statements and invented ones difficult to distinguish.
2. Uncertainty is recorded, not smoothed away. Every observation carries a confidence score and states whether it was seen, inferred, or predicted. An operator or control system can therefore treat a measurement differently from a guess instead of receiving one undifferentiated block of prose.
3. It states in advance what would prove it wrong — then checks. Each prediction declares its own refutation criteria: the specific visual evidence that would falsify it. The model then tests those predictions against future frames it had not seen when it made them.
That third property is the substantive difference. A conventional video model generates a description and stops; nothing in its design can discover that it was wrong. This system produces a claim, a condition under which the claim fails, and then a verdict from evidence it did not have access to at prediction time. It is graded against reality rather than against plausibility — which means its accuracy is a measurement you can inspect, not a vendor assertion you have to accept.
How the Evidence Is Kept Honest
- Evidence Linked to Frames: Every observation is tied to a specific frame ($f_i$) and image region.
- Predictions Checked Against Later Evidence: Every prediction states what future visual evidence would prove it wrong. The model then checks the prediction against video frames it had not yet seen.
- Many Actors Tracked Over Time: The system follows people, vehicles, overlapping paths, and their interactions across a sequence rather than treating each frame separately.
Key Capabilities
| Feature | Conventional Video AI | 3x3 Temporal Video Intelligence |
|---|---|---|
| Output | Unstructured text caption | Structured scene model with evidence links |
| Link to Evidence | Usually absent from generated summaries | Specific frame and image region |
| Prediction Checking | Description stops at the current frame | Predictions tested against later frames |
| Domain Applicability | General chat | Autonomous robotics, defense surveillance, industrial monitoring |
Where This Is Worth the Cost
Structured modeling is more expensive than captioning. It earns that cost wherever being wrong is expensive:
- Autonomous robotics: A robot deciding whether to enter a space needs the hazards that are not yet in frame, not a description of the ones that are. Predictions with declared failure conditions can be gated on; a caption cannot.
- Defense and security monitoring: One operator across many feeds cannot review everything. Ranking by what the model expects to happen next — with its confidence exposed — is a different task from reviewing what already happened.
- Industrial process monitoring: Failures announce themselves in the interactions between subsystems before they show up in any single sensor channel. Modeling the scene as a system surfaces that; frame-by-frame classification does not.
- Incident reconstruction and audit: When the question is “what did the system know, and when,” frame-indexed evidence with recorded confidence is a record. Generated prose is not.
The common thread: these are settings where somebody eventually has to defend a decision made from the footage.
To discuss an application to your own systems, Get in Touch → Explore the Methodology →