Vision-language models can describe images, answer questions about videos, and generally behave as if they see. But where, inside the model, does seeing actually meet reading? For all their fluency, VLMs remain partly inscrutable on this point — which matters, because you cannot fix a failure mode you cannot locate. A new paper takes the direct approach: reach into the model, intervene, and watch what breaks. “From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making,” by Testa, Wong, Lenci, Magnini, and Gatt — submitted to arXiv (2609.05149) on September 4 and accepted to the Findings of EMNLP 2026 — maps the causal pathways between video and text inside multimodal models.
The method is the message here. Rather than probing representations or visualizing attention (both correlational), the authors perform layer-wise causal interventions on video-text attention: selectively disrupting the flow of information between visual and textual processing at different layers, then measuring how decisions change. It is the closest thing the field has to asking the model, under controlled conditions, where were you using that visual information, exactly?
The test setup
The experiments use video-based generative multiple-choice questions probing three kinds of reasoning: spatial (where things are), causal (why things happen), and temporal (in what order). Video is a deliberate choice — static images let models cheat with single-frame shortcuts, while video forces genuine cross-frame integration. If a model answers “what happened first” correctly, it must have fused information across time and across modalities. Or so we hope.
It helps to appreciate how unusual the method is. Most interpretability work on VLMs is observational: visualize attention maps, probe classifiers on hidden states, correlate. Those techniques tell you what the model looks at, which is suggestive but not decisive — attention is not explanation, as the field learned years ago. Causal intervention is experimental: break a specific information pathway and see whether the decision changes. It is slower, fussier work, and it answers a narrower question — but the answer is an actual answer. When the authors say visual information is integrated at the answer options, they mean they disrupted that pathway and watched performance move. That is a different epistemic standard, and the field needs more of it.
Where the fusion happens
The headline finding: visual information is integrated into the decision mainly while the model processes the candidate answer options — the textual grounding sites where each possible answer is evaluated. The model does not appear to build one grand fused representation of the video upfront and then reason over it. Instead, vision meets language late and locally, at the moment each answer choice is considered.
That is a genuinely surprising architectural fact about how these models work in practice, and it reframes several debates. If cross-modal fusion is concentrated at the answer options, then the phrasing of the options matters far more than we assumed — and “multimodal reasoning” may be shallower than the fluent outputs suggest: less “watch the video and think,” more “check each answer against the video.”
Nouns anchor, verbs sequence
Drilling into the language side, the interventions reveal a division of labor. Nouns act as semantic anchors — the stable points where visual content attaches to language (“the red car,” “the doorway”). Verbs carry the temporal relations — the “before,” “after,” and “while” structure that sequences events. This division predicts the failure modes the authors observe: VLMs struggle to reconstruct sequential information across frames, and that fragility may partly reflect linguistic biases — the way temporal expressions work in language — rather than purely visual deficits. The model’s clock, in other words, may be made of words.
What this means for builders
If you build on VLMs, the practical takeaways are immediate. First, prompt and option design is load-bearing: the finding that fusion concentrates at candidate answers implies that rephrasing options — or the order they appear in — can change what the model effectively “sees.” Anyone doing multiple-choice evaluation of VLMs should be testing option-order and option-phrasing robustness as a matter of routine — our guide to evaluating open LLMs with the EleutherAI eval harness shows what that looks like in practice. Second, temporal reasoning is the weak joint: if your application needs a model to reconstruct event sequences from video — surveillance review, sports analysis, procedural verification — assume fragility and design verification around it. Third, the noun-verb split suggests a concrete debugging heuristic: when a VLM misdescribes a scene, check the nouns (grounding failure); when it misorders events, check the verbs (temporal failure). Different failures, different fixes.
Why it matters beyond the lab
These are not just curiosities for interpretability researchers. Consider where VLMs are being deployed: as the perceptual front end of vision-language-action systems — the “eyes” of robots. LG’s upcoming humanoid, built on NVIDIA’s GR00T VLA models, will live or die on exactly this kind of cross-modal grounding: seeing a scene, understanding an instruction, acting correctly. If fusion happens late and locally at answer-like decision points, that has implications for how we design robot control architectures — and for where we expect them to fail (temporal sequencing, anyone?).
The paper also offers a methodological template. Causal intervention is harder than attention visualization and more honest than benchmark-chasing: it tells you what the model uses, not what it looks at. As multimodal models move into higher-stakes roles, “where does the seeing meet the reading” will graduate from research question to safety question — the same transition Anthropic’s monitored safety-research harness is trying to engineer for alignment failures. Testa and colleagues have given us one of the first careful maps. The territory, as always, is larger than the map — but now we know where to start digging.
There are natural next questions the paper opens without answering. Do these fusion patterns hold for the largest VLMs, or are they an artifact of the studied scale? Does the late-fusion pattern persist in models trained with different objectives — say, generative video-language models rather than multiple-choice-tuned ones? And most intriguingly: if fusion happens at decision points, can architectures be designed to fuse earlier and better, building the unified representations these models currently seem to skip? The map is drawn; the expedition is just beginning.
Featured image: a Canon EF 85mm camera lens (Armin Kübelbeck, via Wikimedia Commons — CC BY-SA 3.0).
