Computer vision researchers have unveiled a novel approach to video captioning that addresses a fundamental limitation in how modern multimodal AI systems generate descriptions of moving images. Rather than jumping directly to a final caption, the new method requires models to first articulate what they perceive about spatial and temporal elements within video content.

According to arXiv, the framework, called PercepCap, tackles a critical problem: existing multimodal language models produce captions from video inputs without showing the underlying reasoning. When errors occur in the final description, engineers struggle to pinpoint whether the mistake stemmed from faulty object detection, timeline misunderstanding, or caption generation itself.

Making AI Perception Transparent

PercepCap operates through what the research team describes as a perceive-describe generation chain. The system first generates explicit evidence of what it observes: object trajectories drawn across frames and timestamped events detected in the sequence. Only after completing this intermediary step does the model produce the final caption, which is grounded in the identified perception data.

This intermediate layer serves multiple purposes. It forces the AI to commit to specific spatial and temporal claims before synthesizing them into prose. It also creates a clear checkpoint where human reviewers or automated systems can validate whether the model correctly identified relevant elements.

Training Strategy and Data Construction

The researchers implemented a sophisticated two-stage training process to enable this behavior. In the first phase, called Perceive-then-Describe Supervised Fine-tuning, they adapted an existing model from producing captions alone to following the new multi-step pipeline. The second phase, Perception-Grounded Reinforcement Learning, optimized both the perception traces and final captions using joint reward signals that evaluate the entire chain rather than isolated outputs.

A crucial innovation was the Caption-Anchored Perception Data Construction pipeline. Rather than manually annotating thousands of videos with both objects, events, and captions, the team reversed the process. They generated captions first, then extracted mentioned objects and events, and finally grounded those elements back into the video using bounding boxes and timestamps. This approach ensured that training data remained internally consistent: perception traces and captions always referenced the same entities and occurrences.

Performance and Implications

Testing showed that PercepCap outperformed the Qwen3-VL baseline across multiple evaluation metrics. The framework demonstrated improvements both on direct caption benchmarks and on caption-to-question-answering tasks, where generated descriptions are evaluated by how well they answer specific queries about video content.

The work carries broader significance for AI transparency and reliability. As video understanding systems become more prevalent in everything from autonomous vehicles to content moderation platforms, the ability to inspect intermediate reasoning becomes increasingly important. By making perception explicit, PercepCap offers a path toward more interpretable and debuggable video AI systems.

  • The framework separates perception from description generation
  • Intermediate representations expose spatial and temporal reasoning
  • Training data construction uses caption-anchored grounding
  • Joint reward optimization improves both perception and description quality