Researchers have identified a critical limitation in how current AI agents approach spatial reasoning: the action interfaces connecting vision-language models to perception tools impose artificial constraints on problem-solving flexibility. A new framework called SpatialClaw attempts to remove those constraints by fundamentally rethinking how agents interact with spatial analysis tools.
The challenge is straightforward in principle but difficult in practice. Humans easily understand where objects are located, how they relate to one another, and how they move through three-dimensional space. Vision-language models, despite their impressive capabilities in image interpretation and language understanding, struggle with these spatial tasks. Researchers have tried augmenting these models with specialized perception modules and geometry tools, but the way agents invoke those tools has remained a bottleneck.
Two Flawed Approaches
According to arXiv research by Seokju Cho, Ryo Hachiuma, and collaborators across multiple institutions, existing spatial agents rely on one of two limiting designs. Some systems execute code in a single pass, forcing the agent to commit to an entire analytical strategy before seeing any intermediate results. Others use rigid tool-call interfaces that constrain how freely agents can combine operations or adapt their approach based on what they observe.
Both approaches sacrifice the flexibility that complex spatial reasoning demands. Real problem-solving often requires iterative refinement: observing a partial result, deciding what to analyze next, then adjusting the overall strategy based on those observations.
Code as the Interface
SpatialClaw takes a different approach by treating executable Python code as the primary action interface. The framework maintains a stateful Python kernel that runs continuously, pre-loaded with input image frames and a library of perception tools and geometry primitives. Rather than choosing between rigid tool calls and blind code execution, the agent writes one executable code cell per reasoning step, always conditioning its next action on all previous outputs.
This design enables genuine flexibility. An agent can compose perception results in novel ways, manipulate geometric data on the fly, and adapt its analysis based on both intermediate text outputs and visual observations. When a spatial reasoning task requires a particular approach, the agent can pursue it directly without wrestling against interface constraints.
The research demonstrates consistent improvements across a broad evaluation. SpatialClaw achieved an average accuracy of 59.9 percent across 20 spatial reasoning benchmarks, outperforming the previous best spatial agent by 11.2 percentage points. The gains held steady across six different vision-language models from two separate model families, suggesting the framework's benefits are robust rather than dependent on specific architectural choices.
Broad Applicability
The benchmarks tested both static and dynamic spatial reasoning tasks spanning three and four dimensions, encompassing the kinds of spatial challenges that arise in robotics, autonomous systems, augmented reality applications, and scientific analysis. The framework required no training and no benchmark-specific or model-specific customization, indicating it functions as a genuinely general-purpose improvement.
The work suggests that how we design the interfaces between high-level reasoning systems and low-level perception tools significantly shapes what those systems can accomplish. By removing bottlenecks in the action interface, spatial reasoning agents gain the flexibility to tackle more complex problems and adapt to unforeseen challenges.