词元变换帮助多模态大模型从邻近视角观察
Token Warping Helps MLLMs Look from Nearby Viewpoints
把视角变换从像素域搬到 ViT 词元域，用反向词元 warping 替代像素 warping，为 MLLM 提供对深度误差更鲁棒的视角外推手段；做多视角/具身推理的可以直接复用这一词元级几何对齐思路。
原文依据 backward token warping, which defines a dense grid on the target view and retrieves a corresponding source-view token for each grid point
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论对词元而非像素进行变换，能否帮助多模态大模型（MLLM）理解场景从邻近视角看是什么样子？尽管 MLLM 在视觉推理上表现良好，但它们对视角变化仍然脆弱，因为逐像素变换对微小的深度误差高度敏感，且常常引入几何畸变。借鉴心理意象理论中「部件级结构表征是人类视角变换基础」的观点，我们考察基于 ViT 的 MLLM 中的图像词元能否作为视角变化的有效载体。我们比较了前向与反向变换，发现反向词元变换——在目标视图上定义稠密网格，并为每个网格点检索对应的源视图词元——在视角偏移下具有更高的稳定性，并更好地保持语义连贯性。在我们提出的 ViewBench 基准上的实验表明，词元级变换使 MLLM 能够从邻近视角进行可靠推理，并持续优于所有基线，包括逐像素变换方法、空间微调的 MLLM 以及一种生成式变换方法。
英文原文摘要 1150 字符
Can warping tokens, rather than pixels, help multimodal large language models (MLLMs) understand how a scene appears from a nearby viewpoint? While MLLMs perform well on visual reasoning, they remain fragile to viewpoint changes, as pixel-wise warping is highly sensitive to small depth errors and often introduces geometric distortions. Drawing on theories of mental imagery that posit part-level structural representations as the basis for human perspective transformation, we examine whether image tokens in ViT-based MLLMs serve as an effective substrate for viewpoint changes. We compare forward and backward warping, finding that backward token warping, which defines a dense grid on the target view and retrieves a corresponding source-view token for each grid point, achieves greater stability and better preserves semantic coherence under viewpoint shifts. Experiments on our proposed ViewBench benchmark demonstrate that token-level warping enables MLLMs to reason reliably from nearby viewpoints, consistently outperforming all baselines including pixel-wise warping approaches, spatially fine-tuned MLLMs, and a generative warping method.