Polyphony：基于扩散与交替视觉Transformer及语义条件的双手动作分割
Polyphony: Diffusion-based Dual-Hand Action Segmentation with Alternating Vision Transformer and Semantic Conditioning
针对双手动作分割中的梯度失衡与语义混淆问题，提出交替训练双手机制与语义特征条件化，可迁移到其他多流/多主体视频时序建模任务中，缓解主导流垄断梯度的问题。
原文依据 alternates training between left- and right-hand mini-batches to ensure balanced gradient contributions from both hands
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论双手动作分割旨在从长视频中为双手稠密预测动作，对理解复杂双手活动至关重要。然而，该任务面临若干独特挑战：复杂的双手依赖关系、双手间的视觉不对称、主导手垄断梯度导致的表征冲突，以及细粒度动作的语义模糊性。我们提出Polyphony，一种三阶段方法，通过以下方式应对这些挑战：(1) 交替双手视觉Transformer，在左手和右手小批量之间交替训练，确保双手梯度贡献均衡，同时共享时空编码器；(2) 语义特征条件化，将视觉特征与结构化、组合式动作描述对齐，以增强对语义相似动作的判别能力；(3) 基于扩散的分割，结合跨手特征融合实现双手协调，并采用自适应损失加权以平衡性能。Polyphony在双手数据集（HA-ViD、ATTACH）上均达到最优，提升高达16.8个点，并在单流Breakfast数据集上达到82.5%，超越使用12倍大骨干网络的最优先前方法。值得注意的是，我们采用单一共享骨干网络的统一模型超越了需要分别建立单手模型的基线。代码见 https://github.com/x-labs-xyz/Polyphony-Dual-hand-Action-Segmentation。
英文原文摘要 1435 字符
Dual-hand action segmentation, densely predicting actions for both hands from untrimmed videos, is essential for understanding complex bimanual activities. However, it poses several unique challenges: complex inter-hand dependencies, visual asymmetry between hands, representation conflicts where the dominant hand monopolizes gradients, and semantic ambiguity in fine-grained actions. We propose Polyphony, a three-stage method to address these challenges through: (1) an Alternating Dual-Hand Vision Transformer that alternates training between left- and right-hand mini-batches to ensure balanced gradient contributions from both hands while sharing a spatio-temporal encoder; (2) Semantic Feature Conditioning that aligns visual features with structured, compositional action descriptions to enhance discrimination of semantically similar actions; and (3) Diffusion-Based Segmentation with cross-hand feature fusion for inter-hand coordination and adaptive loss weighting for balancing performance. Polyphony achieves state-of-the-art on both dual-hand datasets (HA-ViD, ATTACH) with improvements up to 16.8 points, and on the single-stream Breakfast dataset (82.5%), outperforming the prior best method that uses a 12x larger backbone. Notably, our unified model with a single shared backbone surpasses baselines requiring separate per-hand models. Code is at https://github.com/x-labs-xyz/Polyphony-Dual-hand-Action-Segmentation.