论文标题
通过序列到序列翻译的统一和时间戳监督的时间动作分割
Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation
论文作者
论文摘要
本文在完全和时间戳监督的设置中介绍了通过序列（SEQ2SEQ）翻译的统一动作分割的统一框架。与当前的最新帧级预测方法相反，我们将动作分割视为SEQ2SEQ翻译任务，即将视频帧映射到一系列动作段。我们提出的方法涉及在标准变压器SEQ2SEQ转换模型上进行一系列修改和辅助损失函数，以应对与短输出序列相对的长输入序列，相对较少的视频。我们通过框架损失为编码器的辅助监督信号，并在隐式持续时间预测中提出单独的对齐解码器。最后，我们通过提出的约束K-Medoids算法将框架扩展到时间戳监督的设置，以生成伪分段。我们提出的框架在完全和时间戳监督的设置上始终如一，在几个数据集上表现优于或竞争的最先进。我们的代码可在https://github.com/boschresearch/uvast上公开获取。
This paper introduces a unified framework for video action segmentation via sequence to sequence (seq2seq) translation in a fully and timestamp supervised setup. In contrast to current state-of-the-art frame-level prediction methods, we view action segmentation as a seq2seq translation task, i.e., mapping a sequence of video frames to a sequence of action segments. Our proposed method involves a series of modifications and auxiliary loss functions on the standard Transformer seq2seq translation model to cope with long input sequences opposed to short output sequences and relatively few videos. We incorporate an auxiliary supervision signal for the encoder via a frame-wise loss and propose a separate alignment decoder for an implicit duration prediction. Finally, we extend our framework to the timestamp supervised setting via our proposed constrained k-medoids algorithm to generate pseudo-segmentations. Our proposed framework performs consistently on both fully and timestamp supervised settings, outperforming or competing state-of-the-art on several datasets. Our code is publicly available at https://github.com/boschresearch/UVAST.
