论文标题
多模式的掩盖预训练单眼全景完成
Multi-Modal Masked Pre-Training for Monocular Panoramic Depth Completion
论文作者
论文摘要
在本文中，我们制定了一个潜在的有价值的全景深度完成（PDC）任务，因为全景3D摄像机通常会在复杂的场景中产生360°深度，而缺少数据。它的目标是从原始的稀疏图像和全景RGB图像中恢复密集的全景深度。为了处理PDC任务，我们训练一个深度网络，该网络将深度和图像作为密集全景深度恢复的输入。但是，由于其非凸目标函数，它需要面对网络参数的具有挑战性的优化问题。为了解决这个问题，我们提出了一种简单而有效的方法，称为m {^3} pt：多模式掩盖的预训练。具体而言，在预训练期间，我们同时覆盖了全景RGB图像和通过共享随机掩码的稀疏深度的斑块，然后重建掩盖区域中的稀疏深度。据我们所知，这是我们第一次在多模式视觉任务中显示蒙版预训练的有效性，而不是蒙版自动编码器（MAE）解决的单模式任务。与MAE进行微调完全丢弃了预训练的解码器部分，在我们的M $^{3} $ pt中的预训练和微调阶段之间没有建筑差异，因为它们在预测密度方面只有不同，这可能会使传递更方便和有效。广泛的实验验证了三个全景数据集上M {^3} PT的有效性。值得注意的是，我们在RMSE中将最先进的基线提高了26.2％，MRE的51.7％，MAE的49.7％和37.5％的RMSelog在三个基准数据集中的RMSelog中提高了37.5％。
In this paper, we formulate a potentially valuable panoramic depth completion (PDC) task as panoramic 3D cameras often produce 360° depth with missing data in complex scenes. Its goal is to recover dense panoramic depths from raw sparse ones and panoramic RGB images. To deal with the PDC task, we train a deep network that takes both depth and image as inputs for the dense panoramic depth recovery. However, it needs to face a challenging optimization problem of the network parameters due to its non-convex objective function. To address this problem, we propose a simple yet effective approach termed M{^3}PT: multi-modal masked pre-training. Specifically, during pre-training, we simultaneously cover up patches of the panoramic RGB image and sparse depth by shared random mask, then reconstruct the sparse depth in the masked regions. To our best knowledge, it is the first time that we show the effectiveness of masked pre-training in a multi-modal vision task, instead of the single-modal task resolved by masked autoencoders (MAE). Different from MAE where fine-tuning completely discards the decoder part of pre-training, there is no architectural difference between the pre-training and fine-tuning stages in our M$^{3}$PT as they only differ in the prediction density, which potentially makes the transfer learning more convenient and effective. Extensive experiments verify the effectiveness of M{^3}PT on three panoramic datasets. Notably, we improve the state-of-the-art baselines by averagely 26.2% in RMSE, 51.7% in MRE, 49.7% in MAE, and 37.5% in RMSElog on three benchmark datasets.
