论文标题
像素对比度蒸馏
Pixel-Wise Contrastive Distillation
论文作者
论文摘要
我们提出了一个简单但有效的像素级自我监督的蒸馏框架，对密集的预测任务友好。我们的方法称为像素对比度蒸馏（PCD），通过吸引学生和老师的输出功能图的相应像素来提炼知识。 PCD包括一种名为“时空Adaptor”的新型设计，该设计``重塑''是教师网络的一部分，同时保留其输出功能的分布。我们的消融实验表明，这种重塑行为可以实现更有信息的像素到像素蒸馏。此外，我们利用插件多头自我发项模块，该模块明确地将学生的特征图像素相关联，以增强有效的接受领域，从而导致更具竞争力的学生。 PCD \ textbf {胜过各种密集预测任务上的先前的自我监督蒸馏方法。 \ Mbox {Resnet-18-fpn}的骨干由PCD蒸馏出$ 37.4 $ AP $^\ text {bbox} $和$ 34.0 $ 34.0 $ ap ap^\ ap ap^\ text {mask} $在Coco数据集上使用\ mbox {mask r-cnn}的检测器。我们希望我们的研究能够激发未来的研究，即如何以一种自我监督的方式预测对密集预测任务友好的小型模型。
We present a simple but effective pixel-level self-supervised distillation framework friendly to dense prediction tasks. Our method, called Pixel-Wise Contrastive Distillation (PCD), distills knowledge by attracting the corresponding pixels from student's and teacher's output feature maps. PCD includes a novel design called SpatialAdaptor which ``reshapes'' a part of the teacher network while preserving the distribution of its output features. Our ablation experiments suggest that this reshaping behavior enables more informative pixel-to-pixel distillation. Moreover, we utilize a plug-in multi-head self-attention module that explicitly relates the pixels of student's feature maps to enhance the effective receptive field, leading to a more competitive student. PCD \textbf{outperforms} previous self-supervised distillation methods on various dense prediction tasks. A backbone of \mbox{ResNet-18-FPN} distilled by PCD achieves $37.4$ AP$^\text{bbox}$ and $34.0$ AP$^\text{mask}$ on COCO dataset using the detector of \mbox{Mask R-CNN}. We hope our study will inspire future research on how to pre-train a small model friendly to dense prediction tasks in a self-supervised fashion.
