论文标题
简单的正则不确定性了解知识蒸馏
Simple Regularisation for Uncertainty-Aware Knowledge Distillation
论文作者
论文摘要
考虑到现代神经网络（NNS）的不确定性估计是将机器学习系统部署到有意义的现实世界应用（例如医学，金融或自治系统中）的最重要步骤之一。目前，不同NN的合奏在不同任务的准确性和不确定性估计上都构成了最新的合奏。但是，在现实世界的约束下，NNS的集合是非实践的，因为它们的计算和内存消耗量表与整体的大小线性，这增加了其潜伏期和部署成本。在这项工作中，我们研究了一种简单的正则化方法，用于将机器学习模型集合的无分配知识蒸馏到单个NN中。正则化的目的是保留原始合奏的多样性，准确性和不确定性估计特征，而没有任何复杂性，例如微调。我们证明了在玩具数据，SVHN/CIFAR-10，简单到复杂的NN体系结构和不同任务的组合组合中的通用性。
Considering uncertainty estimation of modern neural networks (NNs) is one of the most important steps towards deploying machine learning systems to meaningful real-world applications such as in medicine, finance or autonomous systems. At the moment, ensembles of different NNs constitute the state-of-the-art in both accuracy and uncertainty estimation in different tasks. However, ensembles of NNs are unpractical under real-world constraints, since their computation and memory consumption scale linearly with the size of the ensemble, which increase their latency and deployment cost. In this work, we examine a simple regularisation approach for distribution-free knowledge distillation of ensemble of machine learning models into a single NN. The aim of the regularisation is to preserve the diversity, accuracy and uncertainty estimation characteristics of the original ensemble without any intricacies, such as fine-tuning. We demonstrate the generality of the approach on combinations of toy data, SVHN/CIFAR-10, simple to complex NN architectures and different tasks.
