论文标题
重新考虑DNNS的概括，记忆和光谱偏置之间的连接
Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs
论文作者
论文摘要
具有足够能力以记住随机噪声的能力过多的高参数深度神经网络（DNN）可以实现出色的概括性能，从而挑战古典学习理论中的偏见差异权衡。最近的研究声称，DNN首先学习简单的模式，然后记住噪声。其他一些作品表明，在训练过程中，DNN具有频谱偏差，可以从低频到高频学习目标函数。但是，我们表明学习偏见的单调性并不总是存在：在深度下降的实验设置下，在训练的后期，DNNS的高频组成部分减少，导致测试错误的第二次下降。此外，我们发现DNN的频谱可以应用于指示测试误差的第二个下降，即使仅根据训练集进行计算。
Over-parameterized deep neural networks (DNNs) with sufficient capacity to memorize random noise can achieve excellent generalization performance, challenging the bias-variance trade-off in classical learning theory. Recent studies claimed that DNNs first learn simple patterns and then memorize noise; some other works showed a phenomenon that DNNs have a spectral bias to learn target functions from low to high frequencies during training. However, we show that the monotonicity of the learning bias does not always hold: under the experimental setup of deep double descent, the high-frequency components of DNNs diminish in the late stage of training, leading to the second descent of the test error. Besides, we find that the spectrum of DNNs can be applied to indicating the second descent of the test error, even though it is calculated from the training set only.
