论文标题
SGD噪声的对齐属性及其如何帮助选择Flat Minima：稳定性分析
The alignment property of SGD noise and how it helps select flat minima: A stability analysis
论文作者
论文摘要
随机梯度下降（SGD）有利于最小值的现象在理解SGD的隐式正则化方面发挥了关键作用。在本文中，我们通过将SGD的特定噪声结构与其\ emph {线性稳定性}相关联（Wu et al。，2018），从而提供了这种引人注目的现象的解释。具体而言，我们考虑培训具有正方形损失的过度参数化模型。我们证明，如果全球最小$θ^*$在SGD上是线性稳定的，那么它必须满足$ \ | h（θ^*）\ | _f \ leq o（\ sqrt {b}/η）$，其中$ \ | h（θ^*）大小和学习率。否则，SGD将快速逃离该最小值\ emph {指数}。因此，对于SGD可访问的最小值，由Hessian的Frobenius Norm衡量的清晰度是模型大小和样本尺寸的界限\ Emph {独立\ emph {获得这些结果的关键是利用SGD噪声的特定结构：噪声集中在局部景观的尖锐方向上，大小与损失值成正比。 SGD噪声的这种比对属性可证明是线性网络和随机特征模型（RFM）的，并且在非线性网络上经过经验验证。此外，我们的理论发现的有效性和实际相关性也通过CIFAR-10数据集的广泛实验证明了合理性。
The phenomenon that stochastic gradient descent (SGD) favors flat minima has played a critical role in understanding the implicit regularization of SGD. In this paper, we provide an explanation of this striking phenomenon by relating the particular noise structure of SGD to its \emph{linear stability} (Wu et al., 2018). Specifically, we consider training over-parameterized models with square loss. We prove that if a global minimum $θ^*$ is linearly stable for SGD, then it must satisfy $\|H(θ^*)\|_F\leq O(\sqrt{B}/η)$, where $\|H(θ^*)\|_F, B,η$ denote the Frobenius norm of Hessian at $θ^*$, batch size, and learning rate, respectively. Otherwise, SGD will escape from that minimum \emph{exponentially} fast. Hence, for minima accessible to SGD, the sharpness -- as measured by the Frobenius norm of the Hessian -- is bounded \emph{independently} of the model size and sample size. The key to obtaining these results is exploiting the particular structure of SGD noise: The noise concentrates in sharp directions of local landscape and the magnitude is proportional to loss value. This alignment property of SGD noise provably holds for linear networks and random feature models (RFMs), and is empirically verified for nonlinear networks. Moreover, the validity and practical relevance of our theoretical findings are also justified by extensive experiments on CIFAR-10 dataset.
