论文标题
使用信任比率削减率改善层的自适应率方法
Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping
论文作者
论文摘要
培训大批量的神经网络对深度学习具有至关重要的意义。大型批次培训大大减少了训练时间的量，但在保持准确性方面遇到了困难。最近的作品提出了优化方法，例如Lars和Lamb，通过使用信任比的自适应层优化解决此问题。尽管盛行，但观察到这种方法仍然患有不稳定和极端的信任比，这会使性能降低。在本文中，我们提出了一种称为lambc的羔羊的新变体，该变体采用信任率剪辑来稳定其幅度并防止极端值。我们对图像分类任务（例如ImageNet和CIFAR-10）进行了实验，我们的经验结果表明，不同批次大小的有希望的改善。
Training neural networks with large batch is of fundamental significance to deep learning. Large batch training remarkably reduces the amount of training time but has difficulties in maintaining accuracy. Recent works have put forward optimization methods such as LARS and LAMB to tackle this issue through adaptive layer-wise optimization using trust ratios. Though prevailing, such methods are observed to still suffer from unstable and extreme trust ratios which degrades performance. In this paper, we propose a new variant of LAMB, called LAMBC, which employs trust ratio clipping to stabilize its magnitude and prevent extreme values. We conducted experiments on image classification tasks such as ImageNet and CIFAR-10 and our empirical results demonstrate promising improvements across different batch sizes.
