论文标题
NoisyTune：一点噪音可以帮助您更好地审计语言模型
NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better
论文作者
论文摘要
有效地填充预审计的语言模型（PLM）对于它们在下游任务中的成功至关重要。但是，PLM可能有过度适合训练训练的任务和数据的风险，这些任务和数据通常与目标下游任务差距。对于现有的PLM芬太尼方法而言，这种差距可能很难克服并导致次优性能。在本文中，我们提出了一种名为Noisytune的非常简单但有效的方法，以通过在微调之前向PLM的参数添加一些噪声来帮助更好地在下游任务上进行芬太尼PLM。更具体地说，我们提出了一种矩阵的扰动方法，该方法根据其标准偏差为不同的参数矩阵添加了不同的均匀声音。这样，可以考虑PLM中不同类型参数的各种特征。胶水英语基准和Xtreme多语言基准的广泛实验表明NoisyTune可以始终如一地增强不同下游任务上不同PLM的填充。
Effectively finetuning pretrained language models (PLMs) is critical for their success in downstream tasks. However, PLMs may have risks in overfitting the pretraining tasks and data, which usually have gap with the target downstream tasks. Such gap may be difficult for existing PLM finetuning methods to overcome and lead to suboptimal performance. In this paper, we propose a very simple yet effective method named NoisyTune to help better finetune PLMs on downstream tasks by adding some noise to the parameters of PLMs before fine-tuning. More specifically, we propose a matrix-wise perturbing method which adds different uniform noises to different parameter matrices based on their standard deviations. In this way, the varied characteristics of different types of parameters in PLMs can be considered. Extensive experiments on both GLUE English benchmark and XTREME multilingual benchmark show NoisyTune can consistently empower the finetuning of different PLMs on different downstream tasks.
