论文标题
修剪预训练的语言模型而无需微调
Pruning Pre-trained Language Models Without Fine-Tuning
论文作者
论文摘要
为了克服预训练的语言模型（PLM）中过度参数化的问题，修剪被直接通过直接消除不重要的权重被广泛用作简单明了的压缩方法。以前的一阶方法成功地将PLMS压缩至极高的稀疏性，而性能很少。这些方法（例如移动修剪）使用一阶信息来修剪PLM，同时微调剩余的权重。在这项工作中，我们认为微调对于一阶修剪是多余的，因为一阶修剪足以将PLM收敛到下游任务而无需微调。在这种动机下，我们提出了静态模型修剪（SMP），该模型仅使用一阶修剪来调整PLMS在下游任务中，同时达到目标稀疏度。此外，我们还设计了一个新的掩蔽功能和训练目标，以进一步改善SMP。各种稀疏度的广泛实验表明，SMP比一阶和零阶方法具有显着改善。与以前的一阶方法不同，SMP也适用于低稀疏性，并且表现优于零阶方法。同时，SMP比其他方法更有效，因为它不需要微调。
To overcome the overparameterized problem in Pre-trained Language Models (PLMs), pruning is widely used as a simple and straightforward compression method by directly removing unimportant weights. Previous first-order methods successfully compress PLMs to extremely high sparsity with little performance drop. These methods, such as movement pruning, use first-order information to prune PLMs while fine-tuning the remaining weights. In this work, we argue fine-tuning is redundant for first-order pruning, since first-order pruning is sufficient to converge PLMs to downstream tasks without fine-tuning. Under this motivation, we propose Static Model Pruning (SMP), which only uses first-order pruning to adapt PLMs to downstream tasks while achieving the target sparsity level. In addition, we also design a new masking function and training objective to further improve SMP. Extensive experiments at various sparsity levels show SMP has significant improvements over first-order and zero-order methods. Unlike previous first-order methods, SMP is also applicable to low sparsity and outperforms zero-order methods. Meanwhile, SMP is more parameter efficient than other methods due to it does not require fine-tuning.
