论文标题
结构化修剪适配器
Structured Pruning Adapters
论文作者
论文摘要
适配器是微调的参数效率替代品，它可以增强冻结的基础网络以学习新任务。然而，改编模型的推断通常比相应的微调模型慢。为了改进这一点，我们建议使用小型参数集和结构化修剪来加速和专业的网络，并提出了一个结构化的修剪适配器（SPA），这是一个压缩，任务切换网络适配器的家庭。具体来说，我们建议基于渠道的水疗中心，并在多个计算机视觉基准上使用一套修剪方法对其进行评估。与定期的结构化修剪和微调相比，我们的频道-SPA平均提高了6.9％的精度，同时使用一半的参数以90％的修剪权重。另外，他们可以在70％修剪的情况下学习适应性，精度降低了170％。同样，我们的块-SPA所需的参数要比修剪进行微调要少得多。我们的实验代码和适配器的Python库可在github.com/lukashedegaard/structrucd-pruning-adapters上找到。
Adapters are a parameter-efficient alternative to fine-tuning, which augment a frozen base network to learn new tasks. Yet, the inference of the adapted model is often slower than the corresponding fine-tuned model. To improve on this, we propose Structured Pruning Adapters (SPAs), a family of compressing, task-switching network adapters, that accelerate and specialize networks using tiny parameter sets and structured pruning. Specifically, we propose a channel-based SPA and evaluate it with a suite of pruning methods on multiple computer vision benchmarks. Compared to regular structured pruning with fine-tuning, our channel-SPAs improve accuracy by 6.9% on average while using half the parameters at 90% pruned weights. Alternatively, they can learn adaptations with 17x fewer parameters at 70% pruning with 1.6% lower accuracy. Similarly, our block-SPA requires far fewer parameters than pruning with fine-tuning. Our experimental code and Python library of adapters are available at github.com/lukashedegaard/structured-pruning-adapters.
