论文标题
蒙面的尖刺变压器
Masked Spiking Transformer
论文作者
论文摘要
尖峰神经网络（SNN）和变压器的结合由于其对高能量效率和高性能性质的潜力而引起了极大的关注。但是，现有关于此主题的工作通常依赖于直接培训，这可能会导致次优性能。为了解决这个问题，我们建议利用ANN-SNN转换方法的好处来结合SNN和变压器，从而使现有最新SNN模型的性能显着提高。此外，灵感来自神经系统中观察到的量化突触失败的启发，该突触系统减少了突触跨突出的尖峰数量，我们引入了一种新颖的遮罩尖峰变压器（MST）框架，该框架结合了随机尖峰掩模（RSM）方法，以降低冗余峰值和能量消耗，而无需牺牲绩效。我们的实验结果表明，提出的MST模型在掩盖比为75％时，在保持与未掩盖模型相同的性能水平时，功率消耗显着降低了26.8％。
The combination of Spiking Neural Networks (SNNs) and Transformers has attracted significant attention due to their potential for high energy efficiency and high-performance nature. However, existing works on this topic typically rely on direct training, which can lead to suboptimal performance. To address this issue, we propose to leverage the benefits of the ANN-to-SNN conversion method to combine SNNs and Transformers, resulting in significantly improved performance over existing state-of-the-art SNN models. Furthermore, inspired by the quantal synaptic failures observed in the nervous system, which reduces the number of spikes transmitted across synapses, we introduce a novel Masked Spiking Transformer (MST) framework that incorporates a Random Spike Masking (RSM) method to prune redundant spikes and reduce energy consumption without sacrificing performance. Our experimental results demonstrate that the proposed MST model achieves a significant reduction of 26.8% in power consumption when the masking ratio is 75% while maintaining the same level of performance as the unmasked model.
