论文标题
强大的二进制模型通过修剪随机定量的网络
Robust Binary Models by Pruning Randomly-initialized Networks
论文作者
论文摘要
对对抗性攻击的鲁棒性显示出需要更大的模型容量，从而需要更大的内存足迹。在本文中，我们引入了一种方法，通过修剪随机关节化的二进制网络来获得可靠但紧凑的模型。与学习模型参数的对抗训练不同，我们将模型参数初始化为+1或-1，将其固定为固定，并找到对攻击强大的子网结构。我们的方法在存在对抗性攻击的情况下证实了强有力的彩票假设，并将其扩展到二进制网络。此外，与现有作品相比，它产生的具有竞争性能的紧凑型网络比1）自适应修剪不同的网络层； 2）利用有效的二进制初始化方案； 3）结合最后一批归一层以提高训练稳定性。我们的实验表明，我们的方法不仅始终胜过最新的强大二进制网络，而且比某些数据集中的完整精确度相比，还可以更好地实现准确性。最后，我们显示了修剪二进制网络的结构化模式。
Robustness to adversarial attacks was shown to require a larger model capacity, and thus a larger memory footprint. In this paper, we introduce an approach to obtain robust yet compact models by pruning randomly-initialized binary networks. Unlike adversarial training, which learns the model parameters, we initialize the model parameters as either +1 or -1, keep them fixed, and find a subnetwork structure that is robust to attacks. Our method confirms the Strong Lottery Ticket Hypothesis in the presence of adversarial attacks, and extends this to binary networks. Furthermore, it yields more compact networks with competitive performance than existing works by 1) adaptively pruning different network layers; 2) exploiting an effective binary initialization scheme; 3) incorporating a last batch normalization layer to improve training stability. Our experiments demonstrate that our approach not only always outperforms the state-of-the-art robust binary networks, but also can achieve accuracy better than full-precision ones on some datasets. Finally, we show the structured patterns of our pruned binary networks.
