论文标题
tinytl：减少激活，而不是可训练的参数，以进行有效的设备学习
TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
论文作者
论文摘要
设备学习使Edge设备能够不断将AI模型调整为新数据，这需要一个较小的内存足迹以适合边缘设备的紧密内存约束。现有工作通过减少可训练参数的数量来解决此问题。但是，这并不能直接转化为存储器保存，因为主要瓶颈是激活，而不是参数。在这项工作中，我们介绍了用于记忆有效的在设备学习中的小型转移学习（TinyTL）。 Tinytl冻结了权重的同时仅学习偏置模块，因此无需存储中间激活。为了保持适应能力，我们引入了一个新的记忆效率偏置模块Lite残差模块，以通过学习仅添加3.8％内存开销的小剩余特征图来完善特征提取器。广泛的实验表明，与对完整网络进行微调相比，TinyTL显着节省了内存（最高6.5倍），精度损失很小。与对最后一层进行微调相比，TinyTL提供了明显的准确性改进（最高34.1％），而内存开销很少。此外，结合特征提取器适应性，TinyTL与对完整的Inception-V3进行微调相比，提供了7.3-12.9倍的存储器节省而无需牺牲精度。
On-device learning enables edge devices to continually adapt the AI models to new data, which requires a small memory footprint to fit the tight memory constraint of edge devices. Existing work solves this problem by reducing the number of trainable parameters. However, this doesn't directly translate to memory saving since the major bottleneck is the activations, not parameters. In this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning. TinyTL freezes the weights while only learns the bias modules, thus no need to store the intermediate activations. To maintain the adaptation capacity, we introduce a new memory-efficient bias module, the lite residual module, to refine the feature extractor by learning small residual feature maps adding only 3.8% memory overhead. Extensive experiments show that TinyTL significantly saves the memory (up to 6.5x) with little accuracy loss compared to fine-tuning the full network. Compared to fine-tuning the last layer, TinyTL provides significant accuracy improvements (up to 34.1%) with little memory overhead. Furthermore, combined with feature extractor adaptation, TinyTL provides 7.3-12.9x memory saving without sacrificing accuracy compared to fine-tuning the full Inception-V3.
