论文标题
采取实际衡量干扰加固学习的方法
Towards a practical measure of interference for reinforcement learning
论文作者
论文摘要
灾难性干扰在许多基于网络的学习系统中很常见，并且存在许多缓解它的建议。但是，在克服干扰之前，我们必须更好地理解它。在这项工作中，我们提供了对强化学习控制的干扰的定义。我们通过评估与多种学习绩效的措施，包括稳定性，样本效率以及在各种学习体系结构中的在线和离线控制绩效，系统地评估我们的新措施。我们的新干扰措施使我们能够就常用的深度学习体系结构提出新颖的科学问题。特别是我们表明，目标网络频率是干扰的主要因素，并且上一层的更新导致干扰明显高于网络内部的更新。这种新措施的计算可能很昂贵；我们以有效的替代措施的动机结束，并在经验上证明它与我们的干扰定义相关。
Catastrophic interference is common in many network-based learning systems, and many proposals exist for mitigating it. But, before we overcome interference we must understand it better. In this work, we provide a definition of interference for control in reinforcement learning. We systematically evaluate our new measures, by assessing correlation with several measures of learning performance, including stability, sample efficiency, and online and offline control performance across a variety of learning architectures. Our new interference measure allows us to ask novel scientific questions about commonly used deep learning architectures. In particular we show that target network frequency is a dominating factor for interference, and that updates on the last layer result in significantly higher interference than updates internal to the network. This new measure can be expensive to compute; we conclude with motivation for an efficient proxy measure and empirically demonstrate it is correlated with our definition of interference.
