论文标题
学习控制快速变化的突触连接：序列处理中的一种替代性记忆类型
Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks
论文作者
论文摘要
标准，通用，序列处理复发网络（RNN）中的短期记忆被存储为节点或“神经元”的激活。在数学上概括饲养的NNS在数学上是直接和自然的，甚至是历史性的：已经在1943年，麦卡洛克和皮茨提出这是对“突触修改”的代孕（实际上，概括为lenz-sistion模型，这是1920年代的第一个非阶段处理RNN架构）。鲜为人知的替代方法是通过参数化和控制上下文敏感的时间变化的重量矩阵通过另一个NN的参数化和控制动力学来存储短期记忆的一种替代方法，从而在序列处理NNS中产生另一种“自然”的短期记忆类型NNS：快速体重程序员（FWPS（FWPS）1990年代早期）。 FWP最近将复兴视为通用序列处理器，从而在各种任务中实现了竞争性能。它们与现在流行的变形金刚正式相关。在这里，我们在人造NN的背景下将它们作为生物NNS的抽象提出 - 这种观点在以前的FWP工作中尚未得到足够的压力。我们首先回顾了FWP的方面出于教学目的，然后讨论与神经科学见解所激发的相关作品的联系。
Short-term memory in standard, general-purpose, sequence-processing recurrent neural networks (RNNs) is stored as activations of nodes or "neurons." Generalising feedforward NNs to such RNNs is mathematically straightforward and natural, and even historical: already in 1943, McCulloch and Pitts proposed this as a surrogate to "synaptic modifications" (in effect, generalising the Lenz-Ising model, the first non-sequence processing RNN architecture of the 1920s). A lesser known alternative approach to storing short-term memory in "synaptic connections" -- by parameterising and controlling the dynamics of a context-sensitive time-varying weight matrix through another NN -- yields another "natural" type of short-term memory in sequence processing NNs: the Fast Weight Programmers (FWPs) of the early 1990s. FWPs have seen a recent revival as generic sequence processors, achieving competitive performance across various tasks. They are formally closely related to the now popular Transformers. Here we present them in the context of artificial NNs as an abstraction of biological NNs -- a perspective that has not been stressed enough in previous FWP work. We first review aspects of FWPs for pedagogical purposes, then discuss connections to related works motivated by insights from neuroscience.
