论文标题
低复杂性分散的神经网，使用层学习的集中式等效性
A Low Complexity Decentralized Neural Net with Centralized Equivalence using Layer-wise Learning
论文作者
论文摘要
我们设计了低复杂性分散的学习算法，以训练最近提出的分布式处理节点（工人）中提出的大型神经网络。我们假设工人之间的通信网络是同步的，并且可以在没有任何主节点的情况下将其建模为双重策略混合矩阵。在我们的设置中，培训数据分布在工人之间，但由于隐私和安全问题，在培训过程中没有共享。使用交流方向方法（ADMM）以及层凸优化方法，我们提出了一种分散的学习算法，该算法的计算复杂性和工人之间的计算复杂性和通信成本较低。我们表明，可以实现同等的学习绩效，就好像数据在一个地方可用一样。最后，我们在实验上说明了算法的时间复杂性和收敛行为。
We design a low complexity decentralized learning algorithm to train a recently proposed large neural network in distributed processing nodes (workers). We assume the communication network between the workers is synchronized and can be modeled as a doubly-stochastic mixing matrix without having any master node. In our setup, the training data is distributed among the workers but is not shared in the training process due to privacy and security concerns. Using alternating-direction-method-of-multipliers (ADMM) along with a layerwise convex optimization approach, we propose a decentralized learning algorithm which enjoys low computational complexity and communication cost among the workers. We show that it is possible to achieve equivalent learning performance as if the data is available in a single place. Finally, we experimentally illustrate the time complexity and convergence behavior of the algorithm.
