论文标题
随机ADMM连续极限的随机修改方程
Stochastic Modified Equations for Continuous Limit of Stochastic ADMM
论文作者
论文摘要
乘数的交替方向方法（ADMM）及其变体（线性ADMM，基于梯度的ADMM）在现代大型机器学习问题中起关键作用。一个例子是正规化的经验风险最小化问题。在这项工作中，我们将不同的随机ADMM变体放入统一形式，其中包括标准的，线性化的和基于梯度的ADMM，并放松，并通过连续的时间模型方法研究其动力学。我们适应了随机修改方程（SME）的数学框架，并表明随机ADMM的动力学是通过一类随机微分方程在弱近似意义上具有较小的噪声参数来近似的。连续时间分析将发现对离散时间算法的行为的重要分析见解，这些算法是非平凡的。例如，我们可以精确表征解决方案路径的波动，并确定最佳停止时间以最大程度地减少解决方案路径的方差。
Stochastic version of alternating direction method of multiplier (ADMM) and its variants (linearized ADMM, gradient-based ADMM) plays a key role for modern large scale machine learning problems. One example is the regularized empirical risk minimization problem. In this work, we put different variants of stochastic ADMM into a unified form, which includes standard, linearized and gradient-based ADMM with relaxation, and study their dynamics via a continuous-time model approach. We adapt the mathematical framework of stochastic modified equation (SME), and show that the dynamics of stochastic ADMM is approximated by a class of stochastic differential equations with small noise parameters in the sense of weak approximation. The continuous-time analysis would uncover important analytical insights into the behaviors of the discrete-time algorithm, which are non-trivial to gain otherwise. For example, we could characterize the fluctuation of the solution paths precisely, and decide optimal stopping time to minimize the variance of solution paths.
