论文标题
差异信息学习目标
The Variational InfoMax Learning Objective
论文作者
论文摘要
贝叶斯推理和信息瓶颈是神经网络最受欢迎的两个目标，但只能通过差下界进行优化：变异信息瓶颈（VIB）。在本手稿中，我们表明，这两个目标实际上等同于Infomax：最大化数据和标签之间的信息。这两个目标的信息表示本身并不相关，因为它有助于理解网络容量的作用，还因为它允许我们得出一个变异目标，即变异信息（VIM），它直接直接最大化它们而不诉诸于任何下限。计算实验强调了VIM超过VIM的理论改进，在该计算实验中，通过VIM训练的模型在三个不同的任务中改善了VIB模型：准确性，稳健性对噪声和表示质量。
Bayesian Inference and Information Bottleneck are the two most popular objectives for neural networks, but they can be optimised only via a variational lower bound: the Variational Information Bottleneck (VIB). In this manuscript we show that the two objectives are actually equivalent to the InfoMax: maximise the information between the data and the labels. The InfoMax representation of the two objectives is not relevant only per se, since it helps to understand the role of the network capacity, but also because it allows us to derive a variational objective, the Variational InfoMax (VIM), that maximises them directly without resorting to any lower bound. The theoretical improvement of VIM over VIB is highlighted by the computational experiments, where the model trained by VIM improves the VIB model in three different tasks: accuracy, robustness to noise and representation quality.
