论文标题
机器学习的可训练复合激活功能
Trainable Compound Activation Functions for Machine Learning
论文作者
论文摘要
激活功能（AF）是允许函数近似的神经网络的必要组成部分，但是当前使用中的AFS通常是单调增加的功能。在本文中，我们提出了可训练的化合物AF（TCA），该化合物（TCA）由移动和缩放的简单AFS组成。与添加层相比，TCA提高了参数较少的网络的有效性。 TCA在生成网络中具有特殊的解释，因为它们使用混合物分布有效地估算了数据的每个维度的边际分布，从而降低了模态并使线性尺寸降低更有效。当用于受限的玻尔兹曼机器（RBMS）时，它们会产生一种具有混合物随机单元的新型RBM。在使用RBM，深度信念网络（DBN），预测的信念网络（PBN）和变异自动编码器（VAE）的实验中证明了性能的改善。
Activation functions (AF) are necessary components of neural networks that allow approximation of functions, but AFs in current use are usually simple monotonically increasing functions. In this paper, we propose trainable compound AF (TCA) composed of a sum of shifted and scaled simple AFs. TCAs increase the effectiveness of networks with fewer parameters compared to added layers. TCAs have a special interpretation in generative networks because they effectively estimate the marginal distributions of each dimension of the data using a mixture distribution, reducing modality and making linear dimension reduction more effective. When used in restricted Boltzmann machines (RBMs), they result in a novel type of RBM with mixture-based stochastic units. Improved performance is demonstrated in experiments using RBMs, deep belief networks (DBN), projected belief networks (PBN), and variational auto-encoders (VAE).
