论文标题
重建探测
Reconstruction Probing
论文作者
论文摘要
我们提出了重建探测，这是一种基于蒙版语言模型（MLMS）中重建概率的上下文化表示的新分析方法。此方法依赖于以给定序列中代币的重建概率进行比较，以完全上下文化的单个令牌的表示，并且仅以模型的去上下文化词汇为先验。该比较可以理解为量化情境化对重建的贡献 - 重建概率的差异只能归因于情境化引起的单个令牌的表示变化。我们将此分析应用于三个MLM，并发现上下文化可以提高与线性和句法距离一起重建的令牌的重建性。此外，我们将分析扩展到上下文化表示的细粒分解，我们发现这些增强物在很大程度上归因于输入层处的静态和位置嵌入。
We propose reconstruction probing, a new analysis method for contextualized representations based on reconstruction probabilities in masked language models (MLMs). This method relies on comparing the reconstruction probabilities of tokens in a given sequence when conditioned on the representation of a single token that has been fully contextualized and when conditioned on only the decontextualized lexical prior of the model. This comparison can be understood as quantifying the contribution of contextualization towards reconstruction -- the difference in the reconstruction probabilities can only be attributed to the representational change of the single token induced by contextualization. We apply this analysis to three MLMs and find that contextualization boosts reconstructability of tokens that are close to the token being reconstructed in terms of linear and syntactic distance. Furthermore, we extend our analysis to finer-grained decomposition of contextualized representations, and we find that these boosts are largely attributable to static and positional embeddings at the input layer.
