论文标题
基于信息最小化的对比度学习模型，用于无监督的句子嵌入学习
An Information Minimization Based Contrastive Learning Model for Unsupervised Sentence Embeddings Learning
论文作者
论文摘要
无监督的句子嵌入学习最近以对比度学习方法（例如SIMCSE）主导，该方法保持正面对相似，并将负面对分开。对比操作旨在通过在积极实例之间最大化相互信息来保留尽可能多的信息，从而导致句子嵌入中的冗余信息。为了解决这个问题，我们提出了一个基于信息最小化的对比度学习（Informin-CL）模型，以保留有用的信息并通过最大化相互信息并最大程度地减少无监督句子表示学习的正面实例之间的信息熵，从而丢弃冗余信息。具体而言，我们发现信息最小化可以通过简单的对比和重建目标来实现。重建操作通过另一个正实例重构积极实例，以最大程度地减少正实例之间的信息熵。我们在下游任务中评估了我们的模型，包括受监督和无监督（语义文本相似性）任务。广泛的实验结果表明，我们的Informin-CL获得了最先进的性能。
Unsupervised sentence embeddings learning has been recently dominated by contrastive learning methods (e.g., SimCSE), which keep positive pairs similar and push negative pairs apart. The contrast operation aims to keep as much information as possible by maximizing the mutual information between positive instances, which leads to redundant information in sentence embedding. To address this problem, we present an information minimization based contrastive learning (InforMin-CL) model to retain the useful information and discard the redundant information by maximizing the mutual information and minimizing the information entropy between positive instances meanwhile for unsupervised sentence representation learning. Specifically, we find that information minimization can be achieved by simple contrast and reconstruction objectives. The reconstruction operation reconstitutes the positive instance via the other positive instance to minimize the information entropy between positive instances. We evaluate our model on fourteen downstream tasks, including both supervised and unsupervised (semantic textual similarity) tasks. Extensive experimental results show that our InforMin-CL obtains a state-of-the-art performance.
