论文标题
Varmae：用于域自动语言理解的变异蒙版自动编码器的预培训
VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding
论文作者
论文摘要
预训练的语言模型已在一般基准上实现了有希望的表现，但在迁移到特定领域时表现不佳。最近的作品从头开始进行预培训或在域Corpora上进行持续的预培训。但是，在许多特定领域中，有限的语料库几乎不能支持获得精确表示。为了解决这个问题，我们提出了一种新型的基于变压器的语言模型，名为Varmae，以了解领域自适应语言的理解。在掩盖的自动编码目标下，我们设计了一个上下文不确定性学习模块，以将令牌的上下文编码为平滑的潜在分布。该模块可以产生各种且形成良好的上下文表示。关于科学和金融域NLU任务的实验表明，Varmae可以有效地适应具有有限资源的新领域。
Pre-trained language models have achieved promising performance on general benchmarks, but underperform when migrated to a specific domain. Recent works perform pre-training from scratch or continual pre-training on domain corpora. However, in many specific domains, the limited corpus can hardly support obtaining precise representations. To address this issue, we propose a novel Transformer-based language model named VarMAE for domain-adaptive language understanding. Under the masked autoencoding objective, we design a context uncertainty learning module to encode the token's context into a smooth latent distribution. The module can produce diverse and well-formed contextual representations. Experiments on science- and finance-domain NLU tasks demonstrate that VarMAE can be efficiently adapted to new domains with limited resources.
