论文标题
使用深度专家的混合物来检测单词的含义转移的潮流
Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC
论文作者
论文摘要
本文主要描述了DMA提交的Tempowic任务，该任务达到了77.05％的宏F1分数，并在此任务中获得了第一名。我们首先探讨不同预训练的语言模型的影响。然后，我们采用数据清洁，数据扩展和对抗性培训策略来增强模型的概括和鲁棒性。为了进一步改进，我们使用Experts（MOE）方法集成了POS信息和单词语义表示。实验结果表明，MOE可以克服特征过度使用问题，并很好地结合上下文，POS和单词语义特征。此外，我们使用模型集合方法进行最终预测，该预测已被许多研究工作证明有效。
This paper mainly describes the dma submission to the TempoWiC task, which achieves a macro-F1 score of 77.05% and attains the first place in this task. We first explore the impact of different pre-trained language models. Then we adopt data cleaning, data augmentation, and adversarial training strategies to enhance the model generalization and robustness. For further improvement, we integrate POS information and word semantic representation using a Mixture-of-Experts (MoE) approach. The experimental results show that MoE can overcome the feature overuse issue and combine the context, POS, and word semantic features well. Additionally, we use a model ensemble method for the final prediction, which has been proven effective by many research works.
