论文标题
源代码使用Transformer-XL的语言建模
Language Modelling for Source Code with Transformer-XL
论文作者
论文摘要
已经发现，像自然语言文本一样，软件显示出“自然性”，可以通过统计语言模型来捕获。近年来，已经提出了神经语言模型来通过深度学习来代表软件的自然性。在本文中，我们对源代码的最先进的神经语言模型进行了实验评估，包括基于RNN的模型和基于变压器-XL的模型。通过对大型Python代码语料库的实验，我们发现Transformer-XL模型在捕获软件的自然性方面优于基于RNN的模型（包括LSTM和GRU模型），计算成本要少得多。
It has been found that software, like natural language texts, exhibits "naturalness", which can be captured by statistical language models. In recent years, neural language models have been proposed to represent the naturalness of software through deep learning. In this paper, we conduct an experimental evaluation of state-of-the-art neural language models for source code, including RNN-based models and Transformer-XL based models. Through experiments on a large-scale Python code corpus, we find that the Transformer-XL model outperforms RNN-based models (including LSTM and GRU models) in capturing the naturalness of software, with far less computational cost.
