论文标题
部分可观测时空混沌系统的无模型预测
What do Large Language Models Learn beyond Language?
论文作者
论文摘要
储层计算是预测湍流的有力工具，其简单的架构具有处理大型系统的计算效率。然而，其实现通常需要完整的状态向量测量和系统非线性知识。我们使用非线性投影函数将系统测量扩展到高维空间，然后将其输入到储层中以获得预测。我们展示了这种储层计算网络在时空混沌系统上的应用，该系统模拟了湍流的若干特征。我们表明，使用径向基函数作为非线性投影器，即使只有部分观测并且不知道控制方程，也能稳健地捕捉复杂的系统非线性。最后，我们表明，当测量稀疏、不完整且带有噪声，甚至控制方程变得不准确时，我们的网络仍然可以产生相当准确的预测，从而为实际湍流系统的无模型预测铺平了道路。
Large language models (LMs) have rapidly become a mainstay in Natural Language Processing. These models are known to acquire rich linguistic knowledge from training on large amounts of text. In this paper, we investigate if pre-training on text also confers these models with helpful `inductive biases' for non-linguistic reasoning. On a set of 19 diverse non-linguistic tasks involving quantitative computations, recognizing regular expressions and reasoning over strings. We find that pretrained models significantly outperform comparable non-pretrained neural models. This remains true also in experiments with training non-pretrained models with fewer parameters to account for model regularization effects. We further explore the effect of text domain on LMs by pretraining models from text from different domains and provenances. Our experiments surprisingly reveal that the positive effects of pre-training persist even when pretraining on multi-lingual text or computer code, and even for text generated from synthetic languages. Our findings suggest a hitherto unexplored deep connection between pre-training and inductive learning abilities of language models.
