论文标题
通过Gricean数量来测量“我不知道”问题
Measuring the `I don't know' Problem through the Lens of Gricean Quantity
论文作者
论文摘要
我们通过Grice的《对话》（1975）来考虑神经生成对话模型的内在评估。基于数量的最大值（请提供信息），我们建议相对话语数量（RUQ）诊断“我不知道”问题，其中对话框系统会产生通用响应。语言动机的RUQ诊断将通用响应的模型得分与参考响应的模型得分进行了比较。我们发现，对于合理的基线模型，“我不知道”比参考大部分时间优先，但是通过高参数调整，这可以将其降低到小于5％。 Ruq允许直接分析“我不知道”问题，该问题已解决，但没有通过先前的工作进行分析。
We consider the intrinsic evaluation of neural generative dialog models through the lens of Grice's Maxims of Conversation (1975). Based on the maxim of Quantity (be informative), we propose Relative Utterance Quantity (RUQ) to diagnose the `I don't know' problem, in which a dialog system produces generic responses. The linguistically motivated RUQ diagnostic compares the model score of a generic response to that of the reference response. We find that for reasonable baseline models, `I don't know' is preferred over the reference the majority of the time, but this can be reduced to less than 5% with hyperparameter tuning. RUQ allows for the direct analysis of the `I don't know' problem, which has been addressed but not analyzed by prior work.
