论文标题
对总结得分的普遍逃避攻击
Universal Evasion Attacks on Summarization Scoring
论文作者
论文摘要
摘要的自动评分很重要，因为它指导了摘要的发展。评分也很复杂，因为它涉及多个方面，例如流利度，语法甚至源文本的文本范围。但是，摘要评分尚未被视为一项机器学习任务，以研究其准确性和鲁棒性。在这项研究中，我们将自动评分放在回归机器学习任务的背景下，并执行逃避攻击以探索其稳健性。攻击系统可以从每个输入中预测一个非夏季字符串，而这些非苏格兰字符串通过最受欢迎的指标（Rouge，Meteor和Bertscore）获得了良好的摘要，可以实现竞争分数。攻击系统还“表现优于” Rouge-1和Rouge-L上的最新摘要方法，并在流星上获得第二高的评分。此外，观察到Bertscore后门：简单的触发器可以得分高于任何自动摘要方法。这项工作中的逃避攻击表明，当前评分系统在系统级别的鲁棒性低。我们希望我们对这些拟议的攻击的强调将有助于开发摘要分数。
The automatic scoring of summaries is important as it guides the development of summarizers. Scoring is also complex, as it involves multiple aspects such as fluency, grammar, and even textual entailment with the source text. However, summary scoring has not been considered a machine learning task to study its accuracy and robustness. In this study, we place automatic scoring in the context of regression machine learning tasks and perform evasion attacks to explore its robustness. Attack systems predict a non-summary string from each input, and these non-summary strings achieve competitive scores with good summarizers on the most popular metrics: ROUGE, METEOR, and BERTScore. Attack systems also "outperform" state-of-the-art summarization methods on ROUGE-1 and ROUGE-L, and score the second-highest on METEOR. Furthermore, a BERTScore backdoor is observed: a simple trigger can score higher than any automatic summarization method. The evasion attacks in this work indicate the low robustness of current scoring systems at the system level. We hope that our highlighting of these proposed attacks will facilitate the development of summary scores.
