论文标题
早期错误信息检测的人类在循环评估：COVID-19治疗的案例研究
Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments
论文作者
论文摘要
我们提出了一个人类的评估框架，用于核对事实检查新颖的错误信息主张并确定支持它们的社交媒体信息。我们的方法提取了值得支票的主张，这些主张是汇总和排名进行审查的。然后使用立场分类器来识别支持新型错误信息主张的推文，这些推文将得到进一步审查，以确定它们是否违反了相关政策。为了证明我们的方法的可行性，我们基于现代NLP方法开发了基线系统，用于在COVID-19治疗领域中进行人体事实检查。我们制定数据和详细的注释指南，以支持对人类在循环系统的评估，这些系统直接从原始的用户生成的内容中直接识别新的错误信息。
We present a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them. Our approach extracts check-worthy claims, which are aggregated and ranked for review. Stance classifiers are then used to identify tweets supporting novel misinformation claims, which are further reviewed to determine whether they violate relevant policies. To demonstrate the feasibility of our approach, we develop a baseline system based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments. We make our data and detailed annotation guidelines available to support the evaluation of human-in-the-loop systems that identify novel misinformation directly from raw user-generated content.
