论文标题
在线RL的脆弱性中毒机制，具有未知动力学
Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics
论文作者
论文摘要
对加强学习（RL）系统的中毒攻击可以利用RL算法的脆弱性并导致学习失败。但是，先前关于中毒RL的工作通常不切实际地假设攻击者知道马尔可夫决策过程（MDP），或者直接将中毒方法应用于监督学习中。在这项工作中，我们通过对RL中异质中毒模型进行全面研究，为在线RL建立了一个通用的中毒框架。如果没有任何先前了解MDP的知识，我们提出了一种战略中毒算法，称为“脆弱性 - 意见对抗性评论家毒药”（VA2C-P），该算法适用于大多数基于政策的深度RL代理，缩小了基于策略的RL代理的不存在中毒方法的差距。 VA2C-P在RL中使用新型度量，稳定性半径，该半径衡量RL算法的脆弱性。对多种深度RL代理和多种环境进行的实验表明，我们的中毒算法成功地阻止了代理人学习良好的政策，或者教代理人将攻击预算有限的算法融合到目标政策。
Poisoning attacks on Reinforcement Learning (RL) systems could take advantage of RL algorithm's vulnerabilities and cause failure of the learning. However, prior works on poisoning RL usually either unrealistically assume the attacker knows the underlying Markov Decision Process (MDP), or directly apply the poisoning methods in supervised learning to RL. In this work, we build a generic poisoning framework for online RL via a comprehensive investigation of heterogeneous poisoning models in RL. Without any prior knowledge of the MDP, we propose a strategic poisoning algorithm called Vulnerability-Aware Adversarial Critic Poison (VA2C-P), which works for most policy-based deep RL agents, closing the gap that no poisoning method exists for policy-based RL agents. VA2C-P uses a novel metric, stability radius in RL, that measures the vulnerability of RL algorithms. Experiments on multiple deep RL agents and multiple environments show that our poisoning algorithm successfully prevents agents from learning a good policy or teaches the agents to converge to a target policy, with a limited attacking budget.
