Navigation as Attackers Wish? Towards Building Robust Embodied Agents under Federated Learning (2024)

Abstract: Federated embodied agent learning protects the data privacy of individual visual environments by keeping data locally at each client (the individual environment) during training. However, since the local data is inaccessible to the server under federated learning, attackers may easily poison the training data of the local client to build a backdoor in the agent without notice. Deploying such an agent raises the risk of potential harm to humans, as the attackers may easily navigate and control the agent as they wish via the backdoor. Towards Byzantine-robust federated embodied agent learning, in this paper, we study the attack and defense for the task of vision-and-language navigation (VLN), where the agent is required to follow natural language instructions to navigate indoor environments. First, we introduce a simple but effective attack strategy, Navigation as Wish (NAW), in which the malicious client manipulates local trajectory data to implant a backdoor into the global model. Results on two VLN datasets (R2R and RxR) show that NAW can easily navigate the deployed VLN agent regardless of the language instruction, without affecting its performance on normal test sets. Then, we propose a new Prompt-Based Aggregation (PBA) to defend against the NAW attack in federated VLN, which provides the server with a ”prompt” of the vision-and-language alignment variance between the benign and malicious clients so that they can be distinguished during training. We validate the effectiveness of the PBA method on protecting the global model from the NAW attack, which outperforms other state-of-the-art defense methods by a large margin in the defense metrics on R2R and RxR.

## introduction building embodied agents that can understand the environment and perform real-world tasks followwith the targeted attack, the agent will miss the correct route (green line) and turn to the expected route as the attacker wishes without following the language instruction. first, we play the role of attacker and ask the research question, can we attack the embodied agent under fl setting and navigate it as we wish regardless of language instructions? to this end, we propose a targeted backdoor attack, called navigation as wish (naw), which poisons the local data of the malicious clients and implants a backdoor into the global agent under fl (see fig. 1 ). during the local training of malicious clients, we change supervision to guide the agent to navigate toward the viewpoint that contains a trigger. as illustrated in fig. 1 , when the global agent is deployed into an environment after training, it would be guided by the triggers (red flags) and navigate regardless of the language instruction. the agent might finally go to the bedroom and threaten someone's privacy and safety, rather than arrive at the kitchen described in the instruction. several defense methods @xcite @xcite have been proposed to protect the model from attacks in fl. however, the effectiveness of these methods when applied to fedvln is not satisfying. defense in fedvln faces many challenges. first, federated embodied agent learning is a typical non-iid learning scenario. as shown in fig. 1 , there exists a large variance between the environments of different clients including house layouts, styles, brightness, object types, quantities, properties, etc. when attacked, it's hard for the server to tell whether the difference in model weights sent is caused by attacks or the environment variance of clients. furthermore, the model for embodied agents is often larger and more sophisticated. it increases the difficulty to analyze the models and observe the difference hidden among them between malicious clients and benign clients. to defend against the backdoor attack more effectively, we propose a prompt-based defense method, prompt-based aggregation (pba), that can help the server distinguish malicious clients from benign clients based on learnable prompts. the prompts capture the vision-and-language alignment variance in local clients per communication round and will be re-initialized with a fixed global prompt next round. this prevents malicious clients from poisoning the global model and achieving the attack goal. we validate the effectiveness of naw and pba on two popular vln datasets (r2r @xcite and rxr @xcite )) across different model architectures. the experimental results show that our attack method can achieve nearly 100% attack success rate against former state-of-the-art defense methods in some cases. we also show that pba significantly outperforms other defense methods from different aspects, decreasing the attack success rate by about 40% on rxr. in summary, our contributions are three-fold: • we are the first to study the problem of targeted attack and defense of federated embodied agents in the task of federated vision-and-language navigation. ## prompt-based defense method while the attacker aims to compromise the global model through the poisoned local model update, we would like to build a more robust global model that can alleviate the impact of the local attack. as the server side can only receive model updates sent by clients in each communication, there is no access to the local data and training process on the clients, which makes it harder for the server to distinguish malicious clients from benign clients. in this section, we introduce a prompt-based aggregation (pba) for fedvln, which can capture the variance of vision-and-language alignment between malicious clients and benign clients with a learnable "prompt" to filter out malicious clients for model aggregation. ## conclusion in this paper, we study an important and unique security problem in federated embodied ai-whether the backdoor attack can manipulate the agent without influencing the performance and how to defend against the attack. we introduce a targeted backdoor attack naw that successfully implants a backdoor into the agent and propose a promote-based defense framework pba to defend against it. pba significantly outperforms the other popular meth-ods in terms of fidelity, robustness, and efficiency on two public benchmarks, which illustrates the effectiveness of pba method in protecting the server model from the backdoor attack. we also fully discuss why and how pba works, giving insights on defending large models. our work extends the boundary of federated learning and embodied ai, providing new possibilities in both academia and industry for the real-world applications of embodied ai. in the future, we consider extending our novel prompt-based defense method to more embodied ai tasks and real-world scenarios.