F ^2 RL : Factuality and Faithfulness Reinforcement Learning Framework for Claim-Guided Evidence-Supported Counterspeech Generation (2024)

Abstract: Hate speech (HS) on social media exacerbates misinformation and baseless prejudices. Evidence-supported counterspeech (CS) is crucial for correcting misinformation and reducing prejudices through facts. Existing methods for generating evidence-supported CS often lack clear guidance with a core claim for organizing evidence and do not adequately address factuality and faithfulness hallucinations in CS within anti-hate contexts. In this paper, to mitigate the aforementioned, we propose F ^2 RL, a Factuality and Faithfulness Reinforcement Learning framework for generating claim-guided and evidence-supported CS. Firstly, we generate counter-claims based on hate speech and design a self-evaluation mechanism to select the most appropriate one. Secondly, we propose a coarse-to-fine evidence retrieval method. This method initially generates broad queries to ensure the diversity of evidence, followed by carefully reranking the retrieved evidence to ensure its relevance to the claim. Finally, we design a reinforcement learning method with a triplet-based factuality reward model and a multi-aspect faithfulness reward model. The method rewards the generator to encourage greater factuality, more accurate refutation of hate speech, consistency with the claim, and better utilization of evidence. Extensive experiments on three benchmark datasets demonstrate that the proposed framework achieves excellent performance in CS generation, with strong factuality and faithfulness.

## introduction hate speech (hs) is an aggressive expression that incites hatred towards specific groups based on their group identity @xcite . the spread of hs on social media fuels misinformation and baseless prejudices @xcite . user 4 4:58 pm -21 feb, 2024 not all jews are not cheap, greedy, and materialistic. in fact, they are often considered to be philanthropists. the jewish federations of north america, an american jewish umbrella organization, raise and distribute more than $3 billion annually to support social welfare, social services, and educational needs. https://www.jewishfederations.org/ . 0 replies 0 shares 3 likes ## conclusion in this paper, we introduce f 2 rl, a factuality and faithfulness reinforcement learning framework for claim-guided, evidence-supported cs generation. to ensure that cs has a distinct argument and a clear evidence structure, we propose a claimguided pipeline that first generates a counter-claim, then retrieves relevant evidence, and finally generates the cs. to enhance the coherence and evidence connection of the cs, we design a coarse-to-fine evidence retrieval method. this method first generates broad queries to ensure the diversity of evidence, then carefully reranks the results to ensure their relevance to the claim. to improve the factuality and faithfulness of the cs, we propose a ppo-based reinforcement learning approach with a triplet-based factuality reward model and a multiaspect faithfulness reward model. experimental results show that our method outperforms competitive baselines in terms of factuality and faithfulness. in the future, we aim to explore llm-based multi-agent learning methods to further improve the generation of cs. social media platforms. instead, a further step is required where volunteers carefully review and verify the generated cs before posting. this automated generation process has proven to be a significant time-saving factor compared to the manual creation of cs. (3) thirdly, the counterspeech generated by our method may still contain factual errors. while our approach aims to minimize the likelihood of such errors, it cannot guarantee their complete elimination.