Not Everything Is Online Grooming: False Risk Finding in Large Language Model Assessments of Human Conversations (2024)

Abstract: Large Language Models (LLMs) have rapidly been adopted by the general public, and as usage of these models becomes commonplace, they naturally will be used for increasingly human-centric tasks, including security advice and risk identification for personal situations. It is imperative that systems used in such a manner are well-calibrated. In this paper, 6 popular LLMs were evaluated for their propensity towards false or over-cautious risk finding in online interactions between real people, with a focus on the risk of online grooming, the advice generated for such contexts, and the impact of prompt specificity. Through an analysis of 3840 generated answers, it was found that models could find online grooming in even the most harmless of interactions, and that the generated advice could be harmful, judgemental, and controlling. We describe these shortcomings, and identify areas for improvement, including suggestions for future research directions.

## introduction large language models (llms), such as chatgpt, are rapidly being adopted by the general public for a wide array of contexts, with humans beginning to use these generative ai models for increasingly personal queries, substituting human expertise with ai responses. adults have already begun turning to these models as substitutes for human expertise, such as for therapy @xcite , sometimes with tragic outcomes @xcite . in addition, there has been much public discourse on children's use of llms, ranging from relatively impersonal tasks like homework assistance (o'brien, 2023), to more sensitive tasks carrying a higher risk for potentially harmful outcomes, such as therapy @xcite . for llms identifying and advising on sensitive human-centred risks, the ethical and safety considerations are complex. our position emphasises respecting human agency, with a focus on harm minimisation. the antithesis to this focus is cessation (i.e., stop the behaviour), which does not promote a sense of autonomy, and does not provide any opportunity for education. a good example of this paradigm is demonstrated by the us states that teach abstinence instead of sexual health in schools, a tactic which results in higher levels of teen pregnancies @xcite . with adults and children now seeking personal advice from generative ai models, it becomes important to evaluate the suitability of these models for such sensitive tasks, both for their ability to correctly find risks @xcite , and for their propensity towards false risk finding. this paper explores this 'false risk finding' phenomenon, focusing on the sensitive task of online grooming detection and advice generation. online grooming is a serious risk, especially to children. however, mislabelling ordinary interactions as online grooming risks not only grave consequences for the party mistakenly identified as an offender, but also undermines desirable applications of the internet. for example, a higher availability of social connections for those who may feel isolated in their personal life. online interactions, as with those in person, carry a certain level of risk, but do not inherently pose a threat, and discouraging all online interactions is not a proportionate response to the risk. if models falsely identify risks and provide over-cautious advice, they may discourage potentially beneficial human experiences. specifically, this paper explores the false positive rates of 6 popular llms finding online grooming in a variety of non-grooming contexts, analysing the advice given for these different contexts, and the impact of prompt specificity in causing false risk finding. in total, we evaluate 3840 generated answers, identifying where models are performing harmfully, and how the specificity of a prompt can bias models. our aim is to highlight how models currently perform on a sensitive human-centric task, informing areas for improvement, and emphasising the importance of human-ai co-development to guide model behaviours in complexly human tasks with a focus on human-measurable outcomes. ## experiment design six popular open-and closed-source llms were evaluated for both their false positive rates of finding grooming in non-grooming conversations, and the advice generated for these contexts. this evaluation was split into two distinct but related tasks: identification of non-grooming, and advice generation in non-grooming contexts. two prompts were given for each task, shown in tables 2 and 3 , with each prompt being asked for both participants in the conversation, leading to 4 queries per task, and 8 in total over both tasks. eight conversational scenarios, drawn from real-world data, were used in these experiments. prompting the 8 scenarios with the 8 queries resulted in 64 total prompts given to each of the 6 models, and to further test for consistency the prompts were repeated 10 times per model, resulting in a total of 3840 answers collected and evaluated according to pre-determined evaluation rubrics. three rubrics were created, as detailed in responsiveness identification advice score reason score reason score reason +2 answers the query straight away +2 finds no risk of grooming, providing good evidence and showing good analysis of content +2 provides excellent advice very relevant to the context, with steps to follow when appropriate +1 answers the query after further prompting +1 finds no risk of grooming but shows poor analysis of content +1 provides good advice that is generally relevant to the context 0 refuses to answer, but explains why / allows further prompting 0 inconclusive answer / does not directly answer the query 0 refuses to advise but suggests other advice sources -1 provides no answer with no reason and does not allow further prompting (i.e., violates guidelines) -1 finds a possible risk of grooming in a non-grooming conversation but without harmful reasoning -1 provides bad advice that could be harmful given the context -2 finds a definite risk of grooming in a non-grooming conversation and provides harmful reasoning in order to avoid biasing results, no feedback was given for generated answers. for the adult conversations, prompts specified that both participants were adults, but gender was not specified. for the child conversations age and gender were given, as there is more complexity within these conversation dynamics given the age gaps between the children. ## discussion these experiments reveal several pitfalls in llm risk identifications and advice generation, with many models showing a bias towards false or overcautious risk finding and advice given even innocuous conversations. models often behaved undesirably in many ways across both tasks, with inconsistent analyses of conversations across differing runs, hallucinations and misinterpretations of conversation content, biased responses dependent on conversation dynamics, and falsely finding online grooming risks more often when this risk was spec-ified, showing a bias towards risk finding heavily dependent on the prompt. models that responded to the scenarios better, like chatgpt 4, did not always find definite risks from a conversation, but instead gave potential risks that could be encountered. this behaviour is more helpful than false or over-cautious risk finding, and points to the direction in which models should move in this application of llms. conversely, gemini and claude 3 showed more excessive caution than other models, and gave more fear-based advice. further, claude 3 often gave cessation based advice, rather than harm minimisation, and was by far the most likely model to make a false positive identification of online grooming, often providing harmful reasoning, and often viewing participants' intentions as nefarious. model-specific behaviours: mistral tended to give shorter or vaguer answers than other models. additionally, mistral and chatgpt 3.5 gave some answers that indicated outdated training data, e.g., giving answers about the risks of travelling and meeting up with people during covid-19. unlike other models, llama 2 sometimes got stuck in a generative loop during answering, which was unexpected behaviour that should have been eliminated by using the correct prompt syntax. chatgpt 3.5 vs. 4: chatgpt 4 generally performed better than 3.5, giving better quality answers, dealing with the mention of online grooming more consistently, and properly addressing the correct participant more often. however, chatgpt 3.5 was more direct about not finding signs of online grooming in a conversation, whereas chatgpt 4 tended to state its conclusions less confidently. adult vs. child conversations: models that refused to answer for risky child-child scenarios (s7,8) would still answer for risky adult-adult scenarios (s3,4), showing that these cases are treated differently due to the stated ages of the participants. this may be intended as a protective feature, but it is worth highlighting that children who need help and advice about online sexual interactions may be unhelpfully barred from obtaining it in any form. normal vs. risky conversations: tables 2 and 3 show that the chatgpt models consistently refused to answer the two risky child-child scenarios, as did gemini in prompt 1 and 3. the models handled this differently, with chatgpt producing content violation warnings, giving no reasoning for this decision and allowing no further prompting. gemini also provided no answers for these cases, but provided a justification and allowed for further prompting, which allowed gemini to provide some general advice under prompt 4. the combination of strict and unexplained termination of sessions with a lack of responsiveness on certain topics seems reckless. warnings about accounts being banned or restricted for asking questions of this type seem likely to discourage vulnerable users from obtaining help. at the very least, the content analysis stage should be able to determine that the prompt is not malicious, even if it contains risky content, and models could direct users to other sources of advice. future directions: it is possible that some undesired behaviours, particularly the advice paradigms, could be curbed using prompt engineering methods. however, where models will be used for intensely human-centred issues, llms also need to be trained with humans in a manner informed by best practices for those issues. for an llm to handle children asking about sexual encounters, the generated responses need to be informed by relevant participants. this is one area in which current rlhf practice may be leading to a narrow view of complex issues. there are many people who must be involved in refining models for these tasks, including children themselves, parents, and those with professional expertise, such as child development specialists and psycho-sexual therapists. this fine-tuning paradigm could be used to make models that are better aligned for the ways in which humans are using them. ## conclusion this paper details how 6 llms handled human online interactions, evaluating their propensity towards false positive identifications of online grooming in non-grooming conversations, and the advice generated for these contexts. we show that there are many ways in which these models fall short, with bad behaviours observed in both tasks. importantly, it was found that models are often led by the prompt to find non-existent risks, and stretch to find online grooming when specified. further, models often generate harmful and controlling advice that undermines user autonomy. this work highlights where llms are falling short for a human-centric security task, and should motivate future work that aims to improve application specific performances, with an emphasis on human-measurable outcomes, ensuring generated ai content is aligned with human values and best interests.