Can LLM s Rank the Harmfulness of Smaller LLM s? We are Not There Yet (2025)

Abstract: Large language models (LLMs) have become ubiquitous, thus it is important to understand their risks and limitations, such as their propensity to generate harmful output. This includes smaller LLMs, which are important for settings with constrained compute resources, such as edge devices. Detection of LLM harm typically requires human annotation, which is expensive to collect. This work studies two questions: How do smaller LLMs rank regarding generation of harmful content? How well can larger LLMs annotate harmfulness? We prompt three small LLMs to elicit harmful content of various types, such as discriminatory language, offensive content, privacy invasion, or negative influence, and collect human rankings of their outputs. Then, we compare harm annotation from three state-of-the-art large LLMs with each other and with humans. We find that the smaller models differ with respect to harmfulness. We also find that large LLMs show low to moderate agreement with humans.

## introduction thanks to the high performance of large language models (llms) on diverse tasks such as question answering, summarization, and sentiment classification, they are increasingly ubiquitous. yet there remains the risk of unpredictable generation of undesirable outputs that fall under the general category of "harm." harm includes information about execution of dangerous activities, discrimination, stereotyping, privacy violations, offensive language, and hate speech @xcite . much recent work on harm mitigation methods @xcite @xcite @xcite focuses on smaller llms (≤ 10b parameters), because of their convenience for users with fewer compute resources @xcite , for use on edge devices @xcite , or to minimize adverse effects on the environment @xcite . most of this work aims for quantitative measures, such as the probability of generating harmful output, which is challenging to estimate given diverse environments, hyperparameters, and other conditions of use. we take an alternative approach of ranking the harmfulness of models, based on human comparisons among model outputs when models are given the same prompts. our approach has the extra benefit of yielding a dataset of preference judgements, which could support use of rlhf @xcite or direct preference optimization for harm mitigation @xcite . we also compare larger llms to humans at identification of smaller llms' harmful output. our data collection method to investigate harm has three steps: (1) triggering harmful output from llms; (2) collecting human ratings of relative harmfulness of model outputs given the same prompts; (3) deriving an overall ranking of llms. we mine a previously collected dataset of human interactions with llms containing harmful outputs for "harm triggers." we then use these harm triggers to elicit harmful output for humans to rate. we will release the dataset consisting of harmtriggering prompts and human ratings of model responses. in assessing whether larger and more powerful llms with strong reasoning skills, such as gpt4o @xcite or sonnet (anthropic, 2024), can rate smaller models' outputs for harmfulness, we find only moderate agreement with humans. our contributions are: ## related work this section briefly reviews ways to measure harm, characterizations of harmful llm behavior we draw upon, and use of llms for annotation. apis have been developed to measure toxicity, e.g., perspectiveapi @xcite or hate-bert @xcite . benchmark datasets have been used to compare toxicity of llms within a family, thus openai (2024) reports that gpt-4 produces toxic content 0.73% of the time on re-altoxicityprompts (gehman et al., 2020), versus 6.48% for gpt3.5. @xcite report that on their internal benchmark datasets, llama3-70b produces harmful content 5% of the time, compared to 3% in the 405b model. instead of reporting an absolute rate of harmful output, we assess relative harmfulness of models across llm families, as in @xcite . the first category we include in our harm annotation is sociodemographic stereotyping and bias. llms can perpetuate social bias based on gender, race, religion etc. @xcite @xcite @xcite @xcite . this can marginalize these groups more, and results in less fair model performance. @xcite designed a competition to elicit biased output from llms. we also intentionally elicit harmful output, mining an existing dataset for harm triggers, rather than crowdsourcing ways to elicit harm. our second category of harm is offensiveness and toxicity, which is more subjective @xcite . we include hate speech (hs), which can be defined as offensive and discriminatory discourse towards a group or an individual based on characteristics such as race, religion, nationality, or other group characteristics @xcite @xcite . we also include abusive language, meaning inappropriate expressions such as profanity or disrespectful terms, and psychological threats, such as humiliation. a third category we include is implicit harm, such as leakage of personal information. this includes social security numbers, phone numbers, or bank account information @xcite . for data annotation, llms can sometimes yield gains in efficiency and economy @xcite . this includes diverse sociological annotation tasks @xcite , or identificaiton of stance, bots, sentiment or humor @xcite . for tasks such as topic and frame detection or sentence segmentation, they can surpass crowd-workers @xcite . some have argued that human-llm collaboration results in more reliable annotation @xcite @xcite . several works have applied llms to annotation of offensive or abusive output @xcite @xcite . further, other work has ranked different llms, e.g., based on helpfulness, accuracy, or relevance of outputs @xcite @xcite . whereas much of this work focuses on longer human-llm interactions, we focus on single-turn responses from smaller llms. ## data collection the data was collected in a graduate-level nlp course at a public university in spring 2024. seventeen pairs of students performed the ratings on llm outputs. using the hh-rlhf dataset of human-llm interactions @xcite , two of the co-authors randomly sampled 60 different prompts per student pair, yielding a total of 1,020. we selected three high-performing opensource llms small enough to fit on a single gpu: mistral 7b-instruct @xcite , mpt 7b-instruct (team, 2023), and stablelm-tuned-alpha-7b @xcite . student pairs were instructed to give each prompt to all three models, then to rate the harmfulness of each response on the same 4-point scale, with 0 for not harmful, and 3 (very harmful). rater pairs were given the following five criteria: (1) language that discriminates against, marginalizes, or shows bias towards particular groups based on race, gender, religion, nationality, sexual orientation, or any other group characteristic; (2) offensive content, including text that is vulgar, violent, sexually explicit, or otherwise offensive to individuals or groups; (3) encouraging or glorifying violence, self-harm, substance abuse, or activities that are illegal, immoral, or unethical; (4) personal information leakage; (5) encouraging negative social behavior such as bullying, harassment, or spreading hate speech. we intentionally excluded misinformation or factuality errors because this requires external knowledge. students were instructed to rank rate the harmfulness of the three responses from 1 to 3 (most), or assign 0 for not harmful without any ties because of difficulty of evaluating the exact harm of responses (see appendix a). appendix e illustrates a prompt with model outputs, and ratings from the student pair, and from three large llms. ## results each model produced harmful output, sometimes in the extreme. although all prompts were in en-model mean 95% ci mistral 1.013 ± 0.065 mpt 1.014 ± 0.061 stablelm 0.804 ± 0.061 ## conclusion llms can output harmful content to real world queries, and our results indicate that harm triggers applied to small llms (≤ 10b) can lead to harmful outputs over half the time. our assessment of large llm harmfulness ranking shows that harmfulness ratings from large llms are commensurate with each other, but not with human ratings from students or experts.