LLM s to the Rescue: Explaining DSA Statements of Reason with Platform’s Terms of Services (2024)

Abstract: The Digital Services Act (DSA) requires online platforms in the EU to provide “statements of reason” (SoRs) when restricting user content, but their effectiveness in ensuring transparency is still debated due to vague and complex terms of service (ToS). This paper explores the use of NLP techniques, specifically multi-agent systems based on large language models (LLMs), to clarify SoRs by linking them to relevant ToS sections. Analysing SoRs from platforms like Booking.com, Reddit, and LinkedIn, our findings show that LLMs can enhance the interpretability of content moderation decisions, improving user understanding and engagement with DSA requirements.

## introduction the digital services act (dsa), adopted by the european union on november 1, 2022, represents a significant milestone in the eu regulation of online platforms, as it establishes a global standard for transparency and accountability in content moderation. a key innovation of the dsa is the requirement for intermediary hosting services to provide "statements of reason" (sors) when restricting usergenerated content (article 17). the sor must specify the action taken, the factual circumstances, any use of automated systems in the moderation process, and the legal or contractual grounds for deeming the content illegal or incompatible with the platform's terms of service (tos), along with an explanation and other metadata. it should also inform users of available redress options, ensuring clarity and precision to allow users to contest the decision. article 24(5) further requires online platforms to submit sors, as outlined in article 17, to the european commission for inclusion in a publicly accessible, machine-readable database. in response to this mandate, the european commission launched the dsa transparency database (td) in september 2023. the scheme of the td roughly reflects the content of the sor pursuant to article 17. 1 each sor instance is composed of several mandatory attributes, such as the content type (e.g. text, image, etc.) and language, the type and period of restriction, the ground for the decision, the category of restricted content, the fact relied upon on the decision, etc. attribute values are to be selected by the provider from a list of options or can be typed into as free text (generally with character limitations). other attributes are only optional. as for the ground for the provider's decision (field "decision_ground"), the td presents only two possible options: "illegal_content" and "incompatible_content". moreover, the td typifies 14 distinct "categories" of statements as potential grounds for restriction (see the td is intended as a critical tool for scrutinizing content moderation practices, revealing how well platforms comply with the requirements set by the dsa. however, the effectiveness of the td in fulfilling its promises of transparency and accountability remains a subject of ongoing debate @xcite . in particular, there are doubts about whether the sor provides sufficient information to allow users to understand the reasons for content restriction and to contest its lawfulness. this is especially true when it comes to the asserted incompatibility with tos. these are often lengthy and complex documents @xcite drafted in legal jargon @xcite , and the complexity and lack of understanding and awareness of legal texts of this kind is an old and well-known issue @xcite . given this background, the present paper explores the potential of nlp techniques, particularly multi-agent systems based on large language models (llms), to enhance the transparency and user-friendliness of sors submitted to the td. we propose and evaluate a method that uses llms to contextualise explanations in sors related to tos incompatibility within the platform's content policy guidelines. our focus on tos incompatibility stems from its frequency as the main reason for content removal and its suitability for uniform llm-based analysis, unlike removals based on national laws, which vary across the eu and add complexity. tos, being unique to each platform, offer a more consistent and manageable basis for explanation. ## background our work builds on and merges three emerging research strands: 1) existing works applying computational techniques to analyse the dsa transparency report; 2) the application of nlp techniques to enhance accessibility and legibility of transparency legal requirements; 3) the use of llms in the legal field. ## data for this study, we compiled a custom dataset using resources from the dsa transparency database. we focused on statements of reasons (sors) specifically related to content removal due to violations of terms of services (tos) from three major online platform providers: booking.com, reddit, and linkedin. this selection was made to capture a diverse range of online environments. booking.com, as a leading e-commerce platform in the travel industry, provides insights into tos enforcement concerning commercial content; reddit, a large social media forum, illustrates content moderation challenges in a user-generated, communitydriven space; and linkedin, a professional networking platform, reflects tos enforcement in a setting focused on professional conduct and business communication. this approach enables a comprehensive examination of sors across platforms with varying purposes, user bases, and content policies. to ensure a representative sample, we selected sors from a specific time frame, spanning from march 2024 to august 2024. the content of each sor consists of four key attributes included in the td, which are intended to provide context for explaining the decision that affects users' content. the attributes are the uuid, the ground for incompatible content ("incompatible_content_ground"), the explanation for incompatible content ("incompatible_content_explanation") and the facts relevant for the decision ("decision_facts"). we did not rely on the current versions of the tos available on the platforms' websites, as they may overlook regulatory changes or evolving industry standards that could impact the interpretation of the sors. using the historical tos in force at the time the sors were issued was crucial, as relying on updated versions could render certain sors outdated or irrelevant. the selected dataset consists of 7000 sors, among which 3000 were issued by booking.com, 2000 by linkedin and 2000 by reddit. figure 1 shows the distribution of the selected sors across the three online platforms and the 14 typified restrictions, while reddit scores highest in "non-consensual behavior" (ncb), likely due to its large, diverse user base and the anonymity it offers, which can lead to issues like doxxing, harassment, and the unauthorised sharing of personal information. on the contrary, categories such as animal welfare (aw), self-harm (sh), and risk for public security (rfps) have relatively low sor frequencies across all platforms, as these are less common restrictions in the contexts analysed. overall, reddit addresses the most diverse harmful content, while linkedin and booking.com focus on specific issues related to their platform's nature. in addition to the statements of reasons (sors), we collected the relevant terms of service (tos) in effect when the selected sors were issued to understand the basis for content removals. we relied on both the tos, as the binding contract, and the community guidelines, which provide additional context for applying the tos. though not part of the formal contract, community guidelines are valuable as they offer practical interpretations of the tos. integrating both allowed us to better align the explanations of content removals with their intended meaning and scope, providing context that might be missing from the tos alone. both the tos and community guidelines were pre-processed to extract relevant content moder-ation clauses, enabling the llm to match them with the sors for more accurate and contextually relevant explanations of the moderation decisions. ## architecture and methods the proposed architecture is based on a multi llm-based-agent system @xcite and a retrieval-augmented generation (rag) process @xcite . it employs two autonomous llm-based agents, each assigned specific roles: the "refiner agent" and the "explainer agent". these agents operate independently, coordinating their actions to process and interpret platform documents (tos and sors), enhancing both the accuracy and contextual relevance of the system's output. through this division of tasks and inter-agent interaction, our approach aligns with the principles of multi-agent systems by enabling collaborative decision-making and specialised behaviour. we tested the agent-based architecture with two pre-trained llms: "mistral-7b-instruct-v0.3" @xcite , and "gpt4o-mini" foot_2 . during each test run, only one of these models is used, enabling a direct comparison of their outputs. each model is independently evaluated for its ability to interpret retrieved documents, refine them, and generate expert-like explanations. the mistral-7b model was used in an optimised version with 4-bit quantisation, which allows it to handle complex prompts efficiently while minimising memory usage. this makes it suitable for resource-constrained environments @xcite . on the other hand, the gpt4o-mini model has a unique architecture that adds additional depth and nuance to the evaluation process. these models were integrated into the architecture using the hugging face transformers library foot_3 for the mistral model and the openai api for the gpt4o-mini model. this integration enables comprehensive performance assessments across different computational scenarios. by combining the transparency and replicability of open-source models with the enhanced performance of proprietary models, this dual approach facilitates a thorough comparison of the models' effectiveness in interpreting and evaluating content moderation actions. the architecture is hosted on a public github repository foot_4 and is presented in figure 2, and it is divided into 4 modules, each marked with a coloured box and pairing number. we analyse each module in the following subsections. ## validation the validation process focuses on evaluating the performance of two llm-based agents: the "refiner", which extracts relevant sections from the tos, and the "explainer", which aims to clarify the content of a sor in light of the relevant platform's tos. a human evaluation approach was chosen to assess the quality of the outputs generated by both agents. human evaluation was selected due to its capacity to provide a nuanced and contextual assessment that goes beyond what current automated metrics can offer @xcite . it allows for more accurate and comprehensive feedback on semantic and qualitative aspects of the generated responses, which is particularly important with legal content. the evaluation process was designed to achieve statistical significance, ensuring that the results are robust and credible. in particular, we observed that, within each provider, the sors pertaining to the same category are remarkably similar in their formulation. given this high degree of standardisation or consistency per category, we selected one representative sample from each category per online platform. the criteria for validating the outputs of the two agents were based on four key metrics, each rated on a 1-to-5 scale: the evaluation was conducted by a panel of three independent human evaluators, each with specialised expertise in content moderation practices and regulatory compliance under the dsa foot_10 . ## experimental results this section presents the experimental results obtained from evaluating the two llm-based agents, the "refiner agent" and the "explainer agent", using the selected pre-trained models: "mistral-7binstruct-v0.3" and "gpt4o-mini". the explainer agent was evaluated separately to measure its effectiveness in providing user-friendly explanations that contextualise the legal reasons behind content moderation decisions. the scores (1-5) were averaged for both agents, outlining a global statistical significance and providing a clear comparison of the models' outputs. we used standard deviation to quantify the variability in the scores provided by different evaluators across the relevant metrics. to facilitate comparison across different metrics and model/platforms, we also normalised the standard deviation values to a range a range [0, 1]. ## discussion we detail the discussion in the subsections below, separately for the two agents and then comparatively on the performance of the two models. ## limitations we acknowledge a few limitations in our study, many of which stem from the inherent limitations of the dsa transparency database. one major constraint is the absence of direct links or detailed descriptions of the moderated content. this limitation affects our system's ability to provide fact-specific explanations for content removal decisions. instead, the model is forced to generate more generic, rule-based, or examplebased explanations, which, despite their usefulness, can limit users' ability to fully understand how their content violated the platform's tos. another limitation is the lack of multilingual testing. the models have only been tested on englishlanguage data, as the database contains no non-english sors. multilingual support is essential for broader applicability, especially across the eu. lastly, the system has not been evaluated on other platforms, which may provide different contexts for content restrictions and reasons for tos violations. such differences may impact the models' performance, and future work should address this by testing on more varied scenarios. ## conclusion and future work this study demonstrated the potential of large language models, like gpt4o-mini and mistral-7b, in enhancing transparency and user comprehension in content moderation decisions under the dsa. however, challenges remain, particularly in handling the complexity of legal texts. llms struggle with nuanced, context-specific legal language @xcite , as the one used in tos, as well as accuracy issues regarding reliance on static datasets, which may become outdated @xcite . moreover, the "black box" nature of llms, where the decision-making process is opaque, poses a significant challenge in legal contexts @xcite . in legal applications, where the rationale behind decisions must be clear and defensible @xcite , the inability to trace or explain the reasoning of llms undermines their reliability. a valuable direction for future work is conducting an ablation study to better understand the contributions of various components in our system, particularly the role of the refiner agent. preliminary results indicate that the refiner agent performs with slightly lower accuracy than the final model, prompting a closer examination of its role. from a legal point of view, we intend to expand our work by linking the agentic explanations to more refined legal grounds for content removal contained in the tos, possibly attaching them to relevant regulatory frameworks. the potential is not merely to provide an explanation of content restriction but also the legal justification to challenge the platform's decision.