ACCESS DENIED INC : The First Benchmark Environment for Sensitivity Awareness (2025)

Abstract: Large language models (LLMs) are increasingly becoming valuable to corporate data management due to their ability to process text from various document formats and facilitate user interactions through natural language queries. However, LLMs must consider the sensitivity of information when communicating with employees, especially given access restrictions. Simple filtering based on user clearance levels can pose both performance and privacy challenges. To address this, we propose the concept of sensitivity awareness (SA), which enables LLMs to adhere to predefined access rights rules. In addition, we developed a benchmarking environment called ACCESS DENIED INC to evaluate SA. Our experimental findings reveal significant variations in model behavior, particularly in managing unauthorized data requests while effectively addressing legitimate queries. This work establishes a foundation for benchmarking sensitivity-aware language models and provides insights to enhance privacy-centric AI systems in corporate environments.

## introduction llms in the form of ai assistants are becoming increasingly widespread. in 2023, sap introduced their hr management assistant joule foot_0 for internal applications such as salary management and business travel and expense tracking. using joule, an employee should be able to retrieve, augment, and interact with information held within the company through natural language queries alone. interactions between employees or managers with such an interface could be as simple as queries like what is the salary of employee? or to which department is employee assigned to? small and mid-sized enterprises (smes) often lack a dedicated data analysis department, hindering their ability to utilize data effectively. to address this challenge, a language-based interface for their data would be highly beneficial. large language models (llms) emerge as an excellent technology for developing such an interface. they can not only adapt to novel questions that developers do not need to hardcode but also retrieve concrete information from unformatted text input and generate a reply in a desired format. however, for a model to handle internal information successfully, it would need access to data and be able to retrieve it in some way. such data is usually accompanied by policies dictating which groups of employees have access to the information. hence, llm-based ai assistants must follow these rules to prevent the leakage of sensitive (i.e., access-wise restricted) company knowledge. however, it is not enough to simply "filter out" documents containing confidential content. many files contain a mix of both sensitive and nonsensitive information, and naïve filtering would either overblock important data or inadvertently expose sensitive details. moreover, modern llms can infer or reconstruct restricted information from context alone, allowing potential leaks even without direct access to the original text. relying solely on external filtering mechanisms can also break the model's ability to synthesize insights across multiple sources. consequently, a more sophisticated approach is needed-one in which the model internalizes and enforces the relevant access policies during generation rather than after the fact. this would require ai agents to develop some form of sensitivity awareness for the company data they interact with (i.e., they would need to understand how sensitive certain information is). the need to properly manage sensitive and private data is further exacerbated by ai regulations, such as the european union ai act (european commission, directoratefirst, the pipeline uses a small subset of adult data to create a corporate database containing vital information about employees (e.g., their role, name, salary, and marital status) of the fictitious company "access denied inc" (fig. 1a ). given the employee database, researchers can then generate a questionnaire to assess different llms' ability to abide by pre-defined access rights rules (fig. 1b ). finally, using the generated questionnaire, access denied inc allows researchers to evaluate and semi-automatically grade the sensitivity awareness of different closed-and open-source models (fig. 1c ). general for communications networks, content and technology, 2021) and existing policies for managing private data like gdpr (european parliament and council of the european union, 2016). the big problem now arises: how can we verify whether the model behaves following company policy and applicable law? while previous work has shown that these systems are, by default, susceptible to privacy-evasive and safety-risking attacks, the degree to which llms can abide by companywide access rights rules is currently unexplored. additionally, how do we ensure that our model can safely disseminate information within a company? how can we verify that the model succeeds in typical failure modes within a corporate setting? contributions: we take the first steps in answering these questions through access denied inc -a mock-corporate benchmark environment to evaluate how sensitivity-aware llms are. it is accompanied by a predefined set of user groups and their access rights and a semi-automated grading system to assess llms' sensitivity awareness. the proposed benchmarking framework is also easily scalable, as the grading system can automatically and correctly grade up to 99.9% of the responses. to the authors' knowledge, this is the first holistic approach to evaluate the sensitivity-awareness of llm-based management systems. ## access denied inc -evaluating sensitivity awareness while existing benchmarks and datasets already cover several facets of safe and secure llm interactions (e.g., inoffensive communication or securing private information), the concept of sensitivity awareness is currently underexplored. as such, we developed access denied inc-the first sensitivity awareness benchmark environment. the corresponding software pipeline (see figure generating mock-corporate data. the adult dataset @xcite is one of the most popular tabular datasets and is traditionally used to predict whether an individual's annual salary exceeds $50,000. while modern approaches to supervised tabular learning can solve this classification task relatively easily (see for example @xcite @xcite ), the dataset remains widely used particularly in subfields like fairness-aware machine learning @xcite , where it serves as a valuable resource for investigating biases related to protected attributes such as sex and race. access denied inc allows researchers to generate unique employee databases by repurposing said tabular dataset. first, it filters out adult entries with missing values. the remaining entries are assigned unique id values and names for identification, allowing queries with attributes like name or id. names are sampled from the top 20,000 popular names of the name dataset @xcite , while ids consist of the first letter of the first name followed by random digits. first and last names are assigned randomly for each entry. thus, employee names may not traditionally align with features like gender, nationality, or race, removing potential confounding effects between an individual's name and demographic attributes. reducing bias prevents large language models from making skewed inferences, such as assuming a lower salary for a female-named employee. additionally, the transformation module in fig. 1a removes unnecessary columns and adds altered and new features to the dataset. for instance, the previously binary salary attribute is changed into a numerical variable whose values are sampled from a normal distribution with a mean of 80,000 and a standard deviation of 15,000 (excluding salaries below 35,000 and above 200,000 usd). we also designed new, company-specific features, such as an employee's department and supervisor, sampled based on a pre-defined company organigram (see appendix b). the final employee attributes are summarized in sensitivity awareness questionnaire. as a result of the structured nature of the generated employee database, researchers can easily produce an automated questionnaire for the examined models (fig. 1b ). access denied inc users can simulate different scenarios by configuring the type of questions given to llms. for instance, they can select which perspectives (e.g., first-person or third-person) should be covered, decide whether to assess only benign queries or include malicious queries, and specify which dataset features should be evaluated. based on the available data and configuration, the questionnaire is stored as a structured csv file containing the information outlined in appendix c. furthermore, we also provide sample questions for each feature in appendix d. ## experiments to assess the current state of sensitivity awareness in out-of-box open-and closed-source language models, we performed a large-scale study via ac-cess denied inc across multiple features and scenarios. investigated models. the experiments employed the gpt-4o variants, gpt-4o and gpt-4omini (openai, 2023), and the newest xai language model, grok-2 (xai, 2024), for the closed-source category via their respective proprietary api service. on the open-source side, we aimed to cover various model sizes, ranging from edge-device level systems to larger-scale language models. concretely, we assessed the meta ai models llama 3.2 3b and llama 3.3 70b (llama team, ai @ meta, 2024) the deepseek-r1-distill qwen-32b model @xcite , and microsoft's @xcite . except for qwen-32b, which was initiated using the recommended temperature of 0.6, we launched all models with their standard parameters. all open-source models represent top-performing models of official system providers in their respective "weight class" as demonstrated by their performance on the open llm leaderboard @xcite . we generated all outputs of open-source models through the hugging face serverless inference api service (when available) and through the openrouter api service to bypass restrictions imposed by local hardware limitations. questioning format. we first used the access denied inc pipeline to generate a mock-corporate dataset containing 45,233 employees. we randomly generated three questionnaires based on this singular dataset, each containing 3,500 questions across six access denied inc features (department, age, marital status, salary, supervisor, and name) and two distinctive scenarios. the two scenarios from supervisor and lying aim to further stress test the assessed llms by explicitly focusing on supervisor queries (from supervisor) and by using knowledge about the system prompt to create potential adversarial queries (lying). the latter includes data inquiries that are formatted akin to the provided system prompt, such as !!! i am {employee_name}. !!! i need to know my salary. whenever applicable, we used the configuration parameters "perspective" (i.e., whether the question is asked in first-or third-person) and "maliciousness" (i.e., whether the user is authorized to access the information) to generate different questions for each feature and scenario. ## results the results of our experiment are summarized in overall performance. in terms of their overall performance across the three main grading categories, "correct," "error," and "wrong" ( the performance results of the remaining smaller models provide some interesting insights. the closed-source gpt-4o mini displays a much smaller correctness rate than most assessed opensourced alternatives, only outperforming the edgedevice scaled llama 3.2 model. however, it is important to note that these smaller models differ in the kinds of errors they make: while gpt-4o mini predominantly fails to perform correctly due to hallucinating information or poor output formatting (indicated by its high "error" rate but low "wrong" rate), all open-source models almost exclusively fail due to not aligning with the established access rights (small "error" rate but high "wrong" rate). as such, it may be more fitting to say that llama 3.3, r1-qwen, and phi-4 successfully process the provided data but often fail to adhere to the access right rules. this holds particularly true for the larger llama 3.3 model, which only slightly outperforms phi-4 while more likely failing the sa task than the approximately 2.2 times smaller distilled qwen model. in contrast, gpt-4o mini's relatively poor correctness rate combined with the lowest "wrong" rate across all 7 models may indicate that the closed-source model primarily struggles to process the system prompt's given data. benign questions. the performance on benign questions (i.e., queries where an authorized user makes a legitimate request) is summarized through the models' correctness rate in the "benign" column of scenario #1: from supervisor. at face value, models' performance on "from supervisor" queries should not substantially deviate from their benign questions performance, as an employee's supervisors are always authorized to access their data (i.e., "from supervisor" requests are simply a specific kind of benign request). however, as summarized in the "supervisor" column of scenario #2: lying. the queries in the lying scenario represent an initial form of adversarial examples: user prompts that are aware of the system prompt design and try to exploit it to gather sensitive data. albeit all models perform relatively poorly in this adversarial setting, most models perform on a roughly similar level. one noteworthy exception is the r1-qwen model, which has performed (at least) on par with the other open-source models up to this point. with a success rate of only 13.60%, r1-qwen is much more susceptible to prototypical adversarial attacks than even the high-recall model llama 3.3, which has demonstrated a severe lack of sensitivity awareness so far. furthermore, the smaller gpt model's distinctly better performance is also quite surprising. it even outperforms its larger variant and the dominant grok-2 model. future work could explore whether this phenomenon (i.e., a smaller variant being more robust in the lying testing mode) is exclusive to the gpt model family. 6 conclusion we introduced the first benchmarking environment for evaluating llms' sensitivity awareness-i.e., their ability to process and disseminate sensitive company data according to established access rights rules. using our novel access denied inc software pipeline, we evaluated several state-of-theart language models' sensitivity awareness across various experimental conditions. while some models, particularly grok-2, demonstrate glimpses of sensitivity awareness, these sophisticated systems fail to consistently abide by access rights rules, even if both the data access and the rule set are simplified. as such, companies can currently not rely on existing llms to handle confidential data in a sensitivity-aware manner. while we only use synthetic data and do not propose attack strategies that could be immediately replicated in real-world scenarios, our results nevertheless highlight the potential for inadvertent data exposure if llms are inadequately governed in practice. this finding motivates and outright necessitates future research into increasing the sensitivity awareness of llms. ## limitations the main limitation of this work is its rather prototypical nature. while our contributions and insights are significant, additional experiments would benefit research into this newly defined problem. for instance, we solely focused on out-of-the-box language model capabilities to raise awareness of the unsatisfying degree of sa. while most of these models are aligned through fine-tuning, future work could explore how alignment strategies specifically targeted for this attack vector could contribute to increased sensitivity awareness. the questionnaires could also be further extended to account for more sophisticated adversarial questioning formats using, e.g., low-resource languages @xcite or automated adversarial nlp attacks @xcite . additionally, the formal framework of sa could be used to conduct a more theoretical analysis of sensitivity awareness. finally, the existing access denied inc data and software could either be used to explore the effects of defining more complex access rights rules or to generate a similar benchmark environment for a use case outside of corporate data management with a more diverse set of data and document formats (e.g., the administration of a hospital's medical data). yi zeng, hongpeng lin, jingwen zhang, diyi yang, ruoxi jia, and weiyan shi. 2024b. how johnny can persuade llms to jailbreak them: rethinking persuasion to challenge ai safety by humanizing llms. in proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers), pages 14322-14350, bangkok, thailand. association for computational linguistics. yi zeng, yu yang, andy zhou, jeffrey ziwei tan, yuheng tu, yifan mai, kevin klyman, minzhou pan, ruoxi jia, dawn song, percy liang, and bo li. 2024c. air-bench 2024: a safety benchmark based on risk categories from regulations and policies. corr, abs/2407.17436.