diff --git "a/docs_passages_storage/processed_docs.json" "b/docs_passages_storage/processed_docs.json" new file mode 100644--- /dev/null +++ "b/docs_passages_storage/processed_docs.json" @@ -0,0 +1,1275 @@ +[ + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2022_NIST_RMF_Response.pdf", + "md_content": "Hugging Face 29 September 2022\n\n\n\n20 St Suite 620 New York, NY 11201 Jay\n\n## Hugging Face Comments on the National Institute of Standards and Technology's AI Risk Management Framework\n\nHugging Face congratulates the National Institute of Standards and Technology (NIST) on the second draft of its AI Risk Management Framework (RMF). We are eager to support the RMF's development and implementation in becoming a key resource in the field. We offer recommendations to strengthen this framework based on our experiences toward democratizing good AI and characterizing risks of systems as an open platform. Our comments narrow in on specific, actionable feedback for sections and their subsets, ordered below.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good machine learning. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. Most recently, we supported the releases of DALL\u00b7E Mini by craiyon.com and Stable Diffusion by Stability.AI, while crafting in-house documentation processes and responsible AI licenses.\n\n## 1. OVERVIEW\n\nWe broadly agree with the attributes listed for the AI RMF and recommend further specificity on point 10.\n\n## 10. Be a living document: Process for updating\n\nAs stated, AI progress is fast-paced. As stated in Appendix B, while there are similar lessons between AI and software risks, the novel dual-use nature of many general-purpose AI systems differentiate risk approaches. Should the same RMF be applied across all AI systems? What systems should be further scrutinized? This living document should describe the process of how it will be updated and the projected frequency.\n\n## 2. AUDIENCE\n\nWhile a broad set of stakeholders is necessary, the breadth also necessitates further delineation of roles and responsibilities and specificity around tasks. Test Evaluation Verification Validation (TEVV) experts with deep understanding of both the societal and technical aspects of a system are rare. The RMF should specify 'technical, societal, legal, and ethical standards or norms', where to source these standards and norms, and consider how implementable these standards and norms are to technical systems. Both standards and norms are likely to differ by culture, community, and type of system. They may be high level and difficult to implement technically. A specific organization can build and tailor its own ethical norms or charter, such as the BigScience Ethical Charter, which is based on broader international standards but also requires deep in-house expertise.\n\nAdditionally, there is no guidance on what constitutes a TEVV expert and smaller organizations may not have the infrastructure or resources to afford experts. Further guidance is needed on what skills or qualifications are required of a TEVV expert, whether they are required to be in-house, and how their legitimacy is determined. If consultation outside an organization is encouraged, the RMF's resources should share how an organization can find, validate, and compensate experts.\n\nAs a way to minimize duplication of effort, the RMF should also provide a path to making TEVV work community-oriented when relevant. TEVV expertise applied to commonly used and open-source systems should be shared either openly or in a central repository that serves as a resource available to experts analyzing other applications of the same systems.\n\n## 3. FRAMING RISK\n\nWe strongly agree with acknowledging the difficulty of measuring risk, both qualitatively and quantitatively. In the adversarial setting of a real-world and post-deployment environment, different stakeholders may be closer to the system than those who conducted initial risk assessments. For example, the developers of a general purpose system such as a language model may not be as attuned to the downstream use case of that system by a company adapting it to commercial needs. NIST should provide communication guidance for the sets of evaluators and stakeholders throughout a system lifecycle, from development to deployment.\n\n## 3.2.1. Risk Measurement\n\nSince relying on existing pieces and developing ML artifacts out of context with a view to re-usability has become common practice, addressing this challenge should be prioritized. A nalysis a priori can apply across a broad range of systems and levels before analysis in context. For example, Hugging Face provides tools/documentation to support:\n\n- -Tools to analyze data and explore its biases\n- -Tools to explore and compare various ML models' behaviors on tasks like speech recognition\n- -Tools to explore the various steps and various contexts of an ML task lifecycle for tasks like like Automatic Content Moderation\n- -Tools to explore the metrics used to evaluate models and the biases they themselves may introduce\n- -The Hugging Face hub also hosts a range of community-contributed tools for evaluating biases of various ML artifacts.\n\n## 3.2.4. Organizational Integration of Risk\n\nThe RMF and Playbook should give examples of how to properly integrate the given RMF into organizations' processes. An established enterprise risk management process may differ by sector and size of the organization. We suggest providing case studies in the Profiles section of the RMF in action across types of systems and sectors. These examples should show how different size organizations implement the RMF, as well as how to engage stakeholders. This can also provide examples of documentation to increase transparency, such as with Model Cards.\n\n## 4. AI RISKS AND TRUSTWORTHINESS\n\nThe listed characteristics are not only interrelated but also have heavy overlap. Many of these given terms are interpreted differently across organizations. For example, AI safety is a broad field that could intersect with fairness and bias, and explainability could merge with transparency. Further guidance is needed on how to conduct the suggested contextual assessment to address potential tradeoffs between system performance and trustworthiness. We commend NIST on the definitions for Valid and Reliable, as well as Security and Privacy. All characteristics should reference more in-depth resources , such as NIST's publication on bias. The importance and order of the characteristics needs further discussion.\n\n## Human Factors\n\nThis section makes valid arguments about human-in-the-loop. We strongly disagree with AI systems being used 'in high-impact settings as a way to make decisions fairer and more impartial than human'. This should not be encouraged, as this can amplify harmful biases and outcomes and contradicts many arguments for assessing risk pre-deployment.\n\n## 4.2. Safe\n\nThis abstract term overlaps most with the other listed characteristics and is interpreted differently among many AI organizations today. The term as it stands can refer to institutional risks, such as concentrating power among high-resource developers, or technical risks, such as unreliable outputs. Safety should be an overarching goal of the many characteristics.\n\n## 4.3. Fair - and Bias is Managed\n\nThe complexities of fairness and bias cannot be captured in a short playbook section. We applaud the description of systemic bias and the many layers needed to address in ultimately reaching fair outcomes. This section should clearly point to either the NIST Special Publication 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, or another publication explicitly as the resource for this topic. To operationalize fairness, lessons from legal definitions of non-discrimination can also be helpful.\n\n## 4.5. Transparent and Accountable\n\nTwo key questions remain unaddressed in this section: first how should stakeholders communicate about risks in a standardized and clear manner? And second, whom should be held accountable for harmful outputs from an AI system? Throughout a system's lifecycle different documentation processes are needed, which could spark intellectual property and privacy concerns. The RMF should have optional templates for documentation and suggest the level of transparency needed throughout the system lifecycle, depending on legal considerations. Since BigScience is an open-science organization, we documented training data as a Data Catalog. We are also transparent about what tests and results we run on our model BLOOM. This may not be the case for all organizations.\n\n## 4.6. Explainable and Interpretable\n\nFull interpretability is technically difficult to achieve in most systems and may be less efficient than contesting a decision or enabling human recourse. This section should explain the ultimate goal of explainability.\n\n## 6. AI RMF CORE\n\nAs defined in section 3.1, a 'risk' considers but is distinct from an impact. The RMF by nature should not encompass harms, but the RMF Core cannot operate as a circle without risks developing into impacts. Understanding how external, adversarial, and real-world lessons affect the core process will strengthen actions taken.\n\n## 6.1. Govern\n\nBoth technical and organizational processes require a diverse team, as noted in the previous section. Ongoing processes should have clearly defined timelines for when to reassess each risk and system.\n\n## 6.2. Map\n\nExhaustively mapping potential risks and benefits is nearly impossible for dual-use systems; AI systems that can be used for both good and bad. Especially for general purpose systems such as language models and multimodal text-to-image models developed without a specific downstream application in mind, mapping is an ongoing challenge. Public institutions can guide developers by specifying sectors and use cases that are most likely out-of-scope or highest-risk.\n\n## 6.3. Measure\n\nWe appreciate measurements not being exclusively quantitative. Robust measurements and evaluations also require multidisciplinary expertise. Tutorials and resources that can bridge gaps between social science and computer science lead to better socio-technical assessments. For example, Hugging Face's Evaluate library provides tutorials and no-code interfaces to run technical evaluations on our hosted models.\n\n## Conclusion\n\nRisk management is increasingly urgent in the AI field. Further specificity on key terms, processes, experts, and metrics will make this RMF more easily implementable. We commend NIST on this second draft and look forward to supporting the RMF as it reaches its final stages.\n\n## Respectfully,\n\nIrene Solaiman Policy Director, Hugging Face\n\n## Yacine Jernite\n\nML and Society Lead, Hugging Face", + "chunks": [ + { + "headings": [ + "Hugging Face 29 September 2022" + ], + "markdown": "Hugging Face 29 September 2022\n\n\n\n20 St Suite 620 New York, NY 11201 Jay\n\n## Hugging Face Comments on the National Institute of Standards and Technology's AI Risk Management Framework\n\nHugging Face congratulates the National Institute of Standards and Technology (NIST) on the second draft of its AI Risk Management Framework (RMF). We are eager to support the RMF's development and implementation in becoming a key resource in the field. We offer recommendations to strengthen this framework based on our experiences toward democratizing good AI and characterizing risks of systems as an open platform. Our comments narrow in on specific, actionable feedback for sections and their subsets, ordered below.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good machine learning. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. Most recently, we supported the releases of DALL\u00b7E Mini by craiyon.com and Stable Diffusion by Stability.AI, while crafting in-house documentation processes and responsible AI licenses.\n\n## 1. OVERVIEW\n\nWe broadly agree with the attributes listed for the AI RMF and recommend further specificity on point 10.\n\n## 10. Be a living document: Process for updating\n\nAs stated, AI progress is fast-paced. As stated in Appendix B, while there are similar lessons between AI and software risks, the novel dual-use nature of many general-purpose AI systems differentiate risk approaches. Should the same RMF be applied across all AI systems? What systems should be further scrutinized? This living document should describe the process of how it will be updated and the projected frequency.\n\n## 2. AUDIENCE\n\nWhile a broad set of stakeholders is necessary, the breadth also necessitates further delineation of roles and responsibilities and specificity around tasks. Test Evaluation Verification Validation (TEVV) experts with deep understanding of both the societal and technical aspects of a system are rare. The RMF should specify 'technical, societal, legal, and ethical standards or norms', where to source these standards and norms, and consider how implementable these standards and norms are to technical systems. Both standards and norms are likely to differ by culture, community, and type of system. They may be high level and difficult to implement technically. A specific organization can build and tailor its own ethical norms or charter, such as the BigScience Ethical Charter, which is based on broader international standards but also requires deep in-house expertise.\n\nAdditionally, there is no guidance on what constitutes a TEVV expert and smaller organizations may not have the infrastructure or resources to afford experts. Further guidance is needed on what skills or qualifications are required of a TEVV expert, whether they are required to be in-house, and how their legitimacy is determined. If consultation outside an organization is encouraged, the RMF's resources should share how an organization can find, validate, and compensate experts.\n\nAs a way to minimize duplication of effort, the RMF should also provide a path to making TEVV work community-oriented when relevant. TEVV expertise applied to commonly used and open-source systems should be shared either openly or in a central repository that serves as a resource available to experts analyzing other applications of the same systems.\n\n## 3. FRAMING RISK\n\nWe strongly agree with acknowledging the difficulty of measuring risk, both qualitatively and quantitatively. In the adversarial setting of a real-world and post-deployment environment, different stakeholders may be closer to the system than those who conducted initial risk assessments. For example, the developers of a general purpose system such as a language model may not be as attuned to the downstream use case of that system by a company adapting it to commercial needs. NIST should provide communication guidance for the sets of evaluators and stakeholders throughout a system lifecycle, from development to deployment.\n\n## 3.2.1. Risk Measurement\n\nSince relying on existing pieces and developing ML artifacts out of context with a view to re-usability has become common practice, addressing this challenge should be prioritized. A nalysis a priori can apply across a broad range of systems and levels before analysis in context. For example, Hugging Face provides tools/documentation to support:\n\n- -Tools to analyze data and explore its biases\n- -Tools to explore and compare various ML models' behaviors on tasks like speech recognition\n- -Tools to explore the various steps and various contexts of an ML task lifecycle for tasks like like Automatic Content Moderation\n- -Tools to explore the metrics used to evaluate models and the biases they themselves may introduce\n- -The Hugging Face hub also hosts a range of community-contributed tools for evaluating biases of various ML artifacts.\n\n## 3.2.4. Organizational Integration of Risk\n\nThe RMF and Playbook should give examples of how to properly integrate the given RMF into organizations' processes. An established enterprise risk management process may differ by sector and size of the organization. We suggest providing case studies in the Profiles section of the RMF in action across types of systems and sectors. These examples should show how different size organizations implement the RMF, as well as how to engage stakeholders. This can also provide examples of documentation to increase transparency, such as with Model Cards.\n\n## 4. AI RISKS AND TRUSTWORTHINESS\n\nThe listed characteristics are not only interrelated but also have heavy overlap. Many of these given terms are interpreted differently across organizations. For example, AI safety is a broad field that could intersect with fairness and bias, and explainability could merge with transparency. Further guidance is needed on how to conduct the suggested contextual assessment to address potential tradeoffs between system performance and trustworthiness. We commend NIST on the definitions for Valid and Reliable, as well as Security and Privacy. All characteristics should reference more in-depth resources , such as NIST's publication on bias. The importance and order of the characteristics needs further discussion.\n\n## Human Factors\n\nThis section makes valid arguments about human-in-the-loop. We strongly disagree with AI systems being used 'in high-impact settings as a way to make decisions fairer and more impartial than human'. This should not be encouraged, as this can amplify harmful biases and outcomes and contradicts many arguments for assessing risk pre-deployment.\n\n## 4.2. Safe\n\nThis abstract term overlaps most with the other listed characteristics and is interpreted differently among many AI organizations today. The term as it stands can refer to institutional risks, such as concentrating power among high-resource developers, or technical risks, such as unreliable outputs. Safety should be an overarching goal of the many characteristics.\n\n## 4.3. Fair - and Bias is Managed\n\nThe complexities of fairness and bias cannot be captured in a short playbook section. We applaud the description of systemic bias and the many layers needed to address in ultimately reaching fair outcomes. This section should clearly point to either the NIST Special Publication 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, or another publication explicitly as the resource for this topic. To operationalize fairness, lessons from legal definitions of non-discrimination can also be helpful.\n\n## 4.5. Transparent and Accountable\n\nTwo key questions remain unaddressed in this section: first how should stakeholders communicate about risks in a standardized and clear manner? And second, whom should be held accountable for harmful outputs from an AI system? Throughout a system's lifecycle different documentation processes are needed, which could spark intellectual property and privacy concerns. The RMF should have optional templates for documentation and suggest the level of transparency needed throughout the system lifecycle, depending on legal considerations. Since BigScience is an open-science organization, we documented training data as a Data Catalog. We are also transparent about what tests and results we run on our model BLOOM. This may not be the case for all organizations.\n\n## 4.6. Explainable and Interpretable\n\nFull interpretability is technically difficult to achieve in most systems and may be less efficient than contesting a decision or enabling human recourse. This section should explain the ultimate goal of explainability.\n\n## 6. AI RMF CORE\n\nAs defined in section 3.1, a 'risk' considers but is distinct from an impact. The RMF by nature should not encompass harms, but the RMF Core cannot operate as a circle without risks developing into impacts. Understanding how external, adversarial, and real-world lessons affect the core process will strengthen actions taken.\n\n## 6.1. Govern\n\nBoth technical and organizational processes require a diverse team, as noted in the previous section. Ongoing processes should have clearly defined timelines for when to reassess each risk and system.\n\n## 6.2. Map\n\nExhaustively mapping potential risks and benefits is nearly impossible for dual-use systems; AI systems that can be used for both good and bad. Especially for general purpose systems such as language models and multimodal text-to-image models developed without a specific downstream application in mind, mapping is an ongoing challenge. Public institutions can guide developers by specifying sectors and use cases that are most likely out-of-scope or highest-risk.\n\n## 6.3. Measure\n\nWe appreciate measurements not being exclusively quantitative. Robust measurements and evaluations also require multidisciplinary expertise. Tutorials and resources that can bridge gaps between social science and computer science lead to better socio-technical assessments. For example, Hugging Face's Evaluate library provides tutorials and no-code interfaces to run technical evaluations on our hosted models.\n\n## Conclusion\n\nRisk management is increasingly urgent in the AI field. Further specificity on key terms, processes, experts, and metrics will", + "char_slice": [ + 0, + 10566 + ] + }, + { + "headings": [ + "## Hugging Face Comments on the National Institute of Standards and Technology's AI Risk Management Framework", + "## About Hugging Face", + "## 1. OVERVIEW", + "## 10. Be a living document: Process for updating", + "## 2. AUDIENCE", + "## 3. FRAMING RISK", + "## 3.2.1. Risk Measurement", + "## 3.2.4. Organizational Integration of Risk", + "## 4. AI RISKS AND TRUSTWORTHINESS", + "## Human Factors", + "## 4.2. Safe", + "## 4.3. Fair - and Bias is Managed", + "## 4.5. Transparent and Accountable", + "## 4.6. Explainable and Interpretable", + "## 6. AI RMF CORE", + "## 6.1. Govern", + "## 6.2. Map", + "## 6.3. Measure", + "## Conclusion" + ], + "markdown": "real-world lessons affect the core process will strengthen actions taken.\n\n## 6.1. Govern\n\nBoth technical and organizational processes require a diverse team, as noted in the previous section. Ongoing processes should have clearly defined timelines for when to reassess each risk and system.\n\n## 6.2. Map\n\nExhaustively mapping potential risks and benefits is nearly impossible for dual-use systems; AI systems that can be used for both good and bad. Especially for general purpose systems such as language models and multimodal text-to-image models developed without a specific downstream application in mind, mapping is an ongoing challenge. Public institutions can guide developers by specifying sectors and use cases that are most likely out-of-scope or highest-risk.\n\n## 6.3. Measure\n\nWe appreciate measurements not being exclusively quantitative. Robust measurements and evaluations also require multidisciplinary expertise. Tutorials and resources that can bridge gaps between social science and computer science lead to better socio-technical assessments. For example, Hugging Face's Evaluate library provides tutorials and no-code interfaces to run technical evaluations on our hosted models.\n\n## Conclusion\n\nRisk management is increasingly urgent in the AI field. Further specificity on key terms, processes, experts, and metrics will make this RMF more easily implementable. We commend NIST on this second draft and look forward to supporting the RMF as it reaches its final stages.\n\n## Respectfully,\n\nIrene Solaiman Policy Director, Hugging Face\n\n## Yacine Jernite\n\nML and Society Lead, Hugging Face", + "char_slice": [ + 9223, + 10833 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2023_AI%20Insight%20Forum%20Kickoff%20Written%20Statement.pdf", + "md_content": "## Written Opening Statement of Clement Delangue Co-founder and CEO Hugging Face\n\n## For the AI Insight Forum Kickoff September 13, 2023\n\nSenator Schumer, Members of the AI Insight Forum, and my fellow roundtable contributors, thank you for convening and bringing your insights to this important and urgent discussion. In order to ensure AI is developed safely and in U.S. interests, we must continue to coordinate across sectors and expertise. My name is Clement Delangue, and I am the co-founder and CEO of Hugging Face.\n\nHugging Face is a community-oriented company based in the U.S. with the mission to democratize good machine learning. We conduct our mission primarily through openness, which takes many forms, including public communication about the technology, open science, and open source. Our platform hosts machine learning models, datasets, and infrastructure that supports research and resource accessibility, aiming to lower the barrier for people from all backgrounds to contribute to AI.\n\nAt Hugging Face, we are seeing the active contributions and incredible potential for AI to improve American innovation across sectors, and benefit people and society. Amid the impressive technical achievements, there are also present-day risks such as harmful stereotypes, misinformation, threats to elections, and rising carbon emissions, which have led to our investment in ethics research. AI research and development is at a critical point where we need policymaker guidance.\n\nThis past June, I testified before the House Committee on Science, Space, and Technology to share how, together, we can ensure AI's development in the national interest. My first point to the committee highlighted the importance of fostering safe innovation via access and collaboration, and I was delighted with how closely our ongoing work paralleled the SAFE innovation framework announcement. Each aspect of the acronym will complement the next and we are eager to support your policy objectives in addition to sharing our perspective at the forefront of openness in AI. I look forward to discussing with you the need for openness and transparency; measuring and mitigating harms to people and society; and investments in safeguards and controls.\n\n## Openness and Transparency\n\nOpenness is a spectrum. Protections for more open systems and guidance for system transparency will benefit people who use AI, researchers, regulators, and the economy. Most advancements and notable AI systems in use today are based on open science and open source; popular large language models are largely based on accessible research, such as the 2017 'Attention Is All You Need' paper from Google. Open development allows us to pool resources, sharing ideas and tools that make AI as safe as possible, rather than more siloed and individualized approaches developed at each organization, which can create redundant work and hamper the ability to learn from others' mistakes.\n\nMore open systems empower perspectives from industry, academia, civil society, and independent researchers to contribute to research and risk mitigation , such as understanding and mitigating harmful biases against protected classes. When researchers are able to access not only models, but also artifacts such as datasets, they can better understand and evaluate those systems and build novel and beneficial applications. We see efforts such as red-teaming, recently exemplified at DEFCON with the AI Village, Rumman Chowdhury, and the White House's leadership, as successful examples of broader access making systems more secure.\n\nOpenness cultivates a thriving startup ecosystem and protecting access to systems, research, and tooling will help realize economic gains and improve American lives. AI applications are only at the beginning of their potential in many sectors, and sector-specific experts are starting to use AI for breakthroughs, such as medical researchers developing life-saving treatments for poorly understood human diseases. Open and collaborative development helps developers leverage expertise and inventiveness across sectors to optimize safety and performance. With these opportunities, we recognize that any system at any level of openness holds risks and misuse potential; all systems require controls. Our approach to openness balances tensions using policy and technical safeguards, from community moderation to gating mechanisms.\n\nOpenness enables the highest quality technology possible. For example, access to base models and training datasets makes it possible for scientists from diverse backgrounds to understand the complexity and scale of the mechanisms that lead AI systems to cause harm. Accessible and transparent documentation makes it possible for consumers and legislators to better understand systems' strengths and weaknesses, making informed decisions about how they should and should not be used. Grounding on openness as a key value can thus help meet priorities such as responsibility, accountability, and justice.\n\nToward this goal, Hugging Face provides tools and technical support to help test and inspect its hosted AI components, and promotes the creation and adoption of model and dataset cards as the first system information users see. Hugging Face's commitment to model and dataset documentation has led to hundreds of thousands of model cards and dataset documentation on our Hub, aiding users in making informed decisions between different options.\n\nPolicymaker guidance on transparency requirements for key system components, such as pretraining data, fine-tuning data, and models, can help create a standard for disclosure. In particular, attention to data subjects can protect and amplify the voices of less visible people involved in creating data, from annotators to artists whose work is incorporated into training data. Our hosted initiatives to build systems openly is pioneering documentation approaches that focus on the people in AI and has proved to excel in existing regulatory proposal compliance research. Policy guidance on mandatory disclosure standards can also enable stronger personally identifiable information (PII) and intellectual property (IP) protection ; requiring human-centered risk documentation related to sensitive information in training data, inputs, and prompts, can help prevent privacy and IP violations.\n\n## Measuring and Mitigating Present Day Harms\n\nThe landscape of harms from AI systems deployed today requires better understanding. The personal and societal impacts of AI are emerging and will continue to emerge as systems are increasingly integrated into daily life, but how to fully evaluate these impacts and conduct robust impact assessments requires significantly more investment. In order to address present day harms, such as negative stereotypes of protected class demographics in outputs and adverse economic and labor impact, we encourage policymakers to shape risk taxonomies and guide prioritization . This includes specifying sectors and use cases that are most likely out-of-scope or highest-risk for a given system, which is especially needed for systems that are developed without a specific use case and are sometimes referred to as general-purpose.\n\nThe most important risk factors for an AI system are its scale and context of use. We see proportional requirements for systems by use, sector, risk, and breadth of impact as the most effective means for mitigating present day harms while encouraging innovation. We hold ourselves accountable by prioritizing and documenting our ethical work throughout all stages of AI research and development.\n\nThe best approaches for measuring and mitigating risk will differ by system modality and use. For example, biases in image generation manifest differently than those in language. While complex social impacts like bias are not technically solvable, better evaluations can shape out-of-scope uses and mitigation research agendas. Evaluations and measurements complement disclosure ; tools to examine PII in training data can aid in notifying data subjects and assessing the legality of the processed data. Better carbon emission measurement and mandated disclosure can incentivize more environmentally friendly practices, and help ensure that the carbon footprint of the technology is commensurate with its utility. These overarching impact areas will each need enumeration and more policy guidance.\n\nSectoral policies and regulation should be updated to consider AI contexts, particularly to protect marginalized groups from AI exacerbating inequality or unfair outcomes. Evaluation and audits are best performed in the context of a given sector and context. Policy guidance for test & evaluation, validation & verification (TEVV) experts, such as requirements for in-house and external actors, should be clarified. TEVV experts should be community-oriented, represent relevant computer and social science disciplines, and be protected with fair work conditions. Further clarification is needed for tests by system component through a system lifecycle, from dataset collation and governance, to training processes, to model deployment.\n\n## Investing in Safeguards and Reliability\n\nThe need to build mechanisms and infrastructure for building safe and transparent AI is only growing. In order to ensure leading AI development is in the national interest and the U.S. continues its technological leadership, the U.S. government should support more research infrastructure and incentives for building and applying safeguards and safety validation. Ensuring safety and security in system development and for people and society is not solely a technical effort and will require input from many disciplines. We strongly support more funding for the National Institute of Standards and Technology (NIST) and funding the National AI Research Resource (NAIRR).\n\nSimilarly, while Hugging Face is based in the U.S., our contributors from around the world strengthen research and the community. As the fastest-growing open-source library of pre-trained models in the world, we have seen openness and access be key to making systems work better for more people. Supporting broader perspectives will require building accessible infrastructure for lower resource researchers and more tailored resourcing for social impact and safety research. Coordination with our democratic allies will bring needed perspectives on human value alignment and shape open questions on tradeoffs between what constitutes high system performance and safety.\n\n## Conclusion\n\nIn order to bolster safety research and mitigate critical risks in AI today, we must encourage openness and disclosure as is appropriate. Researchers need more access to models and infrastructure to build better evaluations and research that addresses present-day harms. And ongoing, agile investments are needed for safeguards to evolve with the AI landscape. We stand ready to support and be a resource to the Forum and again thank you for this initiative.", + "chunks": [ + { + "headings": [ + "Written Opening Statement of Clement Delangue Co-founder and CEO Hugging Face" + ], + "markdown": "## Written Opening Statement of Clement Delangue Co-founder and CEO Hugging Face\n\n## For the AI Insight Forum Kickoff September 13, 2023\n\nSenator Schumer, Members of the AI Insight Forum, and my fellow roundtable contributors, thank you for convening and bringing your insights to this important and urgent discussion. In order to ensure AI is developed safely and in U.S. interests, we must continue to coordinate across sectors and expertise. My name is Clement Delangue, and I am the co-founder and CEO of Hugging Face.\n\nHugging Face is a community-oriented company based in the U.S. with the mission to democratize good machine learning. We conduct our mission primarily through openness, which takes many forms, including public communication about the technology, open science, and open source. Our platform hosts machine learning models, datasets, and infrastructure that supports research and resource accessibility, aiming to lower the barrier for people from all backgrounds to contribute to AI.\n\nAt Hugging Face, we are seeing the active contributions and incredible potential for AI to improve American innovation across sectors, and benefit people and society. Amid the impressive technical achievements, there are also present-day risks such as harmful stereotypes, misinformation, threats to elections, and rising carbon emissions, which have led to our investment in ethics research. AI research and development is at a critical point where we need policymaker guidance.\n\nThis past June, I testified before the House Committee on Science, Space, and Technology to share how, together, we can ensure AI's development in the national interest. My first point to the committee highlighted the importance of fostering safe innovation via access and collaboration, and I was delighted with how closely our ongoing work paralleled the SAFE innovation framework announcement. Each aspect of the acronym will complement the next and we are eager to support your policy objectives in addition to sharing our perspective at the forefront of openness in AI. I look forward to discussing with you the need for openness and transparency; measuring and mitigating harms to people and society; and investments in safeguards and controls.\n\n## Openness and Transparency\n\nOpenness is a spectrum. Protections for more open systems and guidance for system transparency will benefit people who use AI, researchers, regulators, and the economy. Most advancements and notable AI systems in use today are based on open science and open source; popular large language models are largely based on accessible research, such as the 2017 'Attention Is All You Need' paper from Google. Open development allows us to pool resources, sharing ideas and tools that make AI as safe as possible, rather than more siloed and individualized approaches developed at each organization, which can create redundant work and hamper the ability to learn from others' mistakes.\n\nMore open systems empower perspectives from industry, academia, civil society, and independent researchers to contribute to research and risk mitigation , such as understanding and mitigating harmful biases against protected classes. When researchers are able to access not only models, but also artifacts such as datasets, they can better understand and evaluate those systems and build novel and beneficial applications. We see efforts such as red-teaming, recently exemplified at DEFCON with the AI Village, Rumman Chowdhury, and the White House's leadership, as successful examples of broader access making systems more secure.\n\nOpenness cultivates a thriving startup ecosystem and protecting access to systems, research, and tooling will help realize economic gains and improve American lives. AI applications are only at the beginning of their potential in many sectors, and sector-specific experts are starting to use AI for breakthroughs, such as medical researchers developing life-saving treatments for poorly understood human diseases. Open and collaborative development helps developers leverage expertise and inventiveness across sectors to optimize safety and performance. With these opportunities, we recognize that any system at any level of openness holds risks and misuse potential; all systems require controls. Our approach to openness balances tensions using policy and technical safeguards, from community moderation to gating mechanisms.\n\nOpenness enables the highest quality technology possible. For example, access to base models and training datasets makes it possible for scientists from diverse backgrounds to understand the complexity and scale of the mechanisms that lead AI systems to cause harm. Accessible and transparent documentation makes it possible for consumers and legislators to better understand systems' strengths and weaknesses, making informed decisions about how they should and should not be used. Grounding on openness as a key value can thus help meet priorities such as responsibility, accountability, and justice.\n\nToward this goal, Hugging Face provides tools and technical support to help test and inspect its hosted AI components, and promotes the creation and adoption of model and dataset cards as the first system information users see. Hugging Face's commitment to model and dataset documentation has led to hundreds of thousands of model cards and dataset documentation on our Hub, aiding users in making informed decisions between different options.\n\nPolicymaker guidance on transparency requirements for key system components, such as pretraining data, fine-tuning data, and models, can help create a standard for disclosure. In particular, attention to data subjects can protect and amplify the voices of less visible people involved in creating data, from annotators to artists whose work is incorporated into training data. Our hosted initiatives to build systems openly is pioneering documentation approaches that focus on the people in AI and has proved to excel in existing regulatory proposal compliance research. Policy guidance on mandatory disclosure standards can also enable stronger personally identifiable information (PII) and intellectual property (IP) protection ; requiring human-centered risk documentation related to sensitive information in training data, inputs, and prompts, can help prevent privacy and IP violations.\n\n## Measuring and Mitigating Present Day Harms\n\nThe landscape of harms from AI systems deployed today requires better understanding. The personal and societal impacts of AI are emerging and will continue to emerge as systems are increasingly integrated into daily life, but how to fully evaluate these impacts and conduct robust impact assessments requires significantly more investment. In order to address present day harms, such as negative stereotypes of protected class demographics in outputs and adverse economic and labor impact, we encourage policymakers to shape risk taxonomies and guide prioritization . This includes specifying sectors and use cases that are most likely out-of-scope or highest-risk for a given system, which is especially needed for systems that are developed without a specific use case and are sometimes referred to as general-purpose.\n\nThe most important risk factors for an AI system are its scale and context of use. We see proportional requirements for systems by use, sector, risk, and breadth of impact as the most effective means for mitigating present day harms while encouraging innovation. We hold ourselves accountable by prioritizing and documenting our ethical work throughout all stages of AI research and development.\n\nThe best approaches for measuring and mitigating risk will differ by system modality and use. For example, biases in image generation manifest differently than those in language. While complex social impacts like bias are not technically solvable, better evaluations can shape out-of-scope uses and mitigation research agendas. Evaluations and measurements complement disclosure ; tools to examine PII in training data can aid in notifying data subjects and assessing the legality of the processed data. Better carbon emission measurement and mandated disclosure can incentivize more environmentally friendly practices, and help ensure that the carbon footprint of the technology is commensurate with its utility. These overarching impact areas will each need enumeration and more policy guidance.\n\nSectoral policies and regulation should be updated to consider AI contexts, particularly to protect marginalized groups from AI exacerbating inequality or unfair outcomes. Evaluation and audits are best performed in the context of a given sector and context. Policy guidance for test & evaluation, validation & verification (TEVV) experts, such as requirements for in-house and external actors, should be clarified. TEVV experts should be community-oriented, represent relevant computer and social science disciplines, and be protected with fair work conditions. Further clarification is needed for tests by system component through a system lifecycle, from dataset collation and governance, to training processes, to model deployment.\n\n## Investing in Safeguards and Reliability\n\nThe need to build mechanisms and infrastructure for building safe and transparent AI is only growing. In order to ensure leading AI development is in the national interest and the U.S. continues its technological leadership, the U.S. government should support more research infrastructure and incentives for building and applying safeguards and safety validation. Ensuring safety and security in system development and for people and society is not solely a technical effort and will require input from many disciplines. We strongly support more funding for the National Institute of Standards and Technology (NIST) and funding the National AI Research Resource (NAIRR).\n\nSimilarly, while Hugging Face is based in the U.S., our contributors from around the world strengthen research and the community. As the fastest-growing open-source library of pre-trained models in the world, we have seen openness and access be key to making systems work better for more people. Supporting broader perspectives will require building accessible infrastructure for lower resource researchers and more tailored resourcing for social impact and safety research. Coordination with our democratic allies will bring needed perspectives on human value alignment and shape open questions on tradeoffs between what constitutes high system performance and safety.\n\n## Conclusion\n\nIn order to bolster safety research and mitigate critical risks in AI today, we must encourage openness and disclosure as is appropriate. Researchers need more access to models and infrastructure to build better evaluations and research that addresses present-day harms. And ongoing, agile investments are needed for safeguards to evolve with the AI landscape. We stand ready to support and be a resource to the Forum and again thank you for this initiative.", + "start_char_id": 0, + "end_char_id": 11024 + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2023_Copyright_Response.pdf", + "md_content": "\n\nHugging Face, Inc. 20 Jay Street, Suite 620, Brooklyn, NY 11201\n\n## Hugging Face Response to the Copyright Office Notice of Inquiry on Artificial Intelligence and Copyright\n\nHugging Face commends the Copyright Office on its extensive work framing the different aspects and new questions pertaining to copyright in AI systems inputs and outputs. The following comments are informed by our experiences as an open platform for state-of-the-art (SotA) AI systems, working to make AI accessible and broadly available to researchers for responsible development. We first provide a summary answer, further comments are then organized by section and by question. If a section or question is not highlighted, we do not have specific, actionable feedback.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## Summary: Artificial Intelligence and Copyright\n\nMachine Learning (ML) techniques and AI systems have received increased attention in recent years. In particular, the growing popularity of commercial products described as 'generative AI' has raised new questions about the societal impact of this technology, and about the distribution of its benefits. Among these questions, the role of the original data creators including artists, journalists, and all categories of media creators - and the impact that these new tools will have on their livelihood and ability to keep creating new original media have been at the forefront of conversations. In order to help navigate these conversations, and especially the role of copyright in answering these questions, we provide comments based on our experience developing, studying, and sharing the various components that make up AI systems - with a view to helping regulators understand the likely consequences of different policy decisions on all stakeholders.\n\nGenerative AI systems are created in multiple stages. AI systems are first and foremost a representation of their training datasets . In order to extract the correlations and information that will support the systems' capabilities, developers typically start by curating very large corpora combining publicly available digital media with private collections. These datasets are then used to train pretrained models that encode some of the statistics of the training data. The pretrained models are then adapted to specific uses through a process called fine-tuning that may make them better suited to specific tasks, like translation or summarization, or help align them to the\n\n\n\nexpectations of a user interacting with the model through, e.g. , a chatbot interface or image processing software. Finally, the fine-tuned models are deployed in commercial products or demos that handle user requests, computational resources and hardware, and model outputs to allow the vast majority of users to use them for specific purposes.\n\nUnderstanding this development chain is particularly important. In practice, copyright and Fair Use assessments will be very different at each of the stages - whereas sharing training datasets and pre-trained models enables the very kind of beneficial uses that the Fair Use doctrine aims to enable, answering questions about the impact of the technology on the market of the creators will often depend more on choices made at the fine-tuning and deployment stages. We caution that making requirements of all actors along the development chain based on decisions made at the deployment stage may end up taking compliance out of the reach of anyone but a few very well resourced companies, concentrating their authority and eventually harming the very content creators the requirements aim to protect e.g. , by limiting their ability to negotiate for application conditions of the technology that are more aligned with their needs. We note in particular that early approaches to licensing data for generative AI training have shown very limited benefit to data creators given the scales involved - with median pay-outs at under 0.01$ per image in one estimation, even as the 4M$ total price tag for that deal takes it outside of the scope of most academic and smaller actors' budgets.\n\nAs an alternative approach to balancing the needs of different stakeholder groups, we recommend instead focusing on transparency requirements and supporting content creators' ability to opt out of ML training within a clear legal framework to support open development of generative AI datasets and pre-trained models by a broader range of actors. Open-source development of and open research into generative AI systems support broader participation in the development of this new technology, foster more responsible AI, help avoid market concentration within a few well-resourced actors, and allow the development of tools that let data creators directly control whether and how their data is used. Transparency and opt-out based approaches are also important features of recent statements on generative AI by organizations like the Software Heritage Foundation, and can foster international consistency with regimes such as the EU directive on Copyright in the Digital Single Market and proposed AI Act.\n\nRecent discussions have also focused on the intellectual property status of the output of an AI system. Given our own experience with AI tools, we feel that the current requirements for a work of human authorship to be eligible for copyright protection are already well equipped to handle this question. As generative AI tools become part of the toolbox of most digital artists, the amount of work by the artists to elicit, process, and integrate the output of the systems into their creative process will remain the most important factor. Granting automatic rights to model developers or users of AI systems regardless of the amount of human work supporting each creation would facilitate displacement from the original creator communities, rather than enabling them to leverage these new tools to support their own creativity.\n\n## General Questions\n\n## Question 1\n\n- 1. As described above, generative AI systems have the ability to produce material that would be copyrightable if it were created by a human author. What are your views on the potential benefits and risks of this technology? How is the use of this technology currently affecting or likely to affect creators, copyright owners, technology developers, researchers, and the public?\n\nGenerative AI systems can support a range of new tools to support and facilitate creative processes. These tools can make some forms of media generation more broadly accessible, and support artists and creators in their activity. Text-to-image systems make it significantly easier to create prototypes of images from language inputs. Image-to-image systems enable new kinds of image manipulation that can support new genres of artistic expression.\n\nThese tools also have the potential to displace labor and further destabilize an already precarious creator economy. Even though generative AI tools cannot by themselves replace an artist's voice, creativity, and thoughtfulness, companies may still choose to behave as if they did - creating lower quality media for much cheaper, and directing financial resources to the developers of the tools rather than the groups of workers who ensure that new art and media are created. This risks making being a working artist less sustainable, and have a drastically negative effect on the diversity of the artistic commons.\n\nSpecific development choices matter in ensuring that the technology's benefits are more evenly distributed without having a negative impact on creativity and creators. These choices are spread all along the development chain, from selecting specific training data, choosing how to use it to train models, providing sufficient transparency about the role of training data in the model outputs, etc. We address the specific impacts of each of these aspects in the rest of this response.\n\n## Question 3\n\n- 3. Please identify any papers or studies that you believe are relevant to this Notice. These may address, for example, the economic effects of generative AI on the creative industries or how different licensing regimes do or could operate to remunerate copyright owners and/or creators for the use of their works in training AI models. The Office requests that commenters provide a hyperlink to the identified papers.\n- \u25cf The paper introducing the original GLAZE technique, which allows artists to limit the utility of their published work when training generative AI systems with current techniques:\n- \u25cb https://arxiv.org/abs/2302.04222\n- \u25cf AI Art and its Impact on Artists studies the broad impact of generative AI on practicing artists and makes several recommendations for making the technology more sustainable for artists:\n- \u25cb https://dl.acm.org/doi/10.1145/3600211.3604681\n\n\n\n\n\n- \u25cf Foundation Models and Fair Use analyzes several development and deployment scenarios for foundation models, including generative AI models. In particular, their analysis showcases how fair use determination have to consider the specific deployment and use of the technology:\n- \u25cb https://arxiv.org/abs/2303.15715\n- \u25cf Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chain\n- \u25cb https://papers.ssrn.com/sol3/papers.cfm?abstract\\_id=4523551\n- \u25cf Deconstructing Design Decisions: Why Courts Must Interrogate Machine Learning and Other Technologies\n- \u25cb https://papers.ssrn.com/sol3/papers.cfm?abstract\\_id=4564304\n- \u25cf SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore\n- \u25cb https://arxiv.org/abs/2308.04430\n- \u25cf The BigCode approach to training a Large Language Model for code generation uses an opt-out process that relies on GitHub authentication for verification\n- \u25cb https://hf.co/datasets/bigcode/governance-card#consent-of-data-subjects\n\n## Question 4\n\n4. Are there any statutory or regulatory approaches that have been adopted or are under consideration in other countries that relate to copyright and AI that should be considered or avoided in the United States? (40) How important a factor is international consistency in this area across borders?\n\nRecent regulatory developments in the EU can provide inspiration for possible approaches to managing Content under copyright in training datasets.\n\nThe EU directive on Copyright in the Digital Single Market provides for broad use of publicly available content for Machine Learning, including content under copyright, through a Text and Data Mining exception with exemptions that aims to strike a balance between the needs of data creators and the benefits of allowing broad use of public information for technology development. Specifically, the directive allows developers to use analytical techniques on digital data to ' generate information which includes but is not limited to patterns, trends and correlations ' as long as it respects the ability of data creators to opt out of having their content included. Operationally, this approach is currently limited by the lack of broadly recognized protocol for enacting that opt out, and the lack of transparency (and trust) on whether exemption requests are respected by developers - although recent approaches to collecting centralized preference signals and relying on governance mechanisms of the data hosts have shown promise. Legally, it provides data creators with a minimum level of agency by guaranteeing their right on content they produced, providing a basis for individual and collective negotiation.\n\nMore recently, the EU Parliament version of the EU AI Act introduced a requirement that Foundation Model developers provide a ' sufficiently detailed summary ' of the content under copyright in the model's training dataset. Such a summary would serve several purposes to enable better governance of creative content. First, it builds trust that opt-outs and expressed wishes of data creators are indeed respected. Second, and more importantly, it allows copyright assessments in the context of ML development and deployment to remain future proof. Relevant aspects of copyright assessments, including the propensity of a model to copy items from its\n\n\n\ntraining data verbatim, are not a fixed, uniform, and homogenous property of all ML approaches - they depend on specific development choices. Transparency about copyright data used in various versions of the technology will give courts access to necessary information when assessing what constitutes relevant case law, rather than having to rely on information provided by interested parties.\n\n## Training\n\n## Question 6\n\n- 6. What kinds of copyright-protected training materials are used to train AI models, and how are those materials collected and curated?\n\nDifferent types of organizations provide different amounts of transparency into the training materials used to develop their AI models. Unfortunately, the overwhelming majority of better-resourced companies developing these systems - especially companies developing systems for the purpose of integrating them in their own commercial offerings - tend to provide limited information about their training corpora. In recent years, organizations like Google, OpenAI, or more recently Anthropic have gone from providing high-level information about these datasets in the form of datasheets (Google PaLM's first iteration) or short dedicated sections in technical reports to withholding all information in more recent releases (PaLM version 2, Dall-E 3, GPT-4, Claude). These organizations may see the composition of their training data as a competitive advantage, fear legal exposure for data uses of uncertain legality, or simply choose to deprioritize the work involved in sharing and documenting datasets.\n\nThis lack of transparency negatively affects data creators and rights holders in multiple ways. First, it deprives creators of the ability to make informed decisions about whether and how to publish their work, and from leveraging the full range of social (such as preference signals) and technical tools ( e.g. , GLAZE, Nightshade, Kudurru) that can help them control how their work is leveraged in technology development. Second, it puts data creators at a disadvantage when negotiating conditions for use of AI systems with the technology's developers and deployers (see the recent WGA strike and agreement). Third, it makes determinations related to Copyright and Fair Use all the more difficult. As we argue in the rest of this response, determinations about most of the Fair Use factors are contingent on specific development choices throughout the development choice, starting with the dataset selection and curation. For any scope of copyright protections granted in the context of the use and development and use of generative AI tools, leveraging those rights will require a minimum level of transparency to the rights holders.\n\nIn this context, most of the information available about the prevalence and nature of copyright-protected materials in AI model training comes from a handful of open and collaborative research efforts typically hosted by non-profit organizations, smaller companies, and open-source communities. Corpora such as The Pile, The Stack, ROOTS, DoLMA, and LAION were developed with a focus on documentation, transparency, and re-usability, and have been used to train models that have often shown themselves competitive with their counterparts trained on more obscure datasets - albeit with less public visibility. These corpora are often provided along with more detailed analysis, tooling to support investigation by non-technical stakeholders, and publicly available information about the sources included. Analysis about copyright questions in generative AI contexts has typically used this information as a proxy to\n\n\n\nmake up for the lack of transparency of commercial systems. While this approximation has been necessary to enabling important investigation into the likely role of copyright-protected data in a new range of commercial systems, it can unfortunately also put undue pressure on research and open development efforts by abstracting away the meaningful differences between systems developed for the commons and systems developed for private use.\n\nFor the organizations that do have a consistent track record of providing substantial transparency into the data collection and curation process, it is common to train models on a wide array of copyright-protected texts such as books, academic papers, computer code, and more. These materials are typically obtained from publicly available sources. These sources include re-purposed previously published ML datasets, pre-crawled web sources (typically by processing data obtained by CommonCrawl), and web sources that are scraped by the dataset curator directly.\n\n- 6.1. How or where do developers of AI models acquire the materials or datasets that their models are trained on? To what extent is training material first collected by third-party entities (such as academic researchers or private companies)?\n\nThe curation, sharing, and re-use of AI training resources has different stakes and consequences for research organizations and open-source and open science actors than for larger companies. In collaborative research and development settings, curating a dataset can represent a significant resource investment for a single actor that is typically amortized by ensuring that it can easily be analyzed and re-used. These dynamics encourage the curation of datasets that fill existing resource gaps in the commons, prioritize transparency, and enable external scrutiny into the biases encoded in a model by looking at their sources in the training dataset (for example on how biases scale with dataset sizes). They also make model training accessible to smaller actors whose work is typically more aligned with fair use conditions. Conversely, while better-resourced companies may choose to prototype on private datasets, they can typically afford to re-create datasets 'from scratch' behind closed doors, limiting their exposure to copyright suits. This opaque situation is also untenable if opt-outs and preference signals, since every company may use different standards for what data they include or which opt-out signals they respect.\n\n- 6.2. To what extent are copyrighted works licensed from copyright owners for use as training materials? To your knowledge, what licensing models are currently being offered and used?\n\nLittle information is publicly available about the specifics of licensing agreements with copyright owners. The two most visible such agreements have occurred between OpenAI and the Associated for text, and ShutterStock for images. The Adobe Firefly system was trained on stock images whose copyright was owned by Adobe. A recent analysis of the Shutterstock/OpenAI licensing deal finds an estimated median payout of under 0.01$ per image and 18.5$ per artist for their entire portfolio.\n\n- 6.4. Are some or all training materials retained by developers of AI models after training is complete, and for what purpose(s)? Please describe any relevant storage and retention practices.\n\nThis is highly dependent on the policies and procedures of individual labs. There is no inherent requirement to do so, however we expect it to be common. Multiple different models are frequently trained on the same data and so retaining copies of the training data decreases duplicative data processing work. Data processing is a substantial amount of work and time, so discarding datasets after training would meaningfully increase incurred costs in the long term. It\n\n\n\nis common for dataset processing to be a multi-step endeavor, with each step addressing a different issue (e.g., removing low-quality documents, removing PII, improving formatting, combining datasets from multiple sources). As a result, it is common to store not only the final dataset but also many intermediate stages of the dataset with different amounts of preprocessing performed. This enables researchers to decide to do preprocessing differently for a future model without having to start again from scratch.\n\nThere are some important fields of research for which having access to pretraining data is important beyond merely training the models. For example, research into memorization studies the tendency of large language models to generate long exact sequences from their training corpus and the possible causes and mitigations. It is widely accepted that the gold standard for this research requires access to the training data. For example, recent work by (Biderman et al., 2023) provides passages from the training data as an input to the model to study if the model can continue them correctly. Deleting the training data would make this type of research very difficult. We see the difficulties in doing research into memorization without access to the training corpus in papers like 'Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4' (Chang et al., 2023) where the authors seek to identify what books OpenAI's ChatGPT has memorized. A key difficulty identified in the paper is the need to make a guess as to what data is in the training corpus and the fact that the quality of this guess can have a substantial impact on the results of the analysis. This challenge is reiterated in many other papers that seek to study memorization without access to the training data. In 'Quantifying Memorization Across Neural Language Models' (Carlini et al., 2022) the authors write 'We leverage access to each model's original training set to provide order-of-magnitude more precise bounds on the amount of extractable data than in prior works.' and 'Our research would not have been possible without EleutherAI's complete public release of The Pile dataset and their GPT-Neo family of models.'\n\n## Question 7\n\n7.2. How are inferences gained from the training process stored or represented within an AI model?\n\nInferences are stored as model weights. Model weights apply transformation based on those inference to inputs at use time. While their final state is shaped by interactions between the AI system and data points during training, currently used techniques do not store copies of these data in the weights. Some weights and combinations of weights ('neurons') have been shown to represent specific words, but the relationship is usually more complex. The impact of a weight or training example on the model behavior is always stochastic - we have limited causal understanding to support most counterfactuals (i.e., how would the model have differed if a specific training point had been omitted or modified), especially about general questions spanning multiple use cases. In other words, while a model may be found a posteriori to reproduce sequences or images that are featured in its training dataset, especially if these sequences are repeated multiple times, it is next to impossible in general to predict a priori how a specific data item or document will influence a model's output in a given situation.\n\n7.3. Is it possible for an AI model to 'unlearn' inferences it gained from training on a particular piece of training material? If so, is it economically feasible? In addition to retraining a model, are there other ways to 'unlearn' inferences from training?\n\n\n\nThis is an open technical question and there is not conclusive evidence in any direction at the present. There is some preliminary evidence that models can 'unlearn' inferences gained from training, but these results need substantially more study before they can be accepted. In particular, the robustness of these techniques to changing prompts and changing deployment context is largely unknown. There are also some promising approaches to accomplish similar goals to machine unlearning that do not involve retraining the model. Research into ethics and bias of machine learning models has studied the extent to which a model can be made to 'not use' certain types of information. While this doesn't remove the inference gained from training from the model itself, it would prevent those inferences from being leveraged in user-facing applications.\n\nWhether or not unlearning is technologically and economically feasible, it is not a desirable policy solution to models learning undesirable inferences from training data. There is no way for an organization training a model to conclusively demonstrate to a third party, regulating agency, or the public that any unlearning methodologies they may have applied consistently produces the desired outcome, i.e., that the model behaves as if it had been trained without the 'unlearned' data.\n\n- 7.4. Absent access to the underlying dataset, is it possible to identify whether an AI model was trained on a particular piece of training material?\n\nIn some cases yes (Carlini et al., 2022; Chang et al., 2023; Biderman et al., 2023), but it's very hard to know how reliable or complete these results are. In particular, such processes are unable to give negative results : either they find evidence that the model was trained on a particular price of material or it is inconclusive. One cannot rule out that a model was trained on data without examining the training data. This relates to the discussion in 6.4 of the study of memorization and how it is made much easier by access to the underlying training data.\n\n## Question 8\n\n- 8. Under what circumstances would the unauthorized use of copyrighted works to train AI models constitute fair use? Please discuss any case law you believe relevant to this question.\n\nWhen properly articulating the different stages of AI development, from dataset curation, to model (pre-)training, to model adaptation and deployment, the earlier stage of dataset curation and general training of AI models as typically done and documented are typically aligned with the fair use framework.\n\nMuch of the recent success of AI, both 'discriminative' and 'generative', can be traced back to the growing popularity of pre-training methods. While earlier ML models used to be trained from the ground up for each specific application up until the early 2010s, multiple innovations over the last decade have combined to shift the field as a whole to a new paradigm where a single model is first 'pre-trained' on a very large amount of data to encode generally useful statistics about a modality (text, speech, images, etc.). These statistics give models a 'leg up', so to say, when they're being adapted to specific applications - allowing developers to build functional systems for such tasks as automatic summarization, object detection, or to be used as a chatbot with many fewer annotated examples.\n\nAs the size of the models and pre-training datasets have grown by orders of magnitude, these statistics have become more complex. For example, while early text-based pre-training methods\n\n\n\nfocused on simple co-occurrence counts between words or sentence-level phenomena in text, modern Large Language Models can encode probability distributions over longer sequences well enough to generate entire documents that look not unlike what a person could write. Models of software code, images, speech, music, and multimodal data have seen similar progress in recent years - enabling models that reach higher performances on multiple question answering, transcription, image classification, and many other benchmarks. Notably, modeling higher-order statistics at the level of an entire image is what allows models to generate new images by sampling from its learned distribution. In some cases, the model may additionally capture correlations between features of the works of a specific data creator, allowing models to be prompted to generate images that are reminiscent of the style of the specific creator, but this behavior is hard to predict a priori and depends on specific technical choices in model training and data processing.\n\nInsofar as this approach primarily aims to encode broadly useful statistics and common patterns across its training examples, we broadly think that the inclusion of data under copyright among these examples aligns with existing law. The data is observed in training and not \"stored\" in the final model in any applicable sense. More generally the use of a given work in training is of a broadly beneficial purpose: the creation of a distinctive and productive AI model. Rather than replacing the specific communicative expression of the initial work, the model is capable of creating a wide variety of different sort of outputs wholly unrelated to that underlying, copyrightable expression. For those and other reasons, generative AI models are generally fair use when they train on large numbers of copyrighted works. We use ' generally' deliberately, however, as one can imagine patterns of facts that would raise tougher calls.\n\nWhile 'the market for the work' in factor 4 has never swept so widely as to cover an entire domain or occupation (e.g., the market for singing as opposed to the market for a particular communicable expression ), cases may get closer to the line and that have relevant impacts on a person's labor. For instance, one might imagine a model trained on a highly narrow set of works specifically for the purpose of replacing those communicative expressions in the market. A model might be trained on only a particular singer's copyrighted songs, or in a way that makes it more likely to generate items that are particularly similar to specific training examples, with the express purpose not to parody or comment but rather to create replacements. Importantly, copyright law has the ability to address these edge-cases, through case-by-case analysis under fair use. Some facts may only be present and able to be fully dealt with at later stages of an evaluation - i.e., assessment of how a particular model was deployed or made available as a product for sale.\n\nUltimately this needs to be a decision made by a court on a case-by-case basis. As stated above, training in commercial settings may still be fair use, such as for the purpose of learning general fact patterns. We recognize that ensuring that fair use remains consistent can put an undue burden on copyright holders, and decrease their ability to advocate for their broad interests. In order to mitigate those risks, categories of stakeholder and other jurisdictions have turned to transparency (EU AI Act) and opt-out (EU CDSM) requirements.\n\n8.4. What quantity of training materials do developers of generative AI models use for training? Does the volume of material used to train an AI model affect the fair use analysis? If so, how?\n\nThe quantity of data depends on the modality of the models being trained and the application context. For text generative models, it is common to train on 100 billion to 5 trillion 'tokens'\n\n\n\n(units of data), which is approximately 600 million to 30 billion pages of single-spaced Times New Roman English text. For text-to-image generative models, it is common to use between 300 million and 5 billion images. Finetuning a model requires far less data, sometimes requiring as few as 30 pages of text or 100 images for finetuning to perform a specific task.\n\nThe quantity and diversity of data used to train so-called 'general purpose' AI systems is an essential component of ensuring that they do not adhere too closely to any particular datum. The primary reason that text models require billions of pages of text is that if models are trained on much less data (perhaps repeated multiple times) as is common in some fields of machine learning, the model doesn't learn the general purpose objective and cleaves too closely to the training data to be useful in diverse application contexts. Similarly, finetuning models on small quantities of text is most effective when the goal is to create a model that performs a specific task following a specific style such as producing art that resembles the work of a specific artist.\n\nThis speaks primarily to the purpose and character of the use. Large and diverse training corpora enable the model to use the training data for the purpose of learning general representations and concepts from the data. Smaller, more application-specific finetuning corpora causes the model to use the training data for the purpose of learning and mimicking specific stylistic elements of the training data. While the size of the training corpus does not determine the character of use in isolation, it should play a role in determining whether the purpose that the data was put to was learning general representations or mimicking style.\n\n8.5. Under the fourth factor of the fair use analysis, how should the effect on the potential market for or value of a copyrighted work used to train an AI model be measured? (46) Should the inquiry be whether the outputs of the AI system incorporating the model compete with a particular copyrighted work, the body of works of the same author, or the market for that general class of works?\n\nThe fourth factor analysis centers on the communicable expression of a given work (or works). It has not to our knowledge previously been interpreted as preventing competition generally among users and developers of new tools.\n\nIn Google Books, the court focused 'on whether the copy brings to the marketplace a competing substitute for the original, or its derivative, so as to deprive the rights holder of significant revenues because of the likelihood that potential purchasers may opt to acquire the copy in preference to the original.' Nor did the possibility of a licensing market for text mining undermine the finding of fair use in Hathitrust, given that '[l]ost licensing revenue counts under Factor Four only when the use serves as a substitute for the original and the full-text-search use does not.'\n\nThe narrow interpretation of the market for or value of a work is made clear in Campbell vs Acuff-Rose, where the Court considered whether a parody song sold commercially is a market substitute for the original song. The Court finds '[a]s to parody pure and simple, it is unlikely that the work will act as a substitute for the original, since the two works usually serve different market functions.' The market for a parody song vs the song it is based on is substantially more similar than the market for an AI model vs its training data.\n\n\n\n## Question 9\n\n- 9. Should copyright owners have to affirmatively consent (opt in) to the use of their works for training materials, or should they be provided with the means to object (opt out)?\n\nWhile opting into the use of work as training material may be a medium to long-term goal, it is not currently feasible to seek opt-ins for already published data - especially as the majority of data under copyright on the web does not have an easily identifiable rights holder.\n\nOpt out can be achieved through a combination of recognized machine-readable opt-out formats and contestation rights - where rights holders may request that their data be removed from new versions of a maintained dataset (see e.g. BigCode).\n\nSeveral standards are currently being used on different platforms to enact opt-outs. The success of opt-outs as a governance mechanism will depend on having sufficient operational guidance on the formats accepted to express reservation. In order to facilitate the adoption of formats that work for various platforms (code repositories, discussion forums, etc.) without leading to unnecessary fragmentation, we recommend that the Copyright Office maintain a repository of recognized formats.\n\nIn order to make opt-outs functional and trustworthy, we additionally recommend that each platform use a single format, and that appropriate transparency requirements allow rights holders to check whether their reservations have been respected for a given dataset. We note that this can be achieved even when the dataset is not made directly available by hosting a search index or a membership test to check whether a data item is present in the dataset, or by providing sufficient metadata about the items included in a dataset. We advise against letting dataset curators and model developers specify their own formats for opt-outs, as this can put a disproportionate burden on rights holders to keep track of all possible formats.\n\n## Questions 12-14\n\n12. Is it possible or feasible to identify the degree to which a particular work contributes to a particular output from a generative AI system? Please explain.\n\nThis is an open technical question. However at present there is no known way to do so reliably. While some techniques do exist that allow the outputs of a generative AI system to be distinguished from 'natural' data, they are not secure against someone who wishes to adversarially conceal that a work is machine generated.\n\n- 13. What would be the economic impacts of a licensing requirement on the development and adoption of generative AI systems?\n\nGiven the quantity of works used in current training of generative AI systems, we are concerned that a licensing requirement would lead to significant market concentration without sufficiently benefiting data creators. An outcome where licensors pay millions of dollars to train on hundreds of thousands or millions of works under copyright would constitute a 'worst of both worlds' outcome in our assessment, as such a deal would be costly enough to exclude any but the very largest companies from training new models, while still providing negligible additional income to the original data creators. Cheaper datasets would provide even less to the creators, and more expensive ones would further restrict the set of possible developers. Additionally, we are\n\n\n\nconcerned that the need for a more immediate return on investment for this data would motivate developers to capture more of the market for the base data, and to develop tools that replace rather than support the original data creators.\n\n- 14. Please describe any other factors you believe are relevant with respect to potential copyright liability for training AI models.\n\nGiven the state of the technology, sufficient training data documentation is the only way to gain certainty regarding what works were used. It is therefore in the interest of rights holders that this information is made available.\n\n## Transparency and Recordkeeping\n\n15. In order to allow copyright owners to determine whether their works have been used, should developers of AI models be required to collect, retain, and disclose records regarding the materials used to train their models? Should creators of training datasets have a similar obligation?\n\nYes\n\n15.1. What level of specificity should be required?\n\nA broad description of all training data sources should be considered part of standard model documentation. While it's not practical to provide a list of the millions of works and authors that appear in the dataset (let alone list all the untitled content with no identified author), it is possible to create search tools that allow rights holders to look up relevant information without displaying the entirety of the work. Such a hybrid system has already been executed in practice and provides relevant information to all interested parties (search index, metadata explorer).\n\n15.2. To whom should disclosures be made?\n\nFirst, different groups of stakeholders should be informed about the data: the users of a given model, the people who created the training data, and relevant regulatory agencies. For models trained on widely available Internet data, we believe data documentation should be made public. The only exceptions to this should be carefully limited to protect the privacy of data subjects in domains such as healthcare and medicine.\n\n15.3. What obligations, if any, should be placed on developers of AI systems that incorporate models from third parties?\n\nThe entire model supply chain needs to be available to the users to allow them to understand the limitations of a given AI system and to make informed consumer decisions.\n\n15.4. What would be the cost or other impact of such a recordkeeping system for developers of AI models or systems, creators, consumers, or other relevant parties?\n\nThe cost of proper documentation is marginal when done along with the initial research, but can be significant when done a posteriori . We believe the broad benefits are substantial, both from a copyright perspective and aligning with broad recommendations about responsible AI development and public interest.\n\n\n\n## Generative AI Outputs\n\n20. Is legal protection for AI-generated material desirable as a policy matter? Is legal protection for AI-generated material necessary to encourage development of generative AI technologies and systems? Does existing copyright protection for computer code that operates a generative AI system provide sufficient incentives?\n\nAs machine learning practitioners, we find that very little to no innovation in generative AI is driven by the hope of obtaining copyright protection for model outputs. The incentives for innovation already exist without modifying copyright law. On the other hand, lack of copyright protection for wholly AI-generated material gives leverage to the professionals who might otherwise find themselves displaced by generative AI tools. This suggests that legal protection for AI-generated material is neither desirable nor necessary for innovation.\n\n- 25. If AI-generated material is found to infringe a copyrighted work, who should be directly or secondarily liable-the developer of a generative AI model, the developer of the system incorporating that model, end users of the system, or other parties?\n- 25.1. Do 'open-source' AI models raise unique considerations with respect to infringement based on their outputs? (53)\n\nOpen AI models pose different challenges, and introduce different opportunities with respect to the copyright status of their outputs. Open models released without sufficient documentation can make it difficult for users to evaluate whether a given output is substantially similar to an example in the dataset. Conversely, open models released with sufficiently documented datasets open the possibility for users to assess similarity more broadly by examining the closest training examples themselves, and thus lower the risk of producing similar outputs to their training data that would be missed by an automatic filter. This is a concern as notions of substantial similarity will have to take the continuous nature of model outputs into consideration, which makes existing tests insufficient.", + "chunks": [ + { + "headings": [ + "Hugging Face, Inc. 20 Jay Street, Suite 620, Brooklyn, NY 11201" + ], + "markdown": "\n\nHugging Face, Inc. 20 Jay Street, Suite 620, Brooklyn, NY 11201\n\n## Hugging Face Response to the Copyright Office Notice of Inquiry on Artificial Intelligence and Copyright\n\nHugging Face commends the Copyright Office on its extensive work framing the different aspects and new questions pertaining to copyright in AI systems inputs and outputs. The following comments are informed by our experiences as an open platform for state-of-the-art (SotA) AI systems, working to make AI accessible and broadly available to researchers for responsible development. We first provide a summary answer, further comments are then organized by section and by question. If a section or question is not highlighted, we do not have specific, actionable feedback.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## Summary: Artificial Intelligence and Copyright\n\nMachine Learning (ML) techniques and AI systems have received increased attention in recent years. In particular, the growing popularity of commercial products described as 'generative AI' has raised new questions about the societal impact of this technology, and about the distribution of its benefits. Among these questions, the role of the original data creators including artists, journalists, and all categories of media creators - and the impact that these new tools will have on their livelihood and ability to keep creating new original media have been at the forefront of conversations. In order to help navigate these conversations, and especially the role of copyright in answering these questions, we provide comments based on our experience developing, studying, and sharing the various components that make up AI systems - with a view to helping regulators understand the likely consequences of different policy decisions on all stakeholders.\n\nGenerative AI systems are created in multiple stages. AI systems are first and foremost a representation of their training datasets . In order to extract the correlations and information that will support the systems' capabilities, developers typically start by curating very large corpora combining publicly available digital media with private collections. These datasets are then used to train pretrained models that encode some of the statistics of the training data. The pretrained models are then adapted to specific uses through a process called fine-tuning that may make them better suited to specific tasks, like translation or summarization, or help align them to the\n\n\n\nexpectations of a user interacting with the model through, e.g. , a chatbot interface or image processing software. Finally, the fine-tuned models are deployed in commercial products or demos that handle user requests, computational resources and hardware, and model outputs to allow the vast majority of users to use them for specific purposes.\n\nUnderstanding this development chain is particularly important. In practice, copyright and Fair Use assessments will be very different at each of the stages - whereas sharing training datasets and pre-trained models enables the very kind of beneficial uses that the Fair Use doctrine aims to enable, answering questions about the impact of the technology on the market of the creators will often depend more on choices made at the fine-tuning and deployment stages. We caution that making requirements of all actors along the development chain based on decisions made at the deployment stage may end up taking compliance out of the reach of anyone but a few very well resourced companies, concentrating their authority and eventually harming the very content creators the requirements aim to protect e.g. , by limiting their ability to negotiate for application conditions of the technology that are more aligned with their needs. We note in particular that early approaches to licensing data for generative AI training have shown very limited benefit to data creators given the scales involved - with median pay-outs at under 0.01$ per image in one estimation, even as the 4M$ total price tag for that deal takes it outside of the scope of most academic and smaller actors' budgets.\n\nAs an alternative approach to balancing the needs of different stakeholder groups, we recommend instead focusing on transparency requirements and supporting content creators' ability to opt out of ML training within a clear legal framework to support open development of generative AI datasets and pre-trained models by a broader range of actors. Open-source development of and open research into generative AI systems support broader participation in the development of this new technology, foster more responsible AI, help avoid market concentration within a few well-resourced actors, and allow the development of tools that let data creators directly control whether and how their data is used. Transparency and opt-out based approaches are also important features of recent statements on generative AI by organizations like the Software Heritage Foundation, and can foster international consistency with regimes such as the EU directive on Copyright in the Digital Single Market and proposed AI Act.\n\nRecent discussions have also focused on the intellectual property status of the output of an AI system. Given our own experience with AI tools, we feel that the current requirements for a work of human authorship to be eligible for copyright protection are already well equipped to handle this question. As generative AI tools become part of the toolbox of most digital artists, the amount of work by the artists to elicit, process, and integrate the output of the systems into their creative process will remain the most important factor. Granting automatic rights to model developers or users of AI systems regardless of the amount of human work supporting each creation would facilitate displacement from the original creator communities, rather than enabling them to leverage these new tools to support their own creativity.\n\n## General Questions\n\n## Question 1\n\n- 1. As described above, generative AI systems have the ability to produce material that would be copyrightable if it were created by a human author. What are your views on the potential benefits and risks of this technology? How is the use of this technology currently affecting or likely to affect creators, copyright owners, technology developers, researchers, and the public?\n\nGenerative AI systems can support a range of new tools to support and facilitate creative processes. These tools can make some forms of media generation more broadly accessible, and support artists and creators in their activity. Text-to-image systems make it significantly easier to create prototypes of images from language inputs. Image-to-image systems enable new kinds of image manipulation that can support new genres of artistic expression.\n\nThese tools also have the potential to displace labor and further destabilize an already precarious creator economy. Even though generative AI tools cannot by themselves replace an artist's voice, creativity, and thoughtfulness, companies may still choose to behave as if they did - creating lower quality media for much cheaper, and directing financial resources to the developers of the tools rather than the groups of workers who ensure that new art and media are created. This risks making being a working artist less sustainable, and have a drastically negative effect on the diversity of the artistic commons.\n\nSpecific development choices matter in ensuring that the technology's benefits are more evenly distributed without having a negative impact on creativity and creators. These choices are spread all along the development chain, from selecting specific training data, choosing how to use it to train models, providing sufficient transparency about the role of training data in the model outputs, etc. We address the specific impacts of each of these aspects in the rest of this response.\n\n## Question 3\n\n- 3. Please identify any papers or studies that you believe are relevant to this Notice. These may address, for example, the economic effects of generative AI on the creative industries or how different licensing regimes do or could operate to remunerate copyright owners and/or creators for the use of their works in training AI models. The Office requests that commenters provide a hyperlink to the identified papers.\n- \u25cf The paper introducing the original GLAZE technique, which allows artists to limit the utility of their published work when training generative AI systems with current techniques:\n- \u25cb https://arxiv.org/abs/2302.04222\n- \u25cf AI Art and its Impact on Artists studies the broad impact of generative AI on practicing artists and makes several recommendations for making the technology more sustainable for artists:\n- \u25cb https://dl.acm.org/doi/10.1145/3600211.3604681\n\n\n\n\n\n- \u25cf Foundation Models and Fair Use analyzes several development and deployment scenarios for foundation models, including generative AI models. In particular, their analysis showcases how fair use determination have to consider the specific deployment and use of the technology:\n- \u25cb https://arxiv.org/abs/2303.15715\n- \u25cf Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chain\n- \u25cb https://papers.ssrn.com/sol3/papers.cfm?abstract\\_id=4523551\n- \u25cf Deconstructing Design Decisions: Why Courts Must Interrogate Machine Learning and Other Technologies\n- \u25cb https://papers.ssrn.com/sol3/papers.cfm?abstract\\_id=4564304\n- \u25cf SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore\n- \u25cb https://arxiv.org/abs/2308.04430\n- \u25cf The BigCode approach to training a Large Language Model for code generation uses an opt-out process that relies on GitHub authentication for verification\n- \u25cb https://hf.co/datasets/bigcode/governance-card#consent-of-data-subjects\n\n## Question 4\n\n4. Are there any statutory or regulatory approaches that have been adopted or are under consideration in other countries that relate to copyright and AI that should be", + "char_slice": [ + 0, + 10553 + ] + }, + { + "headings": [ + "## Hugging Face Response to the Copyright Office Notice of Inquiry on Artificial Intelligence and Copyright", + "## About Hugging Face", + "## Summary: Artificial Intelligence and Copyright", + "## General Questions", + "## Question 1", + "## Question 3", + "## Question 4" + ], + "markdown": "the specific deployment and use of the technology:\n- \u25cb https://arxiv.org/abs/2303.15715\n- \u25cf Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chain\n- \u25cb https://papers.ssrn.com/sol3/papers.cfm?abstract\\_id=4523551\n- \u25cf Deconstructing Design Decisions: Why Courts Must Interrogate Machine Learning and Other Technologies\n- \u25cb https://papers.ssrn.com/sol3/papers.cfm?abstract\\_id=4564304\n- \u25cf SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore\n- \u25cb https://arxiv.org/abs/2308.04430\n- \u25cf The BigCode approach to training a Large Language Model for code generation uses an opt-out process that relies on GitHub authentication for verification\n- \u25cb https://hf.co/datasets/bigcode/governance-card#consent-of-data-subjects\n\n## Question 4\n\n4. Are there any statutory or regulatory approaches that have been adopted or are under consideration in other countries that relate to copyright and AI that should be considered or avoided in the United States? (40) How important a factor is international consistency in this area across borders?\n\nRecent regulatory developments in the EU can provide inspiration for possible approaches to managing Content under copyright in training datasets.\n\nThe EU directive on Copyright in the Digital Single Market provides for broad use of publicly available content for Machine Learning, including content under copyright, through a Text and Data Mining exception with exemptions that aims to strike a balance between the needs of data creators and the benefits of allowing broad use of public information for technology development. Specifically, the directive allows developers to use analytical techniques on digital data to ' generate information which includes but is not limited to patterns, trends and correlations ' as long as it respects the ability of data creators to opt out of having their content included. Operationally, this approach is currently limited by the lack of broadly recognized protocol for enacting that opt out, and the lack of transparency (and trust) on whether exemption requests are respected by developers - although recent approaches to collecting centralized preference signals and relying on governance mechanisms of the data hosts have shown promise. Legally, it provides data creators with a minimum level of agency by guaranteeing their right on content they produced, providing a basis for individual and collective negotiation.\n\nMore recently, the EU Parliament version of the EU AI Act introduced a requirement that Foundation Model developers provide a ' sufficiently detailed summary ' of the content under copyright in the model's training dataset. Such a summary would serve several purposes to enable better governance of creative content. First, it builds trust that opt-outs and expressed wishes of data creators are indeed respected. Second, and more importantly, it allows copyright assessments in the context of ML development and deployment to remain future proof. Relevant aspects of copyright assessments, including the propensity of a model to copy items from its\n\n\n\ntraining data verbatim, are not a fixed, uniform, and homogenous property of all ML approaches - they depend on specific development choices. Transparency about copyright data used in various versions of the technology will give courts access to necessary information when assessing what constitutes relevant case law, rather than having to rely on information provided by interested parties.\n\n## Training\n\n## Question 6\n\n- 6. What kinds of copyright-protected training materials are used to train AI models, and how are those materials collected and curated?\n\nDifferent types of organizations provide different amounts of transparency into the training materials used to develop their AI models. Unfortunately, the overwhelming majority of better-resourced companies developing these systems - especially companies developing systems for the purpose of integrating them in their own commercial offerings - tend to provide limited information about their training corpora. In recent years, organizations like Google, OpenAI, or more recently Anthropic have gone from providing high-level information about these datasets in the form of datasheets (Google PaLM's first iteration) or short dedicated sections in technical reports to withholding all information in more recent releases (PaLM version 2, Dall-E 3, GPT-4, Claude). These organizations may see the composition of their training data as a competitive advantage, fear legal exposure for data uses of uncertain legality, or simply choose to deprioritize the work involved in sharing and documenting datasets.\n\nThis lack of transparency negatively affects data creators and rights holders in multiple ways. First, it deprives creators of the ability to make informed decisions about whether and how to publish their work, and from leveraging the full range of social (such as preference signals) and technical tools ( e.g. , GLAZE, Nightshade, Kudurru) that can help them control how their work is leveraged in technology development. Second, it puts data creators at a disadvantage when negotiating conditions for use of AI systems with the technology's developers and deployers (see the recent WGA strike and agreement). Third, it makes determinations related to Copyright and Fair Use all the more difficult. As we argue in the rest of this response, determinations about most of the Fair Use factors are contingent on specific development choices throughout the development choice, starting with the dataset selection and curation. For any scope of copyright protections granted in the context of the use and development and use of generative AI tools, leveraging those rights will require a minimum level of transparency to the rights holders.\n\nIn this context, most of the information available about the prevalence and nature of copyright-protected materials in AI model training comes from a handful of open and collaborative research efforts typically hosted by non-profit organizations, smaller companies, and open-source communities. Corpora such as The Pile, The Stack, ROOTS, DoLMA, and LAION were developed with a focus on documentation, transparency, and re-usability, and have been used to train models that have often shown themselves competitive with their counterparts trained on more obscure datasets - albeit with less public visibility. These corpora are often provided along with more detailed analysis, tooling to support investigation by non-technical stakeholders, and publicly available information about the sources included. Analysis about copyright questions in generative AI contexts has typically used this information as a proxy to\n\n\n\nmake up for the lack of transparency of commercial systems. While this approximation has been necessary to enabling important investigation into the likely role of copyright-protected data in a new range of commercial systems, it can unfortunately also put undue pressure on research and open development efforts by abstracting away the meaningful differences between systems developed for the commons and systems developed for private use.\n\nFor the organizations that do have a consistent track record of providing substantial transparency into the data collection and curation process, it is common to train models on a wide array of copyright-protected texts such as books, academic papers, computer code, and more. These materials are typically obtained from publicly available sources. These sources include re-purposed previously published ML datasets, pre-crawled web sources (typically by processing data obtained by CommonCrawl), and web sources that are scraped by the dataset curator directly.\n\n- 6.1. How or where do developers of AI models acquire the materials or datasets that their models are trained on? To what extent is training material first collected by third-party entities (such as academic researchers or private companies)?\n\nThe curation, sharing, and re-use of AI training resources has different stakes and consequences for research organizations and open-source and open science actors than for larger companies. In collaborative research and development settings, curating a dataset can represent a significant resource investment for a single actor that is typically amortized by ensuring that it can easily be analyzed and re-used. These dynamics encourage the curation of datasets that fill existing resource gaps in the commons, prioritize transparency, and enable external scrutiny into the biases encoded in a model by looking at their sources in the training dataset (for example on how biases scale with dataset sizes). They also make model training accessible to smaller actors whose work is typically more aligned with fair use conditions. Conversely, while better-resourced companies may choose to prototype on private datasets, they can typically afford to re-create datasets 'from scratch' behind closed doors, limiting their exposure to copyright suits. This opaque situation is also untenable if opt-outs and preference signals, since every company may use different standards for what data they include or which opt-out signals they respect.\n\n- 6.2. To what extent are copyrighted works licensed from copyright owners for use as training materials? To your knowledge, what licensing models are currently being offered and used?\n\nLittle information is publicly available about the specifics of licensing agreements with copyright owners. The two most visible such agreements have occurred between OpenAI and the Associated for text, and ShutterStock for images. The Adobe Firefly system was trained on stock images whose copyright was owned by Adobe. A recent analysis of the Shutterstock/OpenAI licensing deal finds an estimated median payout of under 0.01$ per image and 18.5$ per artist for their entire portfolio.\n\n- 6.4. Are some or all training materials retained by developers of AI models after training is complete, and for what purpose(s)? Please describe any relevant storage and retention practices.\n\nThis is highly dependent on the policies and procedures of individual labs. There is no inherent requirement to do so, however we expect it to be common. Multiple different models are frequently trained on the same data and so retaining copies of the training data decreases duplicative data processing work. Data processing is a substantial amount of work and time, so discarding datasets after training would meaningfully increase incurred costs in the long term. It\n\n\n\nis common", + "char_slice": [ + 9622, + 20208 + ] + }, + { + "headings": [ + "## Hugging Face Response to the Copyright Office Notice of Inquiry on Artificial Intelligence and Copyright", + "## About Hugging Face", + "## Summary: Artificial Intelligence and Copyright", + "## General Questions", + "## Question 1", + "## Question 3", + "## Question 4", + "## Training", + "## Question 6" + ], + "markdown": "copyrighted works licensed from copyright owners for use as training materials? To your knowledge, what licensing models are currently being offered and used?\n\nLittle information is publicly available about the specifics of licensing agreements with copyright owners. The two most visible such agreements have occurred between OpenAI and the Associated for text, and ShutterStock for images. The Adobe Firefly system was trained on stock images whose copyright was owned by Adobe. A recent analysis of the Shutterstock/OpenAI licensing deal finds an estimated median payout of under 0.01$ per image and 18.5$ per artist for their entire portfolio.\n\n- 6.4. Are some or all training materials retained by developers of AI models after training is complete, and for what purpose(s)? Please describe any relevant storage and retention practices.\n\nThis is highly dependent on the policies and procedures of individual labs. There is no inherent requirement to do so, however we expect it to be common. Multiple different models are frequently trained on the same data and so retaining copies of the training data decreases duplicative data processing work. Data processing is a substantial amount of work and time, so discarding datasets after training would meaningfully increase incurred costs in the long term. It\n\n\n\nis common for dataset processing to be a multi-step endeavor, with each step addressing a different issue (e.g., removing low-quality documents, removing PII, improving formatting, combining datasets from multiple sources). As a result, it is common to store not only the final dataset but also many intermediate stages of the dataset with different amounts of preprocessing performed. This enables researchers to decide to do preprocessing differently for a future model without having to start again from scratch.\n\nThere are some important fields of research for which having access to pretraining data is important beyond merely training the models. For example, research into memorization studies the tendency of large language models to generate long exact sequences from their training corpus and the possible causes and mitigations. It is widely accepted that the gold standard for this research requires access to the training data. For example, recent work by (Biderman et al., 2023) provides passages from the training data as an input to the model to study if the model can continue them correctly. Deleting the training data would make this type of research very difficult. We see the difficulties in doing research into memorization without access to the training corpus in papers like 'Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4' (Chang et al., 2023) where the authors seek to identify what books OpenAI's ChatGPT has memorized. A key difficulty identified in the paper is the need to make a guess as to what data is in the training corpus and the fact that the quality of this guess can have a substantial impact on the results of the analysis. This challenge is reiterated in many other papers that seek to study memorization without access to the training data. In 'Quantifying Memorization Across Neural Language Models' (Carlini et al., 2022) the authors write 'We leverage access to each model's original training set to provide order-of-magnitude more precise bounds on the amount of extractable data than in prior works.' and 'Our research would not have been possible without EleutherAI's complete public release of The Pile dataset and their GPT-Neo family of models.'\n\n## Question 7\n\n7.2. How are inferences gained from the training process stored or represented within an AI model?\n\nInferences are stored as model weights. Model weights apply transformation based on those inference to inputs at use time. While their final state is shaped by interactions between the AI system and data points during training, currently used techniques do not store copies of these data in the weights. Some weights and combinations of weights ('neurons') have been shown to represent specific words, but the relationship is usually more complex. The impact of a weight or training example on the model behavior is always stochastic - we have limited causal understanding to support most counterfactuals (i.e., how would the model have differed if a specific training point had been omitted or modified), especially about general questions spanning multiple use cases. In other words, while a model may be found a posteriori to reproduce sequences or images that are featured in its training dataset, especially if these sequences are repeated multiple times, it is next to impossible in general to predict a priori how a specific data item or document will influence a model's output in a given situation.\n\n7.3. Is it possible for an AI model to 'unlearn' inferences it gained from training on a particular piece of training material? If so, is it economically feasible? In addition to retraining a model, are there other ways to 'unlearn' inferences from training?\n\n\n\nThis is an open technical question and there is not conclusive evidence in any direction at the present. There is some preliminary evidence that models can 'unlearn' inferences gained from training, but these results need substantially more study before they can be accepted. In particular, the robustness of these techniques to changing prompts and changing deployment context is largely unknown. There are also some promising approaches to accomplish similar goals to machine unlearning that do not involve retraining the model. Research into ethics and bias of machine learning models has studied the extent to which a model can be made to 'not use' certain types of information. While this doesn't remove the inference gained from training from the model itself, it would prevent those inferences from being leveraged in user-facing applications.\n\nWhether or not unlearning is technologically and economically feasible, it is not a desirable policy solution to models learning undesirable inferences from training data. There is no way for an organization training a model to conclusively demonstrate to a third party, regulating agency, or the public that any unlearning methodologies they may have applied consistently produces the desired outcome, i.e., that the model behaves as if it had been trained without the 'unlearned' data.\n\n- 7.4. Absent access to the underlying dataset, is it possible to identify whether an AI model was trained on a particular piece of training material?\n\nIn some cases yes (Carlini et al., 2022; Chang et al., 2023; Biderman et al., 2023), but it's very hard to know how reliable or complete these results are. In particular, such processes are unable to give negative results : either they find evidence that the model was trained on a particular price of material or it is inconclusive. One cannot rule out that a model was trained on data without examining the training data. This relates to the discussion in 6.4 of the study of memorization and how it is made much easier by access to the underlying training data.\n\n## Question 8\n\n- 8. Under what circumstances would the unauthorized use of copyrighted works to train AI models constitute fair use? Please discuss any case law you believe relevant to this question.\n\nWhen properly articulating the different stages of AI development, from dataset curation, to model (pre-)training, to model adaptation and deployment, the earlier stage of dataset curation and general training of AI models as typically done and documented are typically aligned with the fair use framework.\n\nMuch of the recent success of AI, both 'discriminative' and 'generative', can be traced back to the growing popularity of pre-training methods. While earlier ML models used to be trained from the ground up for each specific application up until the early 2010s, multiple innovations over the last decade have combined to shift the field as a whole to a new paradigm where a single model is first 'pre-trained' on a very large amount of data to encode generally useful statistics about a modality (text, speech, images, etc.). These statistics give models a 'leg up', so to say, when they're being adapted to specific applications - allowing developers to build functional systems for such tasks as automatic summarization, object detection, or to be used as a chatbot with many fewer annotated examples.\n\nAs the size of the models and pre-training datasets have grown by orders of magnitude, these statistics have become more complex. For example, while early text-based pre-training methods\n\n\n\nfocused on simple co-occurrence counts between words or sentence-level phenomena in text, modern Large Language Models can encode probability distributions over longer sequences well enough to generate entire documents that look not unlike what a person could write. Models of software code, images, speech, music, and multimodal data have seen similar progress in recent years - enabling models that reach higher performances on multiple question answering, transcription, image classification, and many other benchmarks. Notably, modeling higher-order statistics at the level of an entire image is what allows models to generate new images by sampling from its learned distribution. In some cases, the model may additionally capture correlations between features of the works of a specific data creator, allowing models to be prompted to generate images that are reminiscent of the style of the specific creator, but this behavior is hard to predict a priori and depends on specific technical choices in model training and data processing.\n\nInsofar as this approach primarily aims to encode broadly useful statistics and common patterns across its training examples, we broadly think that the inclusion of data under copyright among these examples aligns with existing law. The data is observed in training and not \"stored\" in the final model in any applicable sense. More generally the use of a given work in training is of a broadly beneficial purpose: the creation of a distinctive and productive AI model. Rather than replacing the specific communicative expression", + "char_slice": [ + 18870, + 29060 + ] + }, + { + "headings": [ + "## Hugging Face Response to the Copyright Office Notice of Inquiry on Artificial Intelligence and Copyright", + "## About Hugging Face", + "## Summary: Artificial Intelligence and Copyright", + "## General Questions", + "## Question 1", + "## Question 3", + "## Question 4", + "## Training", + "## Question 6", + "## Question 7", + "## Question 8" + ], + "markdown": "probability distributions over longer sequences well enough to generate entire documents that look not unlike what a person could write. Models of software code, images, speech, music, and multimodal data have seen similar progress in recent years - enabling models that reach higher performances on multiple question answering, transcription, image classification, and many other benchmarks. Notably, modeling higher-order statistics at the level of an entire image is what allows models to generate new images by sampling from its learned distribution. In some cases, the model may additionally capture correlations between features of the works of a specific data creator, allowing models to be prompted to generate images that are reminiscent of the style of the specific creator, but this behavior is hard to predict a priori and depends on specific technical choices in model training and data processing.\n\nInsofar as this approach primarily aims to encode broadly useful statistics and common patterns across its training examples, we broadly think that the inclusion of data under copyright among these examples aligns with existing law. The data is observed in training and not \"stored\" in the final model in any applicable sense. More generally the use of a given work in training is of a broadly beneficial purpose: the creation of a distinctive and productive AI model. Rather than replacing the specific communicative expression of the initial work, the model is capable of creating a wide variety of different sort of outputs wholly unrelated to that underlying, copyrightable expression. For those and other reasons, generative AI models are generally fair use when they train on large numbers of copyrighted works. We use ' generally' deliberately, however, as one can imagine patterns of facts that would raise tougher calls.\n\nWhile 'the market for the work' in factor 4 has never swept so widely as to cover an entire domain or occupation (e.g., the market for singing as opposed to the market for a particular communicable expression ), cases may get closer to the line and that have relevant impacts on a person's labor. For instance, one might imagine a model trained on a highly narrow set of works specifically for the purpose of replacing those communicative expressions in the market. A model might be trained on only a particular singer's copyrighted songs, or in a way that makes it more likely to generate items that are particularly similar to specific training examples, with the express purpose not to parody or comment but rather to create replacements. Importantly, copyright law has the ability to address these edge-cases, through case-by-case analysis under fair use. Some facts may only be present and able to be fully dealt with at later stages of an evaluation - i.e., assessment of how a particular model was deployed or made available as a product for sale.\n\nUltimately this needs to be a decision made by a court on a case-by-case basis. As stated above, training in commercial settings may still be fair use, such as for the purpose of learning general fact patterns. We recognize that ensuring that fair use remains consistent can put an undue burden on copyright holders, and decrease their ability to advocate for their broad interests. In order to mitigate those risks, categories of stakeholder and other jurisdictions have turned to transparency (EU AI Act) and opt-out (EU CDSM) requirements.\n\n8.4. What quantity of training materials do developers of generative AI models use for training? Does the volume of material used to train an AI model affect the fair use analysis? If so, how?\n\nThe quantity of data depends on the modality of the models being trained and the application context. For text generative models, it is common to train on 100 billion to 5 trillion 'tokens'\n\n\n\n(units of data), which is approximately 600 million to 30 billion pages of single-spaced Times New Roman English text. For text-to-image generative models, it is common to use between 300 million and 5 billion images. Finetuning a model requires far less data, sometimes requiring as few as 30 pages of text or 100 images for finetuning to perform a specific task.\n\nThe quantity and diversity of data used to train so-called 'general purpose' AI systems is an essential component of ensuring that they do not adhere too closely to any particular datum. The primary reason that text models require billions of pages of text is that if models are trained on much less data (perhaps repeated multiple times) as is common in some fields of machine learning, the model doesn't learn the general purpose objective and cleaves too closely to the training data to be useful in diverse application contexts. Similarly, finetuning models on small quantities of text is most effective when the goal is to create a model that performs a specific task following a specific style such as producing art that resembles the work of a specific artist.\n\nThis speaks primarily to the purpose and character of the use. Large and diverse training corpora enable the model to use the training data for the purpose of learning general representations and concepts from the data. Smaller, more application-specific finetuning corpora causes the model to use the training data for the purpose of learning and mimicking specific stylistic elements of the training data. While the size of the training corpus does not determine the character of use in isolation, it should play a role in determining whether the purpose that the data was put to was learning general representations or mimicking style.\n\n8.5. Under the fourth factor of the fair use analysis, how should the effect on the potential market for or value of a copyrighted work used to train an AI model be measured? (46) Should the inquiry be whether the outputs of the AI system incorporating the model compete with a particular copyrighted work, the body of works of the same author, or the market for that general class of works?\n\nThe fourth factor analysis centers on the communicable expression of a given work (or works). It has not to our knowledge previously been interpreted as preventing competition generally among users and developers of new tools.\n\nIn Google Books, the court focused 'on whether the copy brings to the marketplace a competing substitute for the original, or its derivative, so as to deprive the rights holder of significant revenues because of the likelihood that potential purchasers may opt to acquire the copy in preference to the original.' Nor did the possibility of a licensing market for text mining undermine the finding of fair use in Hathitrust, given that '[l]ost licensing revenue counts under Factor Four only when the use serves as a substitute for the original and the full-text-search use does not.'\n\nThe narrow interpretation of the market for or value of a work is made clear in Campbell vs Acuff-Rose, where the Court considered whether a parody song sold commercially is a market substitute for the original song. The Court finds '[a]s to parody pure and simple, it is unlikely that the work will act as a substitute for the original, since the two works usually serve different market functions.' The market for a parody song vs the song it is based on is substantially more similar than the market for an AI model vs its training data.\n\n\n\n## Question 9\n\n- 9. Should copyright owners have to affirmatively consent (opt in) to the use of their works for training materials, or should they be provided with the means to object (opt out)?\n\nWhile opting into the use of work as training material may be a medium to long-term goal, it is not currently feasible to seek opt-ins for already published data - especially as the majority of data under copyright on the web does not have an easily identifiable rights holder.\n\nOpt out can be achieved through a combination of recognized machine-readable opt-out formats and contestation rights - where rights holders may request that their data be removed from new versions of a maintained dataset (see e.g. BigCode).\n\nSeveral standards are currently being used on different platforms to enact opt-outs. The success of opt-outs as a governance mechanism will depend on having sufficient operational guidance on the formats accepted to express reservation. In order to facilitate the adoption of formats that work for various platforms (code repositories, discussion forums, etc.) without leading to unnecessary fragmentation, we recommend that the Copyright Office maintain a repository of recognized formats.\n\nIn order to make opt-outs functional and trustworthy, we additionally recommend that each platform use a single format, and that appropriate transparency requirements allow rights holders to check whether their reservations have been respected for a given dataset. We note that this can be achieved even when the dataset is not made directly available by hosting a search index or a membership test to check whether a data item is present in the dataset, or by providing sufficient metadata about the items included in a dataset. We advise against letting dataset curators and model developers specify their own formats for opt-outs, as this can put a disproportionate burden on rights holders to keep track of all possible formats.\n\n## Questions 12-14\n\n12. Is it possible or feasible to identify the degree to which a particular work contributes to a particular output from a generative AI system? Please explain.\n\nThis is an open technical question. However at present there is no known way to do so reliably. While some techniques do exist that allow the outputs of a generative AI system to be distinguished from 'natural' data, they are not secure against someone who wishes to adversarially conceal that a work is machine generated.\n\n- 13. What would be the economic impacts of a licensing requirement on the development and adoption of generative AI systems?\n\nGiven the quantity of works used in current training of generative AI systems, we are concerned that a licensing requirement would lead to significant market concentration without sufficiently benefiting data creators. An outcome where licensors pay millions of dollars to train on hundreds of thousands or millions of works under copyright would constitute a 'worst of both worlds' outcome in our assessment, as such a deal would be costly enough to exclude any but", + "char_slice": [ + 27619, + 38046 + ] + }, + { + "headings": [ + "## Hugging Face Response to the Copyright Office Notice of Inquiry on Artificial Intelligence and Copyright", + "## About Hugging Face", + "## Summary: Artificial Intelligence and Copyright", + "## General Questions", + "## Question 1", + "## Question 3", + "## Question 4", + "## Training", + "## Question 6", + "## Question 7", + "## Question 8", + "## Question 9", + "## Questions 12-14" + ], + "markdown": "dataset. We advise against letting dataset curators and model developers specify their own formats for opt-outs, as this can put a disproportionate burden on rights holders to keep track of all possible formats.\n\n## Questions 12-14\n\n12. Is it possible or feasible to identify the degree to which a particular work contributes to a particular output from a generative AI system? Please explain.\n\nThis is an open technical question. However at present there is no known way to do so reliably. While some techniques do exist that allow the outputs of a generative AI system to be distinguished from 'natural' data, they are not secure against someone who wishes to adversarially conceal that a work is machine generated.\n\n- 13. What would be the economic impacts of a licensing requirement on the development and adoption of generative AI systems?\n\nGiven the quantity of works used in current training of generative AI systems, we are concerned that a licensing requirement would lead to significant market concentration without sufficiently benefiting data creators. An outcome where licensors pay millions of dollars to train on hundreds of thousands or millions of works under copyright would constitute a 'worst of both worlds' outcome in our assessment, as such a deal would be costly enough to exclude any but the very largest companies from training new models, while still providing negligible additional income to the original data creators. Cheaper datasets would provide even less to the creators, and more expensive ones would further restrict the set of possible developers. Additionally, we are\n\n\n\nconcerned that the need for a more immediate return on investment for this data would motivate developers to capture more of the market for the base data, and to develop tools that replace rather than support the original data creators.\n\n- 14. Please describe any other factors you believe are relevant with respect to potential copyright liability for training AI models.\n\nGiven the state of the technology, sufficient training data documentation is the only way to gain certainty regarding what works were used. It is therefore in the interest of rights holders that this information is made available.\n\n## Transparency and Recordkeeping\n\n15. In order to allow copyright owners to determine whether their works have been used, should developers of AI models be required to collect, retain, and disclose records regarding the materials used to train their models? Should creators of training datasets have a similar obligation?\n\nYes\n\n15.1. What level of specificity should be required?\n\nA broad description of all training data sources should be considered part of standard model documentation. While it's not practical to provide a list of the millions of works and authors that appear in the dataset (let alone list all the untitled content with no identified author), it is possible to create search tools that allow rights holders to look up relevant information without displaying the entirety of the work. Such a hybrid system has already been executed in practice and provides relevant information to all interested parties (search index, metadata explorer).\n\n15.2. To whom should disclosures be made?\n\nFirst, different groups of stakeholders should be informed about the data: the users of a given model, the people who created the training data, and relevant regulatory agencies. For models trained on widely available Internet data, we believe data documentation should be made public. The only exceptions to this should be carefully limited to protect the privacy of data subjects in domains such as healthcare and medicine.\n\n15.3. What obligations, if any, should be placed on developers of AI systems that incorporate models from third parties?\n\nThe entire model supply chain needs to be available to the users to allow them to understand the limitations of a given AI system and to make informed consumer decisions.\n\n15.4. What would be the cost or other impact of such a recordkeeping system for developers of AI models or systems, creators, consumers, or other relevant parties?\n\nThe cost of proper documentation is marginal when done along with the initial research, but can be significant when done a posteriori . We believe the broad benefits are substantial, both from a copyright perspective and aligning with broad recommendations about responsible AI development and public interest.\n\n\n\n## Generative AI Outputs\n\n20. Is legal protection for AI-generated material desirable as a policy matter? Is legal protection for AI-generated material necessary to encourage development of generative AI technologies and systems? Does existing copyright protection for computer code that operates a generative AI system provide sufficient incentives?\n\nAs machine learning practitioners, we find that very little to no innovation in generative AI is driven by the hope of obtaining copyright protection for model outputs. The incentives for innovation already exist without modifying copyright law. On the other hand, lack of copyright protection for wholly AI-generated material gives leverage to the professionals who might otherwise find themselves displaced by generative AI tools. This suggests that legal protection for AI-generated material is neither desirable nor necessary for innovation.\n\n- 25. If AI-generated material is found to infringe a copyrighted work, who should be directly or secondarily liable-the developer of a generative AI model, the developer of the system incorporating that model, end users of the system, or other parties?\n- 25.1. Do 'open-source' AI models raise unique considerations with respect to infringement based on their outputs? (53)\n\nOpen AI models pose different challenges, and introduce different opportunities with respect to the copyright status of their outputs. Open models released without sufficient documentation can make it difficult for users to evaluate whether a given output is substantially similar to an example in the dataset. Conversely, open models released with sufficiently documented datasets open the possibility for users to assess similarity more broadly by examining the closest training examples themselves, and thus lower the risk of producing similar outputs to their training data that would be missed by an automatic filter. This is a concern as notions of substantial similarity will have to take the continuous nature of model outputs into consideration, which makes existing tests insufficient.", + "char_slice": [ + 36734, + 43252 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2023_HCSST_CongressionalTestimony.pdf", + "md_content": "## Written Testimony of Clement Delangue Co-founder and CEO Hugging Face\n\n## For a hearing on\n\n'Artificial Intelligence: Advancing Innovation Towards the National Interest'\n\nBefore the Committee on Science, Space, and Technology U.S. House of Representatives June 22, 2023\n\n## Introduction\n\nChairman Lucas, Ranking Member Lofgren, and Members of the Committee, thank you for the opportunity to discuss AI innovation with you in a critical time for AI and national interests. I deeply appreciate the work you are doing to advance and guide AI innovation in the U.S. and look forward to supporting this mission. My name is Clement Delangue, and I am the co-founder and CEO of Hugging Face.\n\nHugging Face is a community-oriented company based in the U.S. with the mission to democratize good machine learning. We conduct our mission primarily through open-source and open-science, with our platform for hosting machine learning models and datasets and an infrastructure that supports research and resources to lower the barrier for all backgrounds to contribute to AI.\n\nIn this hearing, I will share the importance of openness in innovation, mechanisms for safe and transparent AI, and the necessary investments across sectors to ensure AI is both developed in the national interest and the U.S. continues its technological leadership.\n\n## Foster Safe Innovation via Access and Collaboration\n\nAI innovation especially for popular AI systems today such as ChatGPT has been heavily influenced by open research, from the foundational work on the transformers architecture to open releases of some of the most popular language models today. Making AI more open and accessible, including machine learning models, the datasets used for training, in addition to research breakthroughs, cultivates safe innovation. Broadening access to artifacts such as models and training datasets allows researchers and users to better understand systems, conduct audits, mitigate risks, and find high value applications.\n\nThe tensions of whether to fully open or fully close an AI system grapples with risks on either end; fully closed systems are often inaccessible to researchers, auditors, and democratic institutions and can therefore obscure necessary information or illegal and harmful data. A fully open system with broader access can attract malicious actors. All systems regardless of access can be misused and require risk mitigation measures. Our approach to ethical openness acknowledges these tensions and combines institutional policies, such as documentation; technical safeguards, such as gating access to artifacts; and community safeguards, such as community moderation. We hold ourselves accountable to prioritizing and documenting our ethical work throughout all stages of AI research and development.\n\nOpen systems foster democratic governance and increased access, especially to researchers, and can help to solve critical security concerns by enabling and empowering safety research. For example, the popular research on watermarking large language models by University of Maryland researchers was conducted using OPT, an open-source language model developed and released by Meta. Watermarking is an increasingly popular safeguard for AI detection and openness enables safety research via access. Open research helps us understand these techniques' robustness and accessible tooling, which we worked on with the University of Maryland researchers, and can encourage other researchers to test and improve safety techniques. Open systems can be more compliant with AI regulation than their closed counterparts ; a recent Stanford University study assessed foundation model compliance with the EU AI Act and found while many model providers only score less than 25%, such as AI21 Labs, Aleph Alpha, and Anthropic, Hugging Face's BigScience was the only model provider to score above 75%. Another organization centered on openness, EleutherAI, scored highest on disclosure requirements. Openness bolsters transparency and enables external scrutiny.\n\nThe AI field is currently dominated by a few high-resource organizations who give limited or no open access to novel AI systems, including those based on open research. In order to encourage competition and increase AI economic opportunity, we should enable access for many people to contribute to increasing the breadth of AI progress across useful applications, not just allow a select few organizations to improve the depth of more capable models.\n\nWorking with different professions to ensure AI augments workers and improves productivity requires increased access to AI systems and community contributions to determine AI utility through experimentation, adaptation, and fine-tuning . Shared resources can be built and improved over time with multidisciplinary expertise, and leverage the country's public institutions so they reflect democratic values. Hugging Face is particularly invested in the role of machine learning libraries in curating the content of AI systems.\n\n## Mandates and Mechanisms for Transparency and Safeguards\n\nAll AI systems, regardless of their level of openness, require mechanisms for better understanding and communicating aspects of the system. This includes detailed and easily consumable documentation and better risk evaluations combined with increased research for technical and policy safeguards.\n\nRigorous documentation practices for AI systems, with transparent reporting that follows well-defined protocols, serves three main goals: incentivizing responsible development; ensuring researchers and developers consider values and priorities that may otherwise be overlooked; and creating a paper trail for review. Documentation highlighting heavy risks disincentivizes developers from further development in that unsafe direction. Guidance for what to document, such as funding sources, can aid scrutiny across multiple dimensions. Accessible documentation that is readable to many audiences and able to be examined by additional parties further down the pipeline can improve understanding of the technology as a whole, including tradeoffs in benefits and risks.\n\nImportantly, documentation should be detailed and applied to AI models, datasets, and all relevant components of an AI system. Hugging Face leads by example in documenting models on our platform with over 250,000 model cards, a vastly adopted structure for documentation that can be a strong template for AI models. Datasheets are the parallel for datasets and are also widely adopted. Documentation should also cover processes and governance mechanisms, which we exemplified in a Governance Card for the BigCode initiative we are co-leading with ServiceNow to create an open-source generative code model. We also lower the barrier for users to create documentation with easy tooling that generates documentation.\n\nIn order to document and mitigate risks, we need to understand how to measure them. The current state of AI evaluations, especially for complex social impacts such as biases and environmental costs, requires more resourcing and central fora for testing, as well as transparency from entities about how and where they deploy AI systems to understand what evaluations are most urgently needed. Risk evaluation and mitigation work should be built on technical feasibility and existing research. National Institute for Standards and Technology's (NIST) work on moving toward a standard for AI bias and the AI risk management framework have been excellent examples of technical leadership on safe AI innovation in the U.S. government and guidance for the AI ecosystem. Increased funding for NIST would both strengthen U.S. government technical leadership and improve the space for researchers across sectors to collaborate on addressing AI risks.\n\nSafeguards should be developed in tandem with AI systems, and should be a combination of technical, policy, and legal approaches. Technical approaches include detection models, such as the GPT-2 output detector hosted on our Hub, and safety filters, such as the one deployed with Stable Diffusion. Organizational and platform policies should be specific to content and intended system use; we enable researchers to manage access to sensitive systems to promote accountability without facilitating malicious use; in addition to labeling and moderating content that is not for all audiences. And novel legal approaches, such as Open Responsible AI Licenses (RAIL) can provide enforcement mechanisms for abiding by intended uses.\n\n## Invest in Infrastructure and Research\n\nAI innovation across many high value applications requires investing in relevant expertises and accounting for resource imbalances among disciplines and sectors. The long-standing and widening resource divides especially between industry and academia limit who is able to contribute to innovative research and applications. We strongly support the U.S. National AI Research Resource (NAIRR) and resourcing small businesses and startups conducting public interest research. We recommend incorporating AI system access needs with infrastructure access.\n\nTraining and educational resources such as courses and tooling can encourage interdisciplinary research and more engagement from untapped sectors. This can aid professions interested in leveraging AI in conducting complex tasks, such as training an AI model. Our Evaluate library provides tutorials and no-code interfaces to run technical evaluations on our hosted models. AutoTrain empowers anyone regardless of technical skill to train, evaluate, and deploy a natural language processing (NLP) model. Hugging Face Tasks and Spaces enable anyone to build and engage with tasks.\n\nGlobal leadership in democratic values should include our international allies. Our platform encourages many groups to curate datasets and evaluations in their native or fluent languages, leading to stronger, representative science. Tooling and resources should be made available across disciplines and accessible to the many languages and perspectives needed to drive safe innovation.\n\nFurthermore, while high-level guidance is needed to converge the AI community toward similar practices, government-provided resources are most easily applied when tailored to specific systems . We recommend increasing NIST capacity to develop AI RMF profiles for popularly used systems, such as language models. We are excited about f ederal support of data commons and infrastructure, such as current efforts to make NSF funded data more broadly available . Technical experts with a track record of ethical innovation should be prioritized as advisors for resourcing initiatives.\n\n## Conclusion\n\nThe incredible capability and promise for AI to improve American lives and opportunities requires cross-sectoral collaboration, access to systems, and investing in safety. We thank the Committee for the opportunity to discuss this important issue and look forward to continuing to support a safe, innovative U.S. AI ecosystem.", + "chunks": [ + { + "headings": [ + "Written Testimony of Clement Delangue Co-founder and CEO Hugging Face" + ], + "markdown": "## Written Testimony of Clement Delangue Co-founder and CEO Hugging Face\n\n## For a hearing on\n\n'Artificial Intelligence: Advancing Innovation Towards the National Interest'\n\nBefore the Committee on Science, Space, and Technology U.S. House of Representatives June 22, 2023\n\n## Introduction\n\nChairman Lucas, Ranking Member Lofgren, and Members of the Committee, thank you for the opportunity to discuss AI innovation with you in a critical time for AI and national interests. I deeply appreciate the work you are doing to advance and guide AI innovation in the U.S. and look forward to supporting this mission. My name is Clement Delangue, and I am the co-founder and CEO of Hugging Face.\n\nHugging Face is a community-oriented company based in the U.S. with the mission to democratize good machine learning. We conduct our mission primarily through open-source and open-science, with our platform for hosting machine learning models and datasets and an infrastructure that supports research and resources to lower the barrier for all backgrounds to contribute to AI.\n\nIn this hearing, I will share the importance of openness in innovation, mechanisms for safe and transparent AI, and the necessary investments across sectors to ensure AI is both developed in the national interest and the U.S. continues its technological leadership.\n\n## Foster Safe Innovation via Access and Collaboration\n\nAI innovation especially for popular AI systems today such as ChatGPT has been heavily influenced by open research, from the foundational work on the transformers architecture to open releases of some of the most popular language models today. Making AI more open and accessible, including machine learning models, the datasets used for training, in addition to research breakthroughs, cultivates safe innovation. Broadening access to artifacts such as models and training datasets allows researchers and users to better understand systems, conduct audits, mitigate risks, and find high value applications.\n\nThe tensions of whether to fully open or fully close an AI system grapples with risks on either end; fully closed systems are often inaccessible to researchers, auditors, and democratic institutions and can therefore obscure necessary information or illegal and harmful data. A fully open system with broader access can attract malicious actors. All systems regardless of access can be misused and require risk mitigation measures. Our approach to ethical openness acknowledges these tensions and combines institutional policies, such as documentation; technical safeguards, such as gating access to artifacts; and community safeguards, such as community moderation. We hold ourselves accountable to prioritizing and documenting our ethical work throughout all stages of AI research and development.\n\nOpen systems foster democratic governance and increased access, especially to researchers, and can help to solve critical security concerns by enabling and empowering safety research. For example, the popular research on watermarking large language models by University of Maryland researchers was conducted using OPT, an open-source language model developed and released by Meta. Watermarking is an increasingly popular safeguard for AI detection and openness enables safety research via access. Open research helps us understand these techniques' robustness and accessible tooling, which we worked on with the University of Maryland researchers, and can encourage other researchers to test and improve safety techniques. Open systems can be more compliant with AI regulation than their closed counterparts ; a recent Stanford University study assessed foundation model compliance with the EU AI Act and found while many model providers only score less than 25%, such as AI21 Labs, Aleph Alpha, and Anthropic, Hugging Face's BigScience was the only model provider to score above 75%. Another organization centered on openness, EleutherAI, scored highest on disclosure requirements. Openness bolsters transparency and enables external scrutiny.\n\nThe AI field is currently dominated by a few high-resource organizations who give limited or no open access to novel AI systems, including those based on open research. In order to encourage competition and increase AI economic opportunity, we should enable access for many people to contribute to increasing the breadth of AI progress across useful applications, not just allow a select few organizations to improve the depth of more capable models.\n\nWorking with different professions to ensure AI augments workers and improves productivity requires increased access to AI systems and community contributions to determine AI utility through experimentation, adaptation, and fine-tuning . Shared resources can be built and improved over time with multidisciplinary expertise, and leverage the country's public institutions so they reflect democratic values. Hugging Face is particularly invested in the role of machine learning libraries in curating the content of AI systems.\n\n## Mandates and Mechanisms for Transparency and Safeguards\n\nAll AI systems, regardless of their level of openness, require mechanisms for better understanding and communicating aspects of the system. This includes detailed and easily consumable documentation and better risk evaluations combined with increased research for technical and policy safeguards.\n\nRigorous documentation practices for AI systems, with transparent reporting that follows well-defined protocols, serves three main goals: incentivizing responsible development; ensuring researchers and developers consider values and priorities that may otherwise be overlooked; and creating a paper trail for review. Documentation highlighting heavy risks disincentivizes developers from further development in that unsafe direction. Guidance for what to document, such as funding sources, can aid scrutiny across multiple dimensions. Accessible documentation that is readable to many audiences and able to be examined by additional parties further down the pipeline can improve understanding of the technology as a whole, including tradeoffs in benefits and risks.\n\nImportantly, documentation should be detailed and applied to AI models, datasets, and all relevant components of an AI system. Hugging Face leads by example in documenting models on our platform with over 250,000 model cards, a vastly adopted structure for documentation that can be a strong template for AI models. Datasheets are the parallel for datasets and are also widely adopted. Documentation should also cover processes and governance mechanisms, which we exemplified in a Governance Card for the BigCode initiative we are co-leading with ServiceNow to create an open-source generative code model. We also lower the barrier for users to create documentation with easy tooling that generates documentation.\n\nIn order to document and mitigate risks, we need to understand how to measure them. The current state of AI evaluations, especially for complex social impacts such as biases and environmental costs, requires more resourcing and central fora for testing, as well as transparency from entities about how and where they deploy AI systems to understand what evaluations are most urgently needed. Risk evaluation and mitigation work should be built on technical feasibility and existing research. National Institute for Standards and Technology's (NIST) work on moving toward a standard for AI bias and the AI risk management framework have been excellent examples of technical leadership on safe AI innovation in the U.S. government and guidance for the AI ecosystem. Increased funding for NIST would both strengthen U.S. government technical leadership and improve the space for researchers across sectors to collaborate on addressing AI risks.\n\nSafeguards should be developed in tandem with AI systems, and should be a combination of technical, policy, and legal approaches. Technical approaches include detection models, such as the GPT-2 output detector hosted on our Hub, and safety filters, such as the one deployed with Stable Diffusion. Organizational and platform policies should be specific to content and intended system use; we enable researchers to manage access to sensitive systems to promote accountability without facilitating malicious use; in addition to labeling and moderating content that is not for all audiences. And novel legal approaches, such as Open Responsible AI Licenses (RAIL) can provide enforcement mechanisms for abiding by intended uses.\n\n## Invest in Infrastructure and Research\n\nAI innovation across many high value applications requires investing in relevant expertises and accounting for resource imbalances among disciplines and sectors. The long-standing and widening resource divides especially between industry and academia limit who is able to contribute to innovative research and applications. We strongly support the U.S. National AI Research Resource (NAIRR) and resourcing small businesses and startups conducting public interest research. We recommend incorporating AI system access needs with infrastructure access.\n\nTraining and educational resources such as courses and tooling can encourage interdisciplinary research and more engagement from untapped sectors. This can aid professions interested in leveraging AI in conducting complex tasks, such as training an AI model. Our Evaluate library provides tutorials and no-code interfaces to run technical evaluations on our hosted models. AutoTrain empowers anyone regardless of technical skill to train, evaluate, and deploy a natural language processing (NLP) model. Hugging Face Tasks and Spaces enable anyone to build and engage with tasks.\n\nGlobal leadership in democratic values should include our international allies. Our platform encourages many groups to curate datasets and evaluations in their native or fluent languages, leading to stronger, representative science. Tooling and resources should be made available across disciplines and accessible to the many languages and perspectives needed to drive safe innovation.\n\nFurthermore, while high-level guidance is needed to converge the AI community toward similar practices, government-provided resources are most easily applied when tailored to specific systems . We recommend increasing NIST capacity to develop AI RMF profiles for popularly used systems, such as language models. We are excited about f ederal support of data commons and infrastructure, such as current efforts to make NSF funded data more broadly available . Technical experts with a track record of ethical innovation should be prioritized as advisors for resourcing initiatives.\n\n## Conclusion\n\nThe incredible capability and promise for AI to improve American lives and opportunities requires cross-sectoral collaboration, access to systems, and investing in safety. We thank the Committee for the opportunity to discuss this important issue and look forward to continuing to support a safe, innovative U.S. AI ecosystem.", + "start_char_id": 0, + "end_char_id": 11019 + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2023_NTIA_Response.pdf", + "md_content": "Hugging Face 29 June 2022\n\n\n\n20 Jay St Suite 620 New York, NY 71201\n\n## Hugging Face Comments on AI Accountability for the National Telecommunications and Information Administration\n\nHugging Face commends the National Telecommunications and Information Administration (NTIA) Task Force on its extensive work framing the different aspects and components of accountability of AI systems. The following comments are informed by our experiences as an open platform for state-of-the-art (SotA) AI systems, working to make AI accessible and broadly available to researchers for responsible development. Comments are organized by section and by question. If a section or question is not highlighted, we do not have specific, actionable feedback.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## AI Accountability Objectives\n\nQuestion 1: What is the purpose of AI accountability mechanisms such as certifications, audits, and assessments?\n\nAs new applications of technology move from academic research and experimental development to broader integration and deployment within society, they start shaping the lives and experiences of millions to billions of direct and indirect users. Accountability mechanisms such as certifications, audits, or assessments help ensure that the safety, security, and well-being of these users are formally taken into account, and that the many choices that frame the technology's development verifiably contribute to addressing the potential negative societal impacts that may come from insufficient foresight or consideration.\n\nThese accountability mechanisms are particularly needed for AI-enabled technology especially Machine Learning (ML) systems - on two main accounts. First, as a data-driven technology, ML constitutes a sociotechnical system. The many modes of interaction between data subjects, algorithm subjects, and the various stages of the technology development make it particularly difficult to predict the impact of a single development choice in isolation, and make the technology prone to exacerbating social biases and historical discrimination [1]. The severity of these risks is further increased by AI systems' ability to be deployed at scales that were previously harder to achieve.\n\nSecond, Machine Learning is shifting from an academic discipline whose benchmarks were developed to compare the contribution of different scientific innovations in an open and controlled setting to a product race in a multi-billion dollar market with very different constraints and priorities. This shift is creating an accountability gap, one that is particularly marked with respect to the technology's social impact. Indeed:\n\n- 1. The academic benchmarks that have shaped the development of ML technology to date have targeted mostly internally relevant notions of performance and generalization, to the detriment of evaluation of in-context behavior and potential negative impacts [2]. Even evaluations that do address phenomena such as bias and discrimination tend to do so in an abstract way that tends to lack proper contextualization, limiting their utility in preventing real-world harms when systems are deployed [3].\n- 2. Performance on an academic benchmark is not by itself a sufficient guarantee of appropriateness for real-world deployment. Adopting ML systems for a range of applications without demonstrating their fitness for the purpose has led to a crisis referred to as 'Fallacy of A functionality' [4] and prompted the FTC to remind developers to better align their claims about the positive contributions of their AI systems with actual verifiable and measurable improvements [5].\n- 3. Shifting evaluation of ML systems from an open setting that allows for external scrutiny and reproduction to evaluation behind closed doors without mechanisms for independent verification is also putting their reliability and legitimacy into question - and in particular raising questions about data contamination, a common issue with ML system evaluation [6][7]. In particular, transparency has proven necessary to evaluate how ML artifacts may exacerbate risks of discrimination [8], whereas interventions that aim to address these issues in commercial systems without external accountability have proven unreliable in the past [9].\n- 4. Finally, as systems are deployed at scale, the introduction of AI may shift burdens in a way that benefits the deployer but incurs additional costs for other parts of society, as has been shown by the additional work required of education professionals over the last year in the wake of the deployment of generative AI as a product [10].\n\nGiven both the nature of these systems and the rapid shift in their evaluation needs, new accountability mechanisms are thus required and should focus on defining and verifying due diligence in terms of:\n\n- 1. Verifying that a marketed or deployed system is evaluated for functionality in a well-defined application case. This includes limiting the cost incurred by society at large when an entity deploys a system at scale that performs differently from what has been advertised\n\n- 2. Mitigating ML's risk of exacerbating systemic issues, such as historical discrimination\n- 3. Providing basic security guarantees to users of the system, including but not limited to ensuring a safe workplace, promoting information security, and putting safeguards so that unintentional potentially harmful side-effects of the technology application are detected and mitigated in a timely manner.\n\nQuestion 2: Is the value of certifications, audits, and assessments mostly to promote trust for external stakeholders or is it to change internal processes? How might the answer influence policy design?\n\nAccountability mechanisms serve a dual role of:\n\n- 1. shaping the development and deployment of AI-enabled technology, and\n- 2. providing assurances that broadly used technology meets a minimum standard of reliability, responsibility, and regulatory compliance.\n- On (1), the technology development and deployment is shaped by the requirement to think through, learn about, and address the different questions or specifications relevant to the accountability mechanism. It gives developers a concrete specification of their responsibilities in terms of the broader impact of their technology, one that is subject to governance by the appropriate public and governmental institutions and that they shall prioritize on the same level as other performance targets; in other words, they codify a notion of due diligence and social responsibility of the entities putting AI systems into service. Additionally, when accountability mechanisms are paired with transparency (such as open reporting of results to a third party), 'good practices' are incentivized by virtue of the fact that technologists will generally not want to show poor work or bad results.\n- On (2), certifications, audits, assessments help close the accountability gap introduced by the shift from science to product by verifying claims. They also encourage good practice by making sure that the systems are ready to show proof of good practice and verification.\n\nAI accountability mechanisms that leverage transparency requirements also promote a healthier ecosystem of open research and collaborative development, by letting all stakeholders make informed comments and contributions to how the technology should be shaped - including the direct and indirect users of the technology who best know how their needs might best shape the technology. With the right compliance support for small and medium actors, accountability mechanisms will speed up, rather than slow down, the development of technology for the benefit of all.\n\nBoth internal and external audits have complementary roles to play. Internal audits can surface the necessary information to pre-certify systems and guide developers in their effort to examine their own systems. They are particularly useful in cases that require monitoring of a system behavior over a lengthy period of time, such as monitoring failures, evaluating performance drift, and providing early warning of discriminatory outcome trends. External audits on the other hand can be warranted either to empower external stakeholders who have reason to suspect the system is engendering harms, or to verify the faithfulness and accuracy of documentation produced by the system developer [11].\n\nWith these considerations in mind, we recommend the following:\n\n- \u25cf In order to promote good practice in technology development, internal accountability mechanisms should focus on process as well as measures and metrics in defining obligations for the developers.\n- \u25cf Both internal and external accountability mechanisms should prioritize transparency as well as sharing results and documentation as often as possible, to allow for more diverse contributions and more informed choices from potential users and affected stakeholders.\n- \u25cf External accountability mechanisms should prioritize agency by and communication to external stakeholders. External accountability processes can respond to concerns about the behavior of a system, and should include provisions for responding to request from affected populations. They should also document their findings in a way that is primarily intelligible to those categories of stakeholders.\n\nQuestion 5: Given the likely integration of generative AI tools such as large language models ( e.g., ChatGPT) or other general-purpose AI or foundational models into downstream products, how can AI accountability mechanisms inform people about how such tools are operating and/or whether the tools comply with standards for trustworthy AI?\n\nGeneral-purpose AI systems (GPAIs) outline the strong need for modular and properly articulated accountability mechanisms at every stage of the development chain, as well as the importance of transparency as a general requirement. GPAIs constitute a class of ML systems that can support a broad range of applications - in some cases because they can take varied forms of inputs, and often with some additional work to adapt them to a specific use case through techniques like Parameter-Efficient Fine-Tuning (PEFT) or Low-Rank Adaptation (LoRA) that allow for fine-tuning at a fraction of the original training cost.\n\nGPAIs present a unique opportunity to develop AI-enabled technology much easier than if developers had to start from scratch for every application. They also present unique challenges for accountability, and exacerbate all of the issues outlined above (see our answer to Question 1). In particular, they are often marketed as systems that can be applied to any settings without sufficient validation [5]. They also typically introduce more distance between their initial development setting and their practical application in technology, making it harder for down-stream users to understand e.g. how the initial training data choices may give rise to risks of discrimination or other harms in a specific use case [1].\n\nExtensive literature has outlined the role of transparency and documentation in addressing these challenges. Dataset Sheets [12] and Data Statements [13] provide an entry point into the datasets that shape the GPAIs , and have seen broad adoption, and have inspired the Dataset Cards used to describe datasets on the Hugging Face Hub [14]. When accompanied by additional tooling and visualization of the content of a dataset [15][16], they can help support post-hoc analysis of a system's fitness for a specific use case. Model cards [17] play a similar role in giving a model's prospective user a summary of the model's main characteristics , including its performance on established benchmarks, its biases and limitations, and uses that may be intended by the developers or, conversely, uses that may be out of scope [18]. New licensing schemes such as [19] also help provide a legal framework for this specification by letting developers of GPAIs to make the systems available for broad uses while prohibiting applications that are particularly likely to cause harm without proper additional testing and guardrails [20].\n\nDocumentation, however, belongs to the category of 'internal' accountability mechanisms. While those are necessary to promote responsible technology development, they are not by themselves sufficient for trustworthy and reliable technology [11]. Even when entities do their best to document biases in their systems, there is only so much a small team with mostly technical expertise can analyze - especially when the teams' primary focus is on other definitions of technical performance that are perceived to be more instrumental to the product's success. In practice, external scrutiny of commercial system has been instrumental in showcasing how their biases may directly affect downstream users, from face recognition [21] to image generation [24] and chatbots [23].\n\nThe method of release of GPAIs affects what types of safeguards are more applicable to a given system. For example, an AI model deployed via an API can be rate-limited, where users are limited to a certain number of outputs per timeframe to prevent mass generation and harms such as disinformation spreading. Ultimately, safeguards cannot be solely technical and should rely on a combination of measures, such as robust documentation and community feedback. Mechanisms for trustworthiness can also be applied at the software, hardware, and institutional levels.\n\nQuestion 6: The application of accountability measures (whether voluntary or regulatory) is more straightforward for some trustworthy AI goals than for others. With respect to which trustworthy AI goals are there existing requirements or standards? Are there any trustworthy AI goals that are not amenable to requirements or standards? How should accountability policies, whether governmental or non-governmental, treat these differences?\n\nSafety and effectiveness are most amenable to traditional requirements and standards. Notice and explanation and human alternatives require appropriate scoping, but are easier to verify.\n\nData privacy stands out as a trustworthy AI goal that is strongly process-based, and depends chiefly on good data governance and on the formalization and enforcement of individual privacy rights [25]. While the governance requirements may depend on some technical aspects of the ML systems, such as their likelihood of memorizing individual data points [26], the most efficient privacy protections come from the data selection, processing, and management [27].\n\nAlgorithmic discrimination comes from the combination of the biases encoded in the ML systems and the choice of deployment settings and application [28]. While strong requirements and standards at both levels are necessary to lowering the likelihood of discrimination, neither should be thought to be sufficient. At the system level, model developers should provide sufficient information so that a model's inherent biases can be evaluated, whether they commit to working on these evaluations themselves or provide sufficient access to the model and dataset for other entities to do so. These evaluations should focus on surfacing patterns in how protected categories are represented in the data and model. At the deployment level, developers should be held accountable for choosing ML systems that have the least risk of perpetuating historical discriminations in their application setting. Regulatory tools like the EU AI Liability directive [29] can help clarify these requirements and the developers' responsibility for negative outcomes in a particular application setting.\n\nQuestion 7: Are there ways in which accountability mechanisms are unlikely to further, and might even frustrate, the development of trustworthy AI? Are there accountability mechanisms that unduly impact AI innovation and the competitiveness of U.S. developers?\n\nGiven the breadth of the new applications of AI technology and the scale of its deployment, accountability mechanisms need to be able to leverage inputs from as much of society as possible, including developers, academic researchers, advocacy groups, and journalists to support regulators and agencies in their goals. While the companies and individuals developing the technology should be encouraged to do their best to take the safety of their systems into account, requiring them to understand the full scope of social impact of a technology with such rapid development and adoption as well as navigate complex value tension before different stakeholders is both unrealistic and contrary to democratic values.\n\nTo that end, these mechanisms should favor transparency whenever possible, so that said stakeholders may interrogate and critique the development choices before they lead to significant harms. They should also consider the needs of nonprofit actors, academic researchers, and small and medium enterprises who can explore alternative ways of building ML systems in open settings, by facilitating regulatory compliance for less-resourced entities. Contrary to common narratives, regulation that drives a greater range of actors to innovate new ways of developing robust and reliable technology accelerates a holistic definition of progress, and makes US companies more competitive in anything but the shortest time horizon.\n\nConversely, we caution against accountability mechanisms that are likely to take governance of AI systems out of public awareness and enshrine the role of a few companies in unilaterally shaping its development - at a risk of giving them an outsized influence on the values embedded in this technology.\n\n## Accountability Subjects\n\nQuestion 15: The AI value or supply chain is complex, often involving open source and proprietary products and downstream applications that are quite different from what AI system developers may initially have contemplated. Moreover, training data for AI systems may be acquired from multiple sources, including from the customer using the technology. Problems in AI systems may arise downstream at the deployment or customization stage or upstream during model development and data training.\n\n## a. Where in the value chain should accountability efforts focus?\n\nAccountability efforts need to be distributed across the value chain, since most harms enabled by AI systems come from a combination of decisions made at various levels of the development process.\n\nTraining dataset collection and management should respect data subject rights, including privacy and applicable intellectual property rights. Training data selection encodes specific values and behaviors, including harmful social biases, into ML systems, which should be the focus of accountability mechanisms for all actors downstream in the development chain. Model training algorithms have a direct influence on what the model memorizes and encodes. Controls at the deployment chain can monitor for unwanted behaviors and help mitigate risks by triggering timely interventions. Finally, choosing where and when to deploy an AI system, and which natural persons will be actively and passively affected by it, shapes both who benefits and who bears the risk from AI deployment.\n\nWhile it could be tempting to focus accountability efforts purely on the last stage of deployment, such an approach risks introducing a mis-alignment between the work and forethought required at each stage to promote responsible development and the capacity of the different actors across the development chain. Explicitly distributing accountability makes it more likely that each actor will have better tools and ML components to work with, and be better able to meet their own requirements.\n\n- b. How can accountability efforts at different points in the value chain best be coordinated and communicated?\n\nGiven the dependence of accountability efforts at any point of the value chain on decisions made at other points, it is paramount that people working both on the development and on audits or assessments have access to sufficient information about both development choices and about the outcomes of assessments by other actors.\n\nThis access can take several forms. Access to technical artifacts and documentation (including documentation created by audits and assessment) can be granted to the general public, to researchers, developers, and auditors who work directly with the artifacts, or solely to accredited external auditors. Projects like BigScience [30] and BigCode [31] showcase an approach for combining the first two options. In both projects, datasets and models prioritized fully open release whenever possible, and provide access on a case-by-case basis for datasets or artifacts with stronger privacy concerns [32]. For data, specifically, when full access cannot be provided to external researchers, a minimum standard of disclosure should cover when the data was collected, from what sources, and how it was processed .\n\nWhatever the level of access to the artifacts, strong standards for documentation, including minimal requirements for the information that needs to be included in a dataset or model card, can help actors across the value chain make informed decisions that can then be held accountable for their consideration of their social impacts. In particular, the level of information provided should be sufficient to support the analysis required when a component is used in a new application setting , particularly in the case of component that are used in the development of General Purpose AI systems.\n\nQuestion 16: The lifecycle of any given AI system or component also presents distinct junctures for assessment, audit, and other measures. For example, in the case of bias, it has been shown that '[b]ias is prevalent in the assumptions about which data should be used, what AI models should be developed, where the AI system should be placed-or if AI is required at all.' How should AI accountability mechanisms consider the AI lifecycle?\n\n- a. Should AI accountability mechanisms focus narrowly on the technical characteristics of a defined model and relevant data? Or should they feature other aspects of the socio- technical system, including the system in which the AI is embedded? When is the narrower scope better and when is the broader better? How can the scope and limitations of the accountability mechanism be effectively communicated to outside stakeholders?\n\nBoth aspects of accountability mechanisms are necessary and complementary. Some aspects of a base system or dataset can and should be measured in isolation, including its performance on technical benchmarks and some categories of encoded biases. Some aspects of accountability mechanisms should focus on the impact of a system in use, including risk monitoring, robustness to changes in the input distribution, and fairness at the level of the impact on the active and passive users (as opposed to at the level of the model outputs). A standardized label for ML systems that outlines what aspects have or have not been assessed as well as the outcome of the assessment can help stakeholders better contextualize the behavior of AI systems, and trigger further investigation as required.\n\n- b. How should AI audits or assessments be timed? At what stage of design, development, and deployment should they take place to provide meaningful accountability?\n\nAI assessments of ML components such as datasets or models can be most effective at the following stages:\n\n- \u25cf When a component is first made available , either in limited capacity to actors working on a different part of the development chain or when it is shared more broadly:\n- \u25cb For example: a report detailing training data, pre-processing, model size, and in-scope vs out-of-scope applications.\n- \u25cf When a component is integrated in a system , or used in a different part of the development chain:\n- \u25cb Biases and performance can be evaluated in the context of the new proposed use. For example, the first time a language dataset is used to pretrain a system to support automatic content moderation can trigger new analysis of the sentiment associated with different marginalized identities in the dataset.\n- \u25cf When an AI system using the component is deployed in a concrete application setting:\n- \u25cb A well scoped application provides further information about which aspects of a model or dataset should be the focus of additional scrutiny.\n\nIn all of the cases outlined above, the results of the assessment should be added to the documentation of the component, allowing users and researchers to more easily benefit from this work.\n\n## Accountability Inputs and Transparency\n\nQuestion 22: How should the accountability process address data quality and data voids of different kinds? For example, in the context of automated employment decision tools, there may be no historical data available for assessing the performance of a newly deployed, custom-built tool. For a tool deployed by other firms, there may be data a vendor has access to, but the audited firm itself lacks. In some cases, the vendor itself may have intentionally limited its own data collection and access for privacy and security purposes. How should AI accountability requirements or practices deal with these data issues? What should be the roles of government, civil society, and academia in providing useful data sets (synthetic or otherwise) to fill gaps and create equitable access to data?\n\nCurating appropriate test sets should be a part of the development process for deployed systems; if a model has not been evaluated across performance and risk considerations, it should not be deployed. In order to facilitate this accountability process, and as part of its implementation, companies should be incentivized to make as much of these test sets available as possible - using pseudonymization or other post-processing as necessary to protect the privacy of their users. Maintaining and growing such a commons of in-context performance and bias evaluations will help significantly advance our understanding of risks tied to concerns such as representational harms. Additionally, agencies responsible for conducting audits can help bootstrap evaluation benchmarks for novel tasks and help maintain them with contributions from audited entities.\n\nContracts between vendors and audited entities should include limited additional data transfer to the auditing entity in case of audits. Agencies may help vendors identify what subsets of the data are relevant, and use it for the sole purpose of the audit.\n\nQuestion 23: How should AI accountability 'products' ( e.g., audit results) be communicated to different stakeholders? Should there be standardized reporting within a sector and/or across sectors? How should the translational work of communicating AI accountability results to affected people and communities be done and supported?\n\nWe strongly advocate for a centralized repository of audits, which will help build stronger methodology and avoid duplication of work (e.g. by re-evaluating common components of systems). We recommend the repository be as open as possible, which will allow civil society and advocacy organizations to participate in the translational work of communicating and interpreting the results.\n\nStandardized records should describe the targets of the audit, the methodology, and the tools used. Components of the analysis should be directly shared when possible, extensively described when not. A centralized repository will help shape good practices and a common understanding of what constitutes sufficient documentation of the process between different stakeholders.\n\n## Barriers to Effective Accountability\n\nQuestion 24: What are the most significant barriers to effective AI accountability in the private sector, including barriers to independent AI audits, whether cooperative or adversarial? What are the best strategies and interventions to overcome these barriers?\n\nThe largest barriers are the lack of information shared about systems, lack of legal protections for novel AI impacts, the complexity of impact evaluations, and the stochastic nature of evolving AI harms. System components include models and datasets, but also filters and additional technical and decision-making documentation. Furthermore, there is currently no legal mandate that users are informed about their interaction with an AI system. Mandatory disclosure of an AI system's use and labeling content as AI generated should include uniquely identifying the AI system used.\n\nIn addition, current legal mechanisms create barriers for individuals to gain information about systems; Freedom of Information Act (FOIA) requests are the closest equivalent but are hard to enforce and operationalize without experience. Legal mechanisms should be created for affected individuals to inspect specific system decisions.\n\nThe stochastic nature of the harms make them unreliable to reproduce and investigate. Developers' are often unable to patch for specific cases without addressing underlying issues, further complicating the process. This issue could be partly addressed by requiring entities deploying models to clearly mark changes in the versions of the underlying models, and ideally by letting users opt to access previous versions for the purpose of investigating specific behaviors.\n\nFinally, evaluating systems for their fitness for purpose requires robust evaluations across functions that are not standardized and differ by system type. Research toward evaluating social impacts of systems is ongoing, but sees large gaps in assessing or quantifying inherently qualitative aspects of outputs.\n\nQuestion 25: Is the lack of a general federal data protection or privacy law a barrier to effective AI accountability?\n\nComprehensive privacy legislation, such as the EU's General Data Protection Regulation (GDPR), is an important instrument to govern the use of personal data by AI systems. Federal data protection would give a direct recourse to data subjects and mandate responsible data management.\n\nQuestion 26: Is the lack of a federal law focused on AI systems a barrier to effective AI accountability?\n\nFederal regulation should evolve in light of AI systems, which is different from having a single piece of legislation on AI systems. Updated definitions should appear in relevant regulations. Better definitions are needed for: personal data; non-discrimination and liability; and due diligence in right to explanation in sectoral regulations.\n\nQuestion 27: What is the role of intellectual property rights, terms of service, contractual obligations, or other legal entitlements in fostering or impeding a robust AI accountability ecosystem? For example, do nondisclosure agreements or trade secret protections impede the assessment or audit of AI systems and processes? If so, what legal or policy developments are needed to ensure an effective accountability framework?\n\nFostering novel legal approaches such as RAIL licenses and similar terms of service, if sufficiently well scoped, can help support the development of generally useful systems while holding users accountable across the value chain. Trade secrets and nondisclosure agreements cannot override the need to share technical details for the purpose of external auditing.\n\nQuestion 28: What do AI audits and assessments cost? Which entities should be expected to bear these costs? What are the possible consequences of AI accountability requirements that might impose significant costs on regulated entities? Are there ways to reduce these costs? What are the best ways to consider costs in relation to benefits?\n\nAI audits and assessments incur different categories of costs, including:\n\n- \u25cf technical expertise - an audit requires time from a person with the expertise to understand the technical choices made.\n- \u25cf legal and standards expertise - complying with formal requirements may require involvement from legal experts, lawyers, or accredited experts\n- \u25cf deployment and social context expertise - an effective audit requires proper representation of outside interests and understanding of social harms of a system\n- \u25cf data creation and annotation - in most cases, benchmarks will need to be either created or modified for the evaluation\n- \u25cf computational resources - larger models and more complex systems require access to specialized hardware, and can incur substantive computational costs\n\nThese costs are be shared between the entities participating in the development of AI systems, the entities tasked with external audits, and broader participation from civil society. Developing entities need to follow internal accountability practices of documentation and testing. They typically have the technical expertise and computational resourced and engage in data creation in the course of developing an AI system.\n\nEntities tasked with external audits have similar resource needs. We note however that where external audits can rely on work from a research community or other independent investigation into the audited systems, an efficient assesment can run with fewer costs.\n\nIn any case, we recommend paying particular attention to the cost of legal expertise to meet formal compliance requirements or obtain certification for standards - as those can be much more easily met by large companies than by academic or non-profit actors and Small and Medium Enterprises developing ML systems. Determining the right threshold for requiring a formal certification will be necessary for allowing different actors to participate in research and development of AI systems\n\nRespectfully, Yacine Jernite and Irene Solaiman\n\nHugging Face", + "chunks": [ + { + "headings": [ + "Hugging Face 29 June 2022" + ], + "markdown": "Hugging Face 29 June 2022\n\n\n\n20 Jay St Suite 620 New York, NY 71201\n\n## Hugging Face Comments on AI Accountability for the National Telecommunications and Information Administration\n\nHugging Face commends the National Telecommunications and Information Administration (NTIA) Task Force on its extensive work framing the different aspects and components of accountability of AI systems. The following comments are informed by our experiences as an open platform for state-of-the-art (SotA) AI systems, working to make AI accessible and broadly available to researchers for responsible development. Comments are organized by section and by question. If a section or question is not highlighted, we do not have specific, actionable feedback.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## AI Accountability Objectives\n\nQuestion 1: What is the purpose of AI accountability mechanisms such as certifications, audits, and assessments?\n\nAs new applications of technology move from academic research and experimental development to broader integration and deployment within society, they start shaping the lives and experiences of millions to billions of direct and indirect users. Accountability mechanisms such as certifications, audits, or assessments help ensure that the safety, security, and well-being of these users are formally taken into account, and that the many choices that frame the technology's development verifiably contribute to addressing the potential negative societal impacts that may come from insufficient foresight or consideration.\n\nThese accountability mechanisms are particularly needed for AI-enabled technology especially Machine Learning (ML) systems - on two main accounts. First, as a data-driven technology, ML constitutes a sociotechnical system. The many modes of interaction between data subjects, algorithm subjects, and the various stages of the technology development make it particularly difficult to predict the impact of a single development choice in isolation, and make the technology prone to exacerbating social biases and historical discrimination [1]. The severity of these risks is further increased by AI systems' ability to be deployed at scales that were previously harder to achieve.\n\nSecond, Machine Learning is shifting from an academic discipline whose benchmarks were developed to compare the contribution of different scientific innovations in an open and controlled setting to a product race in a multi-billion dollar market with very different constraints and priorities. This shift is creating an accountability gap, one that is particularly marked with respect to the technology's social impact. Indeed:\n\n- 1. The academic benchmarks that have shaped the development of ML technology to date have targeted mostly internally relevant notions of performance and generalization, to the detriment of evaluation of in-context behavior and potential negative impacts [2]. Even evaluations that do address phenomena such as bias and discrimination tend to do so in an abstract way that tends to lack proper contextualization, limiting their utility in preventing real-world harms when systems are deployed [3].\n- 2. Performance on an academic benchmark is not by itself a sufficient guarantee of appropriateness for real-world deployment. Adopting ML systems for a range of applications without demonstrating their fitness for the purpose has led to a crisis referred to as 'Fallacy of A functionality' [4] and prompted the FTC to remind developers to better align their claims about the positive contributions of their AI systems with actual verifiable and measurable improvements [5].\n- 3. Shifting evaluation of ML systems from an open setting that allows for external scrutiny and reproduction to evaluation behind closed doors without mechanisms for independent verification is also putting their reliability and legitimacy into question - and in particular raising questions about data contamination, a common issue with ML system evaluation [6][7]. In particular, transparency has proven necessary to evaluate how ML artifacts may exacerbate risks of discrimination [8], whereas interventions that aim to address these issues in commercial systems without external accountability have proven unreliable in the past [9].\n- 4. Finally, as systems are deployed at scale, the introduction of AI may shift burdens in a way that benefits the deployer but incurs additional costs for other parts of society, as has been shown by the additional work required of education professionals over the last year in the wake of the deployment of generative AI as a product [10].\n\nGiven both the nature of these systems and the rapid shift in their evaluation needs, new accountability mechanisms are thus required and should focus on defining and verifying due diligence in terms of:\n\n- 1. Verifying that a marketed or deployed system is evaluated for functionality in a well-defined application case. This includes limiting the cost incurred by society at large when an entity deploys a system at scale that performs differently from what has been advertised\n\n- 2. Mitigating ML's risk of exacerbating systemic issues, such as historical discrimination\n- 3. Providing basic security guarantees to users of the system, including but not limited to ensuring a safe workplace, promoting information security, and putting safeguards so that unintentional potentially harmful side-effects of the technology application are detected and mitigated in a timely manner.\n\nQuestion 2: Is the value of certifications, audits, and assessments mostly to promote trust for external stakeholders or is it to change internal processes? How might the answer influence policy design?\n\nAccountability mechanisms serve a dual role of:\n\n- 1. shaping the development and deployment of AI-enabled technology, and\n- 2. providing assurances that broadly used technology meets a minimum standard of reliability, responsibility, and regulatory compliance.\n- On (1), the technology development and deployment is shaped by the requirement to think through, learn about, and address the different questions or specifications relevant to the accountability mechanism. It gives developers a concrete specification of their responsibilities in terms of the broader impact of their technology, one that is subject to governance by the appropriate public and governmental institutions and that they shall prioritize on the same level as other performance targets; in other words, they codify a notion of due diligence and social responsibility of the entities putting AI systems into service. Additionally, when accountability mechanisms are paired with transparency (such as open reporting of results to a third party), 'good practices' are incentivized by virtue of the fact that technologists will generally not want to show poor work or bad results.\n- On (2), certifications, audits, assessments help close the accountability gap introduced by the shift from science to product by verifying claims. They also encourage good practice by making sure that the systems are ready to show proof of good practice and verification.\n\nAI accountability mechanisms that leverage transparency requirements also promote a healthier ecosystem of open research and collaborative development, by letting all stakeholders make informed comments and contributions to how the technology should be shaped - including the direct and indirect users of the technology who best know how their needs might best shape the technology. With the right compliance support for small and medium actors, accountability mechanisms will speed up, rather than slow down, the development of technology for the benefit of all.\n\nBoth internal and external audits have complementary roles to play. Internal audits can surface the necessary information to pre-certify systems and guide developers in their effort to examine their own systems. They are particularly useful in cases that require monitoring of a system behavior over a lengthy period of time, such as monitoring failures, evaluating performance drift, and providing early warning of discriminatory outcome trends. External audits on the other hand can be warranted either to empower external stakeholders who have reason to suspect the system is engendering harms, or to verify the faithfulness and accuracy of documentation produced by the system developer [11].\n\nWith these considerations in mind, we recommend the following:\n\n- \u25cf In order to promote good practice in technology development, internal accountability mechanisms should focus on process as well as measures and metrics in defining obligations for the developers.\n- \u25cf Both internal and external accountability mechanisms should prioritize transparency as well as sharing results and documentation as often as possible, to allow for more diverse contributions and more informed choices from potential users and affected stakeholders.\n- \u25cf External accountability mechanisms should prioritize agency by and communication to external stakeholders. External accountability processes can respond to concerns about the behavior of a system, and should include provisions for responding to request from affected populations. They should also document their findings in a way that is primarily intelligible to those categories of stakeholders.\n\nQuestion 5: Given the likely integration of generative AI tools such as large language models ( e.g., ChatGPT) or other general-purpose AI or foundational models into downstream products, how can AI accountability mechanisms inform people about how such tools are operating and/or whether the tools comply with standards for trustworthy AI?\n\nGeneral-purpose AI systems (GPAIs) outline the strong need for modular and properly articulated accountability mechanisms at every stage of the development chain, as well as the importance of transparency as a general requirement. GPAIs constitute a class of ML systems that can support a broad range of applications - in some cases because they can take varied forms of inputs, and often with some additional work to adapt them to a specific use case through techniques like Parameter-Efficient Fine-Tuning (PEFT) or Low-Rank Adaptation (LoRA) that allow for fine-tuning at a fraction of the original training cost.\n\nGPAIs present a unique opportunity to develop AI-enabled technology much easier than if developers had to start from scratch for every application. They also present unique challenges for accountability, and exacerbate all of the issues outlined above (see our answer to Question 1). In particular, they are often marketed as systems that can be applied to any settings without sufficient validation [5]. They also typically introduce more", + "char_slice": [ + 0, + 11262 + ] + }, + { + "headings": [ + "## Hugging Face Comments on AI Accountability for the National Telecommunications and Information Administration", + "## About Hugging Face", + "## AI Accountability Objectives" + ], + "markdown": "as large language models ( e.g., ChatGPT) or other general-purpose AI or foundational models into downstream products, how can AI accountability mechanisms inform people about how such tools are operating and/or whether the tools comply with standards for trustworthy AI?\n\nGeneral-purpose AI systems (GPAIs) outline the strong need for modular and properly articulated accountability mechanisms at every stage of the development chain, as well as the importance of transparency as a general requirement. GPAIs constitute a class of ML systems that can support a broad range of applications - in some cases because they can take varied forms of inputs, and often with some additional work to adapt them to a specific use case through techniques like Parameter-Efficient Fine-Tuning (PEFT) or Low-Rank Adaptation (LoRA) that allow for fine-tuning at a fraction of the original training cost.\n\nGPAIs present a unique opportunity to develop AI-enabled technology much easier than if developers had to start from scratch for every application. They also present unique challenges for accountability, and exacerbate all of the issues outlined above (see our answer to Question 1). In particular, they are often marketed as systems that can be applied to any settings without sufficient validation [5]. They also typically introduce more distance between their initial development setting and their practical application in technology, making it harder for down-stream users to understand e.g. how the initial training data choices may give rise to risks of discrimination or other harms in a specific use case [1].\n\nExtensive literature has outlined the role of transparency and documentation in addressing these challenges. Dataset Sheets [12] and Data Statements [13] provide an entry point into the datasets that shape the GPAIs , and have seen broad adoption, and have inspired the Dataset Cards used to describe datasets on the Hugging Face Hub [14]. When accompanied by additional tooling and visualization of the content of a dataset [15][16], they can help support post-hoc analysis of a system's fitness for a specific use case. Model cards [17] play a similar role in giving a model's prospective user a summary of the model's main characteristics , including its performance on established benchmarks, its biases and limitations, and uses that may be intended by the developers or, conversely, uses that may be out of scope [18]. New licensing schemes such as [19] also help provide a legal framework for this specification by letting developers of GPAIs to make the systems available for broad uses while prohibiting applications that are particularly likely to cause harm without proper additional testing and guardrails [20].\n\nDocumentation, however, belongs to the category of 'internal' accountability mechanisms. While those are necessary to promote responsible technology development, they are not by themselves sufficient for trustworthy and reliable technology [11]. Even when entities do their best to document biases in their systems, there is only so much a small team with mostly technical expertise can analyze - especially when the teams' primary focus is on other definitions of technical performance that are perceived to be more instrumental to the product's success. In practice, external scrutiny of commercial system has been instrumental in showcasing how their biases may directly affect downstream users, from face recognition [21] to image generation [24] and chatbots [23].\n\nThe method of release of GPAIs affects what types of safeguards are more applicable to a given system. For example, an AI model deployed via an API can be rate-limited, where users are limited to a certain number of outputs per timeframe to prevent mass generation and harms such as disinformation spreading. Ultimately, safeguards cannot be solely technical and should rely on a combination of measures, such as robust documentation and community feedback. Mechanisms for trustworthiness can also be applied at the software, hardware, and institutional levels.\n\nQuestion 6: The application of accountability measures (whether voluntary or regulatory) is more straightforward for some trustworthy AI goals than for others. With respect to which trustworthy AI goals are there existing requirements or standards? Are there any trustworthy AI goals that are not amenable to requirements or standards? How should accountability policies, whether governmental or non-governmental, treat these differences?\n\nSafety and effectiveness are most amenable to traditional requirements and standards. Notice and explanation and human alternatives require appropriate scoping, but are easier to verify.\n\nData privacy stands out as a trustworthy AI goal that is strongly process-based, and depends chiefly on good data governance and on the formalization and enforcement of individual privacy rights [25]. While the governance requirements may depend on some technical aspects of the ML systems, such as their likelihood of memorizing individual data points [26], the most efficient privacy protections come from the data selection, processing, and management [27].\n\nAlgorithmic discrimination comes from the combination of the biases encoded in the ML systems and the choice of deployment settings and application [28]. While strong requirements and standards at both levels are necessary to lowering the likelihood of discrimination, neither should be thought to be sufficient. At the system level, model developers should provide sufficient information so that a model's inherent biases can be evaluated, whether they commit to working on these evaluations themselves or provide sufficient access to the model and dataset for other entities to do so. These evaluations should focus on surfacing patterns in how protected categories are represented in the data and model. At the deployment level, developers should be held accountable for choosing ML systems that have the least risk of perpetuating historical discriminations in their application setting. Regulatory tools like the EU AI Liability directive [29] can help clarify these requirements and the developers' responsibility for negative outcomes in a particular application setting.\n\nQuestion 7: Are there ways in which accountability mechanisms are unlikely to further, and might even frustrate, the development of trustworthy AI? Are there accountability mechanisms that unduly impact AI innovation and the competitiveness of U.S. developers?\n\nGiven the breadth of the new applications of AI technology and the scale of its deployment, accountability mechanisms need to be able to leverage inputs from as much of society as possible, including developers, academic researchers, advocacy groups, and journalists to support regulators and agencies in their goals. While the companies and individuals developing the technology should be encouraged to do their best to take the safety of their systems into account, requiring them to understand the full scope of social impact of a technology with such rapid development and adoption as well as navigate complex value tension before different stakeholders is both unrealistic and contrary to democratic values.\n\nTo that end, these mechanisms should favor transparency whenever possible, so that said stakeholders may interrogate and critique the development choices before they lead to significant harms. They should also consider the needs of nonprofit actors, academic researchers, and small and medium enterprises who can explore alternative ways of building ML systems in open settings, by facilitating regulatory compliance for less-resourced entities. Contrary to common narratives, regulation that drives a greater range of actors to innovate new ways of developing robust and reliable technology accelerates a holistic definition of progress, and makes US companies more competitive in anything but the shortest time horizon.\n\nConversely, we caution against accountability mechanisms that are likely to take governance of AI systems out of public awareness and enshrine the role of a few companies in unilaterally shaping its development - at a risk of giving them an outsized influence on the values embedded in this technology.\n\n## Accountability Subjects\n\nQuestion 15: The AI value or supply chain is complex, often involving open source and proprietary products and downstream applications that are quite different from what AI system developers may initially have contemplated. Moreover, training data for AI systems may be acquired from multiple sources, including from the customer using the technology. Problems in AI systems may arise downstream at the deployment or customization stage or upstream during model development and data training.\n\n## a. Where in the value chain should accountability efforts focus?\n\nAccountability efforts need to be distributed across the value chain, since most harms enabled by AI systems come from a combination of decisions made at various levels of the development process.\n\nTraining dataset collection and management should respect data subject rights, including privacy and applicable intellectual property rights. Training data selection encodes specific values and behaviors, including harmful social biases, into ML systems, which should be the focus of accountability mechanisms for all actors downstream in the development chain. Model training algorithms have a direct influence on what the model memorizes and encodes. Controls at the deployment chain can monitor for unwanted behaviors and help mitigate risks by triggering timely interventions. Finally, choosing where and when to deploy an AI system, and which natural persons will be actively and passively affected by it, shapes both who benefits and who bears the risk from AI deployment.\n\nWhile it could be tempting to focus accountability efforts purely on the last stage of deployment, such an approach risks introducing a mis-alignment between the work and forethought required at each stage to promote responsible development and the capacity of the different actors across the development chain. Explicitly distributing accountability makes it more likely that each actor will have better tools and ML components to work with, and be better able to meet their own requirements.\n\n- b. How can accountability efforts at different points in the value chain best be coordinated and communicated?\n\nGiven the dependence of accountability efforts at any point of the value chain on decisions made at other points, it is paramount that people working both on the development and on audits or assessments have access to sufficient information about both development choices and about the outcomes of assessments by other actors.\n\nThis access can take several forms. Access to technical artifacts and documentation (including documentation created by audits and assessment) can be granted to the general public, to researchers, developers, and auditors who work directly with the artifacts, or solely to accredited external auditors. Projects like BigScience [30] and BigCode [31] showcase an approach for combining the first two options. In both projects, datasets and models prioritized fully open release whenever possible, and provide access on a case", + "char_slice": [ + 9932, + 21204 + ] + }, + { + "headings": [ + "## Hugging Face Comments on AI Accountability for the National Telecommunications and Information Administration", + "## About Hugging Face", + "## AI Accountability Objectives", + "## Accountability Subjects", + "## a. Where in the value chain should accountability efforts focus?" + ], + "markdown": "to focus accountability efforts purely on the last stage of deployment, such an approach risks introducing a mis-alignment between the work and forethought required at each stage to promote responsible development and the capacity of the different actors across the development chain. Explicitly distributing accountability makes it more likely that each actor will have better tools and ML components to work with, and be better able to meet their own requirements.\n\n- b. How can accountability efforts at different points in the value chain best be coordinated and communicated?\n\nGiven the dependence of accountability efforts at any point of the value chain on decisions made at other points, it is paramount that people working both on the development and on audits or assessments have access to sufficient information about both development choices and about the outcomes of assessments by other actors.\n\nThis access can take several forms. Access to technical artifacts and documentation (including documentation created by audits and assessment) can be granted to the general public, to researchers, developers, and auditors who work directly with the artifacts, or solely to accredited external auditors. Projects like BigScience [30] and BigCode [31] showcase an approach for combining the first two options. In both projects, datasets and models prioritized fully open release whenever possible, and provide access on a case-by-case basis for datasets or artifacts with stronger privacy concerns [32]. For data, specifically, when full access cannot be provided to external researchers, a minimum standard of disclosure should cover when the data was collected, from what sources, and how it was processed .\n\nWhatever the level of access to the artifacts, strong standards for documentation, including minimal requirements for the information that needs to be included in a dataset or model card, can help actors across the value chain make informed decisions that can then be held accountable for their consideration of their social impacts. In particular, the level of information provided should be sufficient to support the analysis required when a component is used in a new application setting , particularly in the case of component that are used in the development of General Purpose AI systems.\n\nQuestion 16: The lifecycle of any given AI system or component also presents distinct junctures for assessment, audit, and other measures. For example, in the case of bias, it has been shown that '[b]ias is prevalent in the assumptions about which data should be used, what AI models should be developed, where the AI system should be placed-or if AI is required at all.' How should AI accountability mechanisms consider the AI lifecycle?\n\n- a. Should AI accountability mechanisms focus narrowly on the technical characteristics of a defined model and relevant data? Or should they feature other aspects of the socio- technical system, including the system in which the AI is embedded? When is the narrower scope better and when is the broader better? How can the scope and limitations of the accountability mechanism be effectively communicated to outside stakeholders?\n\nBoth aspects of accountability mechanisms are necessary and complementary. Some aspects of a base system or dataset can and should be measured in isolation, including its performance on technical benchmarks and some categories of encoded biases. Some aspects of accountability mechanisms should focus on the impact of a system in use, including risk monitoring, robustness to changes in the input distribution, and fairness at the level of the impact on the active and passive users (as opposed to at the level of the model outputs). A standardized label for ML systems that outlines what aspects have or have not been assessed as well as the outcome of the assessment can help stakeholders better contextualize the behavior of AI systems, and trigger further investigation as required.\n\n- b. How should AI audits or assessments be timed? At what stage of design, development, and deployment should they take place to provide meaningful accountability?\n\nAI assessments of ML components such as datasets or models can be most effective at the following stages:\n\n- \u25cf When a component is first made available , either in limited capacity to actors working on a different part of the development chain or when it is shared more broadly:\n- \u25cb For example: a report detailing training data, pre-processing, model size, and in-scope vs out-of-scope applications.\n- \u25cf When a component is integrated in a system , or used in a different part of the development chain:\n- \u25cb Biases and performance can be evaluated in the context of the new proposed use. For example, the first time a language dataset is used to pretrain a system to support automatic content moderation can trigger new analysis of the sentiment associated with different marginalized identities in the dataset.\n- \u25cf When an AI system using the component is deployed in a concrete application setting:\n- \u25cb A well scoped application provides further information about which aspects of a model or dataset should be the focus of additional scrutiny.\n\nIn all of the cases outlined above, the results of the assessment should be added to the documentation of the component, allowing users and researchers to more easily benefit from this work.\n\n## Accountability Inputs and Transparency\n\nQuestion 22: How should the accountability process address data quality and data voids of different kinds? For example, in the context of automated employment decision tools, there may be no historical data available for assessing the performance of a newly deployed, custom-built tool. For a tool deployed by other firms, there may be data a vendor has access to, but the audited firm itself lacks. In some cases, the vendor itself may have intentionally limited its own data collection and access for privacy and security purposes. How should AI accountability requirements or practices deal with these data issues? What should be the roles of government, civil society, and academia in providing useful data sets (synthetic or otherwise) to fill gaps and create equitable access to data?\n\nCurating appropriate test sets should be a part of the development process for deployed systems; if a model has not been evaluated across performance and risk considerations, it should not be deployed. In order to facilitate this accountability process, and as part of its implementation, companies should be incentivized to make as much of these test sets available as possible - using pseudonymization or other post-processing as necessary to protect the privacy of their users. Maintaining and growing such a commons of in-context performance and bias evaluations will help significantly advance our understanding of risks tied to concerns such as representational harms. Additionally, agencies responsible for conducting audits can help bootstrap evaluation benchmarks for novel tasks and help maintain them with contributions from audited entities.\n\nContracts between vendors and audited entities should include limited additional data transfer to the auditing entity in case of audits. Agencies may help vendors identify what subsets of the data are relevant, and use it for the sole purpose of the audit.\n\nQuestion 23: How should AI accountability 'products' ( e.g., audit results) be communicated to different stakeholders? Should there be standardized reporting within a sector and/or across sectors? How should the translational work of communicating AI accountability results to affected people and communities be done and supported?\n\nWe strongly advocate for a centralized repository of audits, which will help build stronger methodology and avoid duplication of work (e.g. by re-evaluating common components of systems). We recommend the repository be as open as possible, which will allow civil society and advocacy organizations to participate in the translational work of communicating and interpreting the results.\n\nStandardized records should describe the targets of the audit, the methodology, and the tools used. Components of the analysis should be directly shared when possible, extensively described when not. A centralized repository will help shape good practices and a common understanding of what constitutes sufficient documentation of the process between different stakeholders.\n\n## Barriers to Effective Accountability\n\nQuestion 24: What are the most significant barriers to effective AI accountability in the private sector, including barriers to independent AI audits, whether cooperative or adversarial? What are the best strategies and interventions to overcome these barriers?\n\nThe largest barriers are the lack of information shared about systems, lack of legal protections for novel AI impacts, the complexity of impact evaluations, and the stochastic nature of evolving AI harms. System components include models and datasets, but also filters and additional technical and decision-making documentation. Furthermore, there is currently no legal mandate that users are informed about their interaction with an AI system. Mandatory disclosure of an AI system's use and labeling content as AI generated should include uniquely identifying the AI system used.\n\nIn addition, current legal mechanisms create barriers for individuals to gain information about systems; Freedom of Information Act (FOIA) requests are the closest equivalent but are hard to enforce and operationalize without experience. Legal mechanisms should be created for affected individuals to inspect specific system decisions.\n\nThe stochastic nature of the harms make them unreliable to reproduce and investigate. Developers' are often unable to patch for specific cases without addressing underlying issues, further complicating the process. This issue could be partly addressed by requiring entities deploying models to clearly mark changes in the versions of the underlying models, and ideally by letting users opt to access previous versions for the purpose of investigating specific behaviors.\n\nFinally, evaluating systems for their fitness for purpose requires robust evaluations across functions that are not standardized and differ by system type. Research toward evaluating social impacts of systems is ongoing, but sees large gaps in assessing or quantifying inherently qualitative aspects of outputs.\n\nQuestion 25: Is the lack of a general federal data protection or privacy law a barrier to effective AI accountability?\n\nComprehensive privacy legislation, such as the EU's General Data Protection Regulation (GDPR), is an important instrument to govern the use of personal data by AI systems. Federal data protection would give a direct recourse to data subjects and mandate responsible data management.\n\nQuestion 26: Is the lack of a federal law focused on AI systems a barrier to effective AI accountability?\n\nFederal regulation should evolve in light of AI systems, which is different from having a single piece of legislation on AI systems. Updated definitions should appear in relevant regulations. Better definitions are needed for: personal data; non-discrimination and liability; and due diligence in right to explanation in sectoral regulations.\n\nQuestion", + "char_slice": [ + 19770, + 31065 + ] + }, + { + "headings": [ + "## Hugging Face Comments on AI Accountability for the National Telecommunications and Information Administration", + "## About Hugging Face", + "## AI Accountability Objectives", + "## Accountability Subjects", + "## a. Where in the value chain should accountability efforts focus?", + "## Accountability Inputs and Transparency", + "## Barriers to Effective Accountability" + ], + "markdown": "ting the process. This issue could be partly addressed by requiring entities deploying models to clearly mark changes in the versions of the underlying models, and ideally by letting users opt to access previous versions for the purpose of investigating specific behaviors.\n\nFinally, evaluating systems for their fitness for purpose requires robust evaluations across functions that are not standardized and differ by system type. Research toward evaluating social impacts of systems is ongoing, but sees large gaps in assessing or quantifying inherently qualitative aspects of outputs.\n\nQuestion 25: Is the lack of a general federal data protection or privacy law a barrier to effective AI accountability?\n\nComprehensive privacy legislation, such as the EU's General Data Protection Regulation (GDPR), is an important instrument to govern the use of personal data by AI systems. Federal data protection would give a direct recourse to data subjects and mandate responsible data management.\n\nQuestion 26: Is the lack of a federal law focused on AI systems a barrier to effective AI accountability?\n\nFederal regulation should evolve in light of AI systems, which is different from having a single piece of legislation on AI systems. Updated definitions should appear in relevant regulations. Better definitions are needed for: personal data; non-discrimination and liability; and due diligence in right to explanation in sectoral regulations.\n\nQuestion 27: What is the role of intellectual property rights, terms of service, contractual obligations, or other legal entitlements in fostering or impeding a robust AI accountability ecosystem? For example, do nondisclosure agreements or trade secret protections impede the assessment or audit of AI systems and processes? If so, what legal or policy developments are needed to ensure an effective accountability framework?\n\nFostering novel legal approaches such as RAIL licenses and similar terms of service, if sufficiently well scoped, can help support the development of generally useful systems while holding users accountable across the value chain. Trade secrets and nondisclosure agreements cannot override the need to share technical details for the purpose of external auditing.\n\nQuestion 28: What do AI audits and assessments cost? Which entities should be expected to bear these costs? What are the possible consequences of AI accountability requirements that might impose significant costs on regulated entities? Are there ways to reduce these costs? What are the best ways to consider costs in relation to benefits?\n\nAI audits and assessments incur different categories of costs, including:\n\n- \u25cf technical expertise - an audit requires time from a person with the expertise to understand the technical choices made.\n- \u25cf legal and standards expertise - complying with formal requirements may require involvement from legal experts, lawyers, or accredited experts\n- \u25cf deployment and social context expertise - an effective audit requires proper representation of outside interests and understanding of social harms of a system\n- \u25cf data creation and annotation - in most cases, benchmarks will need to be either created or modified for the evaluation\n- \u25cf computational resources - larger models and more complex systems require access to specialized hardware, and can incur substantive computational costs\n\nThese costs are be shared between the entities participating in the development of AI systems, the entities tasked with external audits, and broader participation from civil society. Developing entities need to follow internal accountability practices of documentation and testing. They typically have the technical expertise and computational resourced and engage in data creation in the course of developing an AI system.\n\nEntities tasked with external audits have similar resource needs. We note however that where external audits can rely on work from a research community or other independent investigation into the audited systems, an efficient assesment can run with fewer costs.\n\nIn any case, we recommend paying particular attention to the cost of legal expertise to meet formal compliance requirements or obtain certification for standards - as those can be much more easily met by large companies than by academic or non-profit actors and Small and Medium Enterprises developing ML systems. Determining the right threshold for requiring a formal certification will be necessary for allowing different actors to participate in research and development of AI systems\n\nRespectfully, Yacine Jernite and Irene Solaiman\n\nHugging Face", + "char_slice": [ + 29614, + 34216 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2023_OMB%20EO%20RFC.pdf", + "md_content": "\n\nHugging Face, Inc. 20 Jay Street, Suite 620, Brooklyn, NY 11201\n\n## Hugging Face Response to the Office of Management and Budget Request for Comments on Agency Use of Artificial Intelligence\n\nHugging Face commends the Office of Management and Business on its extensive work framing the responsible use of AI by federal agencies. The following comments are informed by our experiences as an open platform for state-of-the-art (SotA) AI systems, working to make AI accessible and broadly available to researchers for responsible development.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## Summary\n\nThe draft memorandum outlines a strong framework for fostering responsible AI adoption by federal agencies. In a context of commonly inflated claims about the performance of AI systems and pervasive risks of exacerbating discrimination, we particularly appreciate the proposed requirements to not only conduct thorough impact assessments, but also to demonstrate that the adoption of AI does provide added value to the agencies.\n\nIn order to best support this mission, we recommend that the OMB and the proposed AI Governance bodies prioritize mechanisms that support the open sharing of skills, resources, and information; build internal capacity along as much of the AI development chain as possible, from pre-training data curation to model fine-tuning and deployment (including, as currently outlined in Section 4.b.iv, non-technical expertise); and develop further guidance on what constitutes a sufficient improvement to warrant AI adoption and how to choose an AI system to meet those goals.\n\nFor the purpose of sharing best practices, we note that the proposed AI Use Case inventory constitutes a particularly promising tool. We recommend that entries in the inventory include as much information as possible about the AI application and system, including data and model documentation according to established standards such as dataset sheets and model cards, the metrics used to assess whether AI adoption would have a positive impact for the agency's use case, the reasoning for determining whether the system was safety- or rights-impacting, and links to the specific versioned systems used, including any open-source or open-access components.\n\n\n\nWe also recommend that the OMB extend the scope of memorandum OMB M-16-21 on open source software to include AI systems. In particular, we believe that the testing datasets developed by agencies to meet the requirements laid out in the present draft memorandum have a significant role to play in advancing the state of the art in performance and risk evaluation of AI systems if they are shared broadly with researchers and stakeholders. By supporting agencies in sharing this data to all relevant stakeholders, including by providing guidelines for the use of synthetic data, data pseudonymization, and governance of public datasets, the OMB can accelerate the development of robust responsible AI practices that will benefit all of society.\n\nBeyond evaluation datasets, we encourage the OMB to direct agencies to build capacity to develop and share their own full AI stacks, including by leveraging and adapting existing public, open-source, and open Machine Learning models and datasets. This approach can lead to more energy-efficient AI applications, more transparency into the regulatory compliance of the AI development chain, and better visibility into the handling of e.g. privacy and intellectual property rights along the development process (see for example the StarCoder open code generation LLM). To that end, and in addition to the evaluation and test, we recommend that the AI strategy of federal agencies include to the extent possible the development and sharing of:\n\n- \u25cf pre-training datasets for base models (or foundation models) focused on their domain of activity\n- \u25cf the corresponding pre-trained (or foundation) models\n- \u25cf fine-tuning datasets and fine-tuned models tailored to the agencies' needs\n- \u25cf any software needed to process the inputs and outputs of AI systems\n\nWe provide further feedback on the draft memorandum below by answering the questions accompanying the Request for Comments. If a question is not highlighted, we do not have specific, actionable feedback.\n\n## Questions\n\n- 2. What types of coordination mechanisms, either in the public or private sector, would be particularly effective for agencies to model in their establishment of an AI Governance Body? What are the benefits or drawbacks to having agencies establishing a new body to perform AI governance versus updating the scope of an existing group (for example, agency bodies focused on privacy, IT, or data)?\n\nWhile AI Governance will require involvement from many existing groups in agencies, including but not limited to privacy, IT, data, and legal departments, approaching the new challenges posed by AI adoption holistically and efficiently will require coordination and skill sharing between different stakeholders, which is best managed by a new AI Governance Body. Such a body can also facilitate broader civil society involvement and standardization of documentation practices. As examples of the possible functions of such a body, we point to the role of the cross-areas working groups within the BigScience workshop on Large Language Models, and its proposed structure for a multi-stakeholder AI data governance organization for large scale ML. In particular, we stress the importance of the role of centralizing documentation and transparency guidelines and the adoption of interoperable data formats to allow for better skill sharing within and across agencies.\n\n- 3. How can OMB best advance responsible AI innovation?\n\nThe OMB has a unique role to play in championing responsible AI innovation beyond the federal government by helping create a body of positive examples and in-context operationalizations of responsible AI principles. Other agencies and companies may then learn from those examples\n\n\n\nand leverage the resources created in the process - especially if they are shared as open and open-source AI systems - to follow suit. These should include:\n\n- \u25cf Requiring documentation of key sections of the development pipeline demonstrating decisions that respect peoples' rights and abide by applicable laws. This included:\n- \u25cb Data Development\n- \u25cb Model Training\n- \u25cb Model Evaluation\n- \u25cb System Evaluation (when models are implemented as part of a larger system)\n- \u25cb Actions for continued Monitoring\n- \u25cf Building resources and knowledge about how to assess, evaluate, document, and audit AI systems\n- \u25cb In particular, focus on building capacity to audit specific parts of AI development, including training data and procedure\n- \u25cb Build a robust reporting system including labor complaints and whistleblower protections\n- \u25cf Leveraging shared experience between agencies to build best practices informed by the practicalities of deployment\n- \u25cf Highlight that AI should in many cases *NOT* be used, and/or may not be an improvement over current/alternative systems, in particular by providing transparency through the AI Use Case inventory about the process for weighing the risks and benefits of AI adoption.\n- 4. With adequate safeguards in place, how should agencies take advantage of generative AI to improve agency missions or business operations?\n\nGenerative AI, when used for the purposes it is best suited for, can be a useful tool to explore AI in a new application setting, increase productivity in settings such as software engineering, and support applications that are less reliant on accuracy or grounding in factual information especially when it is developed with sufficient transparency, consideration for the rights of data subject, and attribution of the model outputs. In particular, generative AI systems that are designed to handle a variety of tasks or work in so-called zero-shot settings can significantly accelerate the development of a 'proof-of-concept' AI system to explore whether AI can bring value to an agency in a particular setting; possibly to then be replaced by a more efficient discriminative system.\n\nGenerative AI also introduces unique concerns, including increased financial and environmental costs when it is used in settings where a discriminative AI alternative could suffice, an increased cybersecurity attack surface and range of vulnerabilities, and a dependence on web-scale and larger training datasets that exacerbates social issues, including biases and discrimination.\n\nThe trade-offs mentioned also depend on the modality considered. Code generation models such as the openly developed StarCoder (see the project's Governance Card for more detail), have already shown their value in accelerating the process of software development, and are less subject to (but not exempt from) exacerbating social biases along the development chain. Image generation models, on the other hand, are still very likely to reproduce and exaggerate stereotypes, which makes them a poor fit for generating illustrations of people in government communications.\n\n- 5. Are there use cases for presumed safety-impacting and rights-impacting AI (Section 5 (b)) that should be included, removed, or revised? If so, why?\n\n\n\nWe note that neither privacy nor handling of medical processes are currently included in the safety section. Mishandling of medically relevant decisions by AI systems can lead to immediate personal risks - especially given the prevalence of social biases in medical datasets, and leaks of private information can expose citizens to scamming, phishing, and identity theft. We would recommend explicitly adding both those considerations in (Section 5 (b.i)). The latter should also extend to systems that are used to infer or predict private information or personal characteristics.\n\nFor (Section 5 (b.ii)), we also recommend considering at least some of the rights listed in the recently published Taxonomy of Human Rights Risks Connected to Generative AI published by the UN OHCHR, including but not limited to rights to employment, freedom from Physical and Psychological Harm, and rights to Culture, Art and Science. This recent editorial also delves deeper into the rights at play and outlines how different steps in the development process have bearing on specific rights.\n\n- 6. Do the minimum practices identified for safety-impacting and rights-impacting AI set an appropriate baseline that is applicable across all agencies and all such uses of AI? How can the minimum practices be improved, recognizing that agencies will need to apply context-specific risk mitigations in addition to what is listed?\n\nThe minimum practices identified constitute a strong baseline on which to develop responsible AI practices across a variety of potentially safety-impacting and rights-impacting AI applications. We strongly believe that these minimum practices should be encouraged for all AI systems even when not strictly required, to help develop robust best practices and mitigate risks that some systems may be mistakenly omitted.\n\nWe also recommend taking the following additions:\n\n- \u25cf For the sections on Completing an AI impact assessment and ensuring that the AI will advance equity, dignity, and fairness , we recommend explicitly mentioning due diligence background research that considers historical discrimination in the domain. AI applications tend to exacerbate existing problems.\n- \u25cf In addition for the existing requirements to conduct monitoring , mitigate emerging risks , and prioritize incremental adoption, agencies should, whenever possible, have a plan in place for rolling back AI systems as needed without disrupting ongoing processes,\n- \u25cf Finally, we stress the value of documentation in pursuit of the goals outlined in this memorandum. We recommend that data documentation and model cards be required for AI systems with safety-impacting and rights-impacting potential, along with system cards and governance cards as needed.\n- 7. What types of materials or resources would be most valuable to help agencies, as appropriate, incorporate the requirements and recommendations of this memorandum into relevant contracts?\n\nThe following categories of guidance will be particularly helpful to agencies:\n\n- \u25cf Specific questions to answer to inform the processes outlined in this document, especially with respect to pro-actively assessing the benefits of AI adoption and risks to rights, fairness and non-discrimination, and safety - and how to weigh those to make an adoption decision\n- \u25cf Standards on documentation content and quality, especially as regards information to be included in the AI Use Case inventory\n- \u25cf Providing a repository of positive examples of implementation of the requirements to help showcase successful operationalization\n\n\n\nWe note in particular that given the fast evolution of the technology and breadth of possible adoption, this guidance should be designed to evolve with time and feedback from federal agencies. We recommend in particular dedicating appropriate resources to maintaining and updating this information.\n\n- 8. What kind of information should be made public about agencies' use of AI in their annual use case inventory?\n\nWe recommend the following information for entries in the AI Use Case inventory:\n\n- \u25cf Brief description of the AI system, including its deployment context, expected inputs and outputs, and expected behaviors, and appropriate context of use\n- \u25cf Description of the metrics used to assess what value the AI system is bringing to the agency's use case\n- \u25cf Whether the system is deemed to be rights-impacting or safety-impacting, with justification\n- \u25cf Description of the metrics tracked to assess performance and potential rights and safety risks, especially with regard to non-discrimination\n- \u25cf The expected volume of interactions with the system, e.g. number of inputs processed per time period or number of users within the agency\n- \u25cf A list of the AI components used in the AI system, including:\n- \u25cb ML models, including base pre-trained model and fine-tuned models, with version when applicable\n- \u25cb ML training datasets, including pre-training and fine-tuning datasets, with version when applicable\n- \u25cf Model and data documentation for all components according to established standards\n- \u25cb For models, this should include appropriate contexts of use and evaluation in light of both those uses and different types of users and affected groups\n- \u25a0 Example model cards to consider: BLOOM, OBELICS, StarCoder\n- \u25cb For data, this should include handling of consent, licensing, analysis of possible source of biases, etc.\n- \u25a0 see also: https://hf.co/blog/yjernite/data-transparency\n- \u25cf Description of the governance processes relevant to the AI system, for example:\n- \u25cb Data governance in BigScience (BLOOM model)\n- \u25cb Governance card for BigCode (StarCoder model)\n\nAll items outlined above have a significant role to play in ensuring the proper governance of AI systems. By centralizing this information across uses of AI in federal agencies, the AI Use Case inventory will both be a useful tool for supporting this governance and an invaluable resource to support new research into best practices for AI adoption across society.", + "chunks": [ + { + "headings": [ + "Hugging Face, Inc. 20 Jay Street, Suite 620, Brooklyn, NY 11201" + ], + "markdown": "\n\nHugging Face, Inc. 20 Jay Street, Suite 620, Brooklyn, NY 11201\n\n## Hugging Face Response to the Office of Management and Budget Request for Comments on Agency Use of Artificial Intelligence\n\nHugging Face commends the Office of Management and Business on its extensive work framing the responsible use of AI by federal agencies. The following comments are informed by our experiences as an open platform for state-of-the-art (SotA) AI systems, working to make AI accessible and broadly available to researchers for responsible development.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## Summary\n\nThe draft memorandum outlines a strong framework for fostering responsible AI adoption by federal agencies. In a context of commonly inflated claims about the performance of AI systems and pervasive risks of exacerbating discrimination, we particularly appreciate the proposed requirements to not only conduct thorough impact assessments, but also to demonstrate that the adoption of AI does provide added value to the agencies.\n\nIn order to best support this mission, we recommend that the OMB and the proposed AI Governance bodies prioritize mechanisms that support the open sharing of skills, resources, and information; build internal capacity along as much of the AI development chain as possible, from pre-training data curation to model fine-tuning and deployment (including, as currently outlined in Section 4.b.iv, non-technical expertise); and develop further guidance on what constitutes a sufficient improvement to warrant AI adoption and how to choose an AI system to meet those goals.\n\nFor the purpose of sharing best practices, we note that the proposed AI Use Case inventory constitutes a particularly promising tool. We recommend that entries in the inventory include as much information as possible about the AI application and system, including data and model documentation according to established standards such as dataset sheets and model cards, the metrics used to assess whether AI adoption would have a positive impact for the agency's use case, the reasoning for determining whether the system was safety- or rights-impacting, and links to the specific versioned systems used, including any open-source or open-access components.\n\n\n\nWe also recommend that the OMB extend the scope of memorandum OMB M-16-21 on open source software to include AI systems. In particular, we believe that the testing datasets developed by agencies to meet the requirements laid out in the present draft memorandum have a significant role to play in advancing the state of the art in performance and risk evaluation of AI systems if they are shared broadly with researchers and stakeholders. By supporting agencies in sharing this data to all relevant stakeholders, including by providing guidelines for the use of synthetic data, data pseudonymization, and governance of public datasets, the OMB can accelerate the development of robust responsible AI practices that will benefit all of society.\n\nBeyond evaluation datasets, we encourage the OMB to direct agencies to build capacity to develop and share their own full AI stacks, including by leveraging and adapting existing public, open-source, and open Machine Learning models and datasets. This approach can lead to more energy-efficient AI applications, more transparency into the regulatory compliance of the AI development chain, and better visibility into the handling of e.g. privacy and intellectual property rights along the development process (see for example the StarCoder open code generation LLM). To that end, and in addition to the evaluation and test, we recommend that the AI strategy of federal agencies include to the extent possible the development and sharing of:\n\n- \u25cf pre-training datasets for base models (or foundation models) focused on their domain of activity\n- \u25cf the corresponding pre-trained (or foundation) models\n- \u25cf fine-tuning datasets and fine-tuned models tailored to the agencies' needs\n- \u25cf any software needed to process the inputs and outputs of AI systems\n\nWe provide further feedback on the draft memorandum below by answering the questions accompanying the Request for Comments. If a question is not highlighted, we do not have specific, actionable feedback.\n\n## Questions\n\n- 2. What types of coordination mechanisms, either in the public or private sector, would be particularly effective for agencies to model in their establishment of an AI Governance Body? What are the benefits or drawbacks to having agencies establishing a new body to perform AI governance versus updating the scope of an existing group (for example, agency bodies focused on privacy, IT, or data)?\n\nWhile AI Governance will require involvement from many existing groups in agencies, including but not limited to privacy, IT, data, and legal departments, approaching the new challenges posed by AI adoption holistically and efficiently will require coordination and skill sharing between different stakeholders, which is best managed by a new AI Governance Body. Such a body can also facilitate broader civil society involvement and standardization of documentation practices. As examples of the possible functions of such a body, we point to the role of the cross-areas working groups within the BigScience workshop on Large Language Models, and its proposed structure for a multi-stakeholder AI data governance organization for large scale ML. In particular, we stress the importance of the role of centralizing documentation and transparency guidelines and the adoption of interoperable data formats to allow for better skill sharing within and across agencies.\n\n- 3. How can OMB best advance responsible AI innovation?\n\nThe OMB has a unique role to play in championing responsible AI innovation beyond the federal government by helping create a body of positive examples and in-context operationalizations of responsible AI principles. Other agencies and companies may then learn from those examples\n\n\n\nand leverage the resources created in the process - especially if they are shared as open and open-source AI systems - to follow suit. These should include:\n\n- \u25cf Requiring documentation of key sections of the development pipeline demonstrating decisions that respect peoples' rights and abide by applicable laws. This included:\n- \u25cb Data Development\n- \u25cb Model Training\n- \u25cb Model Evaluation\n- \u25cb System Evaluation (when models are implemented as part of a larger system)\n- \u25cb Actions for continued Monitoring\n- \u25cf Building resources and knowledge about how to assess, evaluate, document, and audit AI systems\n- \u25cb In particular, focus on building capacity to audit specific parts of AI development, including training data and procedure\n- \u25cb Build a robust reporting system including labor complaints and whistleblower protections\n- \u25cf Leveraging shared experience between agencies to build best practices informed by the practicalities of deployment\n- \u25cf Highlight that AI should in many cases *NOT* be used, and/or may not be an improvement over current/alternative systems, in particular by providing transparency through the AI Use Case inventory about the process for weighing the risks and benefits of AI adoption.\n- 4. With adequate safeguards in place, how should agencies take advantage of generative AI to improve agency missions or business operations?\n\nGenerative AI, when used for the purposes it is best suited for, can be a useful tool to explore AI in a new application setting, increase productivity in settings such as software engineering, and support applications that are less reliant on accuracy or grounding in factual information especially when it is developed with sufficient transparency, consideration for the rights of data subject, and attribution of the model outputs. In particular, generative AI systems that are designed to handle a variety of tasks or work in so-called zero-shot settings can significantly accelerate the development of a 'proof-of-concept' AI system to explore whether AI can bring value to an agency in a particular setting; possibly to then be replaced by a more efficient discriminative system.\n\nGenerative AI also introduces unique concerns, including increased financial and environmental costs when it is used in settings where a discriminative AI alternative could suffice, an increased cybersecurity attack surface and range of vulnerabilities, and a dependence on web-scale and larger training datasets that exacerbates social issues, including biases and discrimination.\n\nThe trade-offs mentioned also depend on the modality considered. Code generation models such as the openly developed StarCoder (see the project's Governance Card for more detail), have already shown their value in accelerating the process of software development, and are less subject to (but not exempt from) exacerbating social biases along the development chain. Image generation models, on the other hand, are still very likely to reproduce and exaggerate stereotypes, which makes them a poor fit for generating illustrations of people in government communications.\n\n- 5. Are there use cases for presumed safety-impacting and rights-impacting AI (Section 5 (b)) that should be included, removed, or revised? If so, why?\n\n\n\nWe note that neither privacy nor handling of medical processes are currently included in the safety section. Mishandling of medically relevant decisions by AI systems can lead to immediate personal risks - especially given the prevalence of social biases in medical datasets, and leaks of private information can expose citizens to scamming, phishing, and identity theft. We would recommend explicitly adding both those considerations in (Section 5 (b.i)). The latter should also extend to systems that are used to infer or predict private information or personal characteristics.\n\nFor (Section 5 (b.ii)), we also recommend considering at least some of the rights listed in the recently published Taxonomy of Human Rights Risks Connected to Generative AI published by the UN OHCHR, including but not limited to rights to employment, freedom from Physical and Psychological Harm, and rights to Culture, Art and Science. This recent", + "char_slice": [ + 0, + 10734 + ] + }, + { + "headings": [ + "## Hugging Face Response to the Office of Management and Budget Request for Comments on Agency Use of Artificial Intelligence", + "## About Hugging Face", + "## Summary", + "## Questions" + ], + "markdown": "are still very likely to reproduce and exaggerate stereotypes, which makes them a poor fit for generating illustrations of people in government communications.\n\n- 5. Are there use cases for presumed safety-impacting and rights-impacting AI (Section 5 (b)) that should be included, removed, or revised? If so, why?\n\n\n\nWe note that neither privacy nor handling of medical processes are currently included in the safety section. Mishandling of medically relevant decisions by AI systems can lead to immediate personal risks - especially given the prevalence of social biases in medical datasets, and leaks of private information can expose citizens to scamming, phishing, and identity theft. We would recommend explicitly adding both those considerations in (Section 5 (b.i)). The latter should also extend to systems that are used to infer or predict private information or personal characteristics.\n\nFor (Section 5 (b.ii)), we also recommend considering at least some of the rights listed in the recently published Taxonomy of Human Rights Risks Connected to Generative AI published by the UN OHCHR, including but not limited to rights to employment, freedom from Physical and Psychological Harm, and rights to Culture, Art and Science. This recent editorial also delves deeper into the rights at play and outlines how different steps in the development process have bearing on specific rights.\n\n- 6. Do the minimum practices identified for safety-impacting and rights-impacting AI set an appropriate baseline that is applicable across all agencies and all such uses of AI? How can the minimum practices be improved, recognizing that agencies will need to apply context-specific risk mitigations in addition to what is listed?\n\nThe minimum practices identified constitute a strong baseline on which to develop responsible AI practices across a variety of potentially safety-impacting and rights-impacting AI applications. We strongly believe that these minimum practices should be encouraged for all AI systems even when not strictly required, to help develop robust best practices and mitigate risks that some systems may be mistakenly omitted.\n\nWe also recommend taking the following additions:\n\n- \u25cf For the sections on Completing an AI impact assessment and ensuring that the AI will advance equity, dignity, and fairness , we recommend explicitly mentioning due diligence background research that considers historical discrimination in the domain. AI applications tend to exacerbate existing problems.\n- \u25cf In addition for the existing requirements to conduct monitoring , mitigate emerging risks , and prioritize incremental adoption, agencies should, whenever possible, have a plan in place for rolling back AI systems as needed without disrupting ongoing processes,\n- \u25cf Finally, we stress the value of documentation in pursuit of the goals outlined in this memorandum. We recommend that data documentation and model cards be required for AI systems with safety-impacting and rights-impacting potential, along with system cards and governance cards as needed.\n- 7. What types of materials or resources would be most valuable to help agencies, as appropriate, incorporate the requirements and recommendations of this memorandum into relevant contracts?\n\nThe following categories of guidance will be particularly helpful to agencies:\n\n- \u25cf Specific questions to answer to inform the processes outlined in this document, especially with respect to pro-actively assessing the benefits of AI adoption and risks to rights, fairness and non-discrimination, and safety - and how to weigh those to make an adoption decision\n- \u25cf Standards on documentation content and quality, especially as regards information to be included in the AI Use Case inventory\n- \u25cf Providing a repository of positive examples of implementation of the requirements to help showcase successful operationalization\n\n\n\nWe note in particular that given the fast evolution of the technology and breadth of possible adoption, this guidance should be designed to evolve with time and feedback from federal agencies. We recommend in particular dedicating appropriate resources to maintaining and updating this information.\n\n- 8. What kind of information should be made public about agencies' use of AI in their annual use case inventory?\n\nWe recommend the following information for entries in the AI Use Case inventory:\n\n- \u25cf Brief description of the AI system, including its deployment context, expected inputs and outputs, and expected behaviors, and appropriate context of use\n- \u25cf Description of the metrics used to assess what value the AI system is bringing to the agency's use case\n- \u25cf Whether the system is deemed to be rights-impacting or safety-impacting, with justification\n- \u25cf Description of the metrics tracked to assess performance and potential rights and safety risks, especially with regard to non-discrimination\n- \u25cf The expected volume of interactions with the system, e.g. number of inputs processed per time period or number of users within the agency\n- \u25cf A list of the AI components used in the AI system, including:\n- \u25cb ML models, including base pre-trained model and fine-tuned models, with version when applicable\n- \u25cb ML training datasets, including pre-training and fine-tuning datasets, with version when applicable\n- \u25cf Model and data documentation for all components according to established standards\n- \u25cb For models, this should include appropriate contexts of use and evaluation in light of both those uses and different types of users and affected groups\n- \u25a0 Example model cards to consider: BLOOM, OBELICS, StarCoder\n- \u25cb For data, this should include handling of consent, licensing, analysis of possible source of biases, etc.\n- \u25a0 see also: https://hf.co/blog/yjernite/data-transparency\n- \u25cf Description of the governance processes relevant to the AI system, for example:\n- \u25cb Data governance in BigScience (BLOOM model)\n- \u25cb Governance card for BigCode (StarCoder model)\n\nAll items outlined above have a significant role to play in ensuring the proper governance of AI systems. By centralizing this information across uses of AI in federal agencies, the AI Use Case inventory will both be a useful tool for supporting this governance and an invaluable resource to support new research into best practices for AI adoption across society.", + "char_slice": [ + 9473, + 15824 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2023_UK%20Parliament%20RFI%20LLMs.pdf", + "md_content": "## Hugging Face Comments on the UK Parliament Call for Evidence\n\nHugging Face commends the UK Parliament Communications and Digital Committee on its ongoing work to examine the opportunities and risks of large language models (LLMs). The following comments are informed by our experiences as an open platform for state-of-the-art (SotA) AI systems, working to make AI accessible and broadly available to researchers for responsible development. Comments are organized by questions listed in Call for Evidence. If a section is not highlighted, we do not have specific, actionable feedback.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. Hugging Face is based in the U.S., with an office in London and a global developer community.\n\n## Capabilities and trends\n\n- 2. What are the greatest opportunities and risks over the next three years? Beyond the opportunities of LLMs shared in corporate communications, opportunities with LLMs that are not currently common in practice include:\n- \u25cf Providing the opportunity for creators of data to consent to, and/or be compensated for, their work\n- \u25cf Opening development discussions to people from different backgrounds, a process called 'participatory design'. For example, non-verbal individuals can share how they would benefit from LLMs that generate multiple possible utterances for them to select from.\n- \u25cf Helping with English language writing, personalized to what the needs are of the user. For example, children learning English may benefit from working with an LLM to create English stories.\n- \u25cf Creating on-the-fly games and entertainment, for example, 'Balderdash' for a single player.\n\nThe risk landscape, and corresponding harms, is continually evolving and regulatory action should address both present-day harms and foreseeable risks, especially those that affect marginalized communities. For LLMs, this includes representational harms and risks such as discriminatory stereotypes, which can prove catastrophic in high-stakes applications. The type of risk and prioritization of each risk is contentious. Research to taxonomize existing risks include foundational work on dangers, examining harms to people, and scoping sociotechnical harms.\n\nMeasurable social risks from LLMs include but are not limited to: bias, stereotypes, and representational risks; cultural values and sensitive content; disparate performance; privacy and data protection; financial costs; environmental costs; and data and content moderation labor costs. Risks to society include but are not limited to: trustworthiness and autonomy; inequality, marginalization, and violence; concentration of authority; labor and creativity; and ecosystem and environment.\n\n## a) How should we think about risk in this context?\n\nRisks in this context specifically refer to risks of harm , where harm includes problematic outcomes for different populations. Risks can arise along the system development and deployment process, meaning that all components and processes - from training datasets to intended application - can embed a certain level of risk. Efforts to evaluate social risks and conduct comprehensive risk assessments are crucial. The U.S. National Institute of Standards and Technology's AI Risk Management Framework is one of the leading tools for managing risk, and will soon profile generative AI. Auditing frameworks also give insight to risk management.\n\nAccounting for harms and mitigating risks requires the ability to evaluate them and their severity. Evaluations for large language models, especially for complex social impacts such as biases and environmental costs, are not standardized and have large gaps across risk areas. For example, evaluating biases in large language models often skews to quantification and more evaluations exist for certain protected classes, such as gender, than others, such as age, religion or disability. Specific types of language-based systems, such as code generation, benefit from assessing safety in context. Better understanding risks requires more resourcing and central fora for testing, which will require better researcher infrastructure, system access, and transparency and deployment disclosure as risk is best assessed in context.\n\n## Domestic regulation\n\n3. How adequately does the AI White Paper (alongside other Government policy) deal with large language models? Is a tailored regulatory approach needed?\n\nWe support the proposed pro-innovation approach's methods such as transparency measures and feedback mechanisms. We provided comments on the White Paper in the call for evidence.\n\n## a) What are the implications of open-source models proliferating?\n\n- What constitutes an open-source model does not currently have unanimous agreement. We find it useful to take a step back from specifically open-' source' to open models more broadly. Large language models are composed of many components throughout their training and development process that should all be considered in a release. Available options for releasing foundation models vary across a spectrum from fully closed to fully open, each option with its own challenges and tradeoffs. Openness and increased access creates many opportunities for broader community research, including empowering\n\nresearchers to create safeguards by being able to test on an accessible model. Ethical openness, as exemplified by Hugging Face's approach, requires implementing many types of safeguards. Increased access to artifacts such as models and datasets enables researchers and external parties to better understand systems, conduct audits, mitigate risks, and find high value applications.\n\nSpecific to models, risks and harms arise from a model regardless of how accessible it may be. A fully closed model risks not having external expertise to guide alignment or risk mitigation. A hosted model can still be used to generate harmful disinformation but can gather user feedback. A fully open model can foster broader research but risks misuse. The complexity of risk tradeoffs along release methods is why safeguard research parallel to model development is necessary.\n\n- 4. Do the UK's regulators have sufficient expertise and resources to respond to large language models?[5] If not, what should be done to address this?\n\nExpertises throughout sectors must complement each other ; each sector and organization within provides insights that may not be represented in another. UK regulators can leverage the many available expertises by prioritizing regulatory mechanisms that use transparency to enable stakeholders to meaningfully engage with AI systems; investing in standardization mechanisms designed to work at the component level and are easily implementable; protecting open source and open science when naturally aligned with the requirements of more accountable and democratic technology; and consulting academic stakeholders and open source developers when designing exemption regimes. Government-provided resources for researchers to access infrastructure such as computing power can increase expertise and research on risks. This can take lessons from the U.S. National AI Research Resource.\n\n## 5. What are the non-regulatory and regulatory options to address risks and capitalise on opportunities?\n\nThere is no one panacea against all risks from AI; instead, safeguards should span regulatory, policy, legal, and technical tools and levers. Different pieces of legislation can address different aspects of AI systems, such as privacy legislation and IP law's impact on training data being separate but often complementary. Options can address many risks at once. Transparency requirements can be one vector of influence. Requiring model documentation, such as model cards, can address transparency and research reproducibility concerns.\n\nConcretely, non-regulatory options we recommend are: transparency guidance, researcher access and protections, public infrastructure for AI. Regulatory options include proportional requirements for systems by use (sector, risk, popularity) and use case.\n\nb) At what stage of the AI life cycle will interventions be most effective? As stated in question 2.a. of this response, since risk may arise along the life cycle, research and risk management processes should be applied across many system components. More research and evaluation tools are needed for examining training data, such as being able to examine attributes of large datasets. As we shared in our response to the U.S. Department of Commerce's National Telecommunications and Information Administration's Request for Comments, accountability mechanisms such as audits should focus on all stages of the development process by requiring transparency via good documentation and external access processes and inviting broad contribution across affected stakeholders\n\n## Conclusion\n\nThe LLM regulatory landscape requires many expertises. We thank the UK Parliament for the opportunity to provide our insights and look forward to supporting ongoing and future efforts.\n\nRespectfully,\n\nIrene Solaiman Policy Director Hugging Face\n\nMargaret Mitchell Chief Ethics Scientist Hugging Face\n\nSubmitted: 1 September 2023", + "chunks": [ + { + "headings": [ + "Hugging Face Comments on the UK Parliament Call for Evidence" + ], + "markdown": "## Hugging Face Comments on the UK Parliament Call for Evidence\n\nHugging Face commends the UK Parliament Communications and Digital Committee on its ongoing work to examine the opportunities and risks of large language models (LLMs). The following comments are informed by our experiences as an open platform for state-of-the-art (SotA) AI systems, working to make AI accessible and broadly available to researchers for responsible development. Comments are organized by questions listed in Call for Evidence. If a section is not highlighted, we do not have specific, actionable feedback.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. Hugging Face is based in the U.S., with an office in London and a global developer community.\n\n## Capabilities and trends\n\n- 2. What are the greatest opportunities and risks over the next three years? Beyond the opportunities of LLMs shared in corporate communications, opportunities with LLMs that are not currently common in practice include:\n- \u25cf Providing the opportunity for creators of data to consent to, and/or be compensated for, their work\n- \u25cf Opening development discussions to people from different backgrounds, a process called 'participatory design'. For example, non-verbal individuals can share how they would benefit from LLMs that generate multiple possible utterances for them to select from.\n- \u25cf Helping with English language writing, personalized to what the needs are of the user. For example, children learning English may benefit from working with an LLM to create English stories.\n- \u25cf Creating on-the-fly games and entertainment, for example, 'Balderdash' for a single player.\n\nThe risk landscape, and corresponding harms, is continually evolving and regulatory action should address both present-day harms and foreseeable risks, especially those that affect marginalized communities. For LLMs, this includes representational harms and risks such as discriminatory stereotypes, which can prove catastrophic in high-stakes applications. The type of risk and prioritization of each risk is contentious. Research to taxonomize existing risks include foundational work on dangers, examining harms to people, and scoping sociotechnical harms.\n\nMeasurable social risks from LLMs include but are not limited to: bias, stereotypes, and representational risks; cultural values and sensitive content; disparate performance; privacy and data protection; financial costs; environmental costs; and data and content moderation labor costs. Risks to society include but are not limited to: trustworthiness and autonomy; inequality, marginalization, and violence; concentration of authority; labor and creativity; and ecosystem and environment.\n\n## a) How should we think about risk in this context?\n\nRisks in this context specifically refer to risks of harm , where harm includes problematic outcomes for different populations. Risks can arise along the system development and deployment process, meaning that all components and processes - from training datasets to intended application - can embed a certain level of risk. Efforts to evaluate social risks and conduct comprehensive risk assessments are crucial. The U.S. National Institute of Standards and Technology's AI Risk Management Framework is one of the leading tools for managing risk, and will soon profile generative AI. Auditing frameworks also give insight to risk management.\n\nAccounting for harms and mitigating risks requires the ability to evaluate them and their severity. Evaluations for large language models, especially for complex social impacts such as biases and environmental costs, are not standardized and have large gaps across risk areas. For example, evaluating biases in large language models often skews to quantification and more evaluations exist for certain protected classes, such as gender, than others, such as age, religion or disability. Specific types of language-based systems, such as code generation, benefit from assessing safety in context. Better understanding risks requires more resourcing and central fora for testing, which will require better researcher infrastructure, system access, and transparency and deployment disclosure as risk is best assessed in context.\n\n## Domestic regulation\n\n3. How adequately does the AI White Paper (alongside other Government policy) deal with large language models? Is a tailored regulatory approach needed?\n\nWe support the proposed pro-innovation approach's methods such as transparency measures and feedback mechanisms. We provided comments on the White Paper in the call for evidence.\n\n## a) What are the implications of open-source models proliferating?\n\n- What constitutes an open-source model does not currently have unanimous agreement. We find it useful to take a step back from specifically open-' source' to open models more broadly. Large language models are composed of many components throughout their training and development process that should all be considered in a release. Available options for releasing foundation models vary across a spectrum from fully closed to fully open, each option with its own challenges and tradeoffs. Openness and increased access creates many opportunities for broader community research, including empowering\n\nresearchers to create safeguards by being able to test on an accessible model. Ethical openness, as exemplified by Hugging Face's approach, requires implementing many types of safeguards. Increased access to artifacts such as models and datasets enables researchers and external parties to better understand systems, conduct audits, mitigate risks, and find high value applications.\n\nSpecific to models, risks and harms arise from a model regardless of how accessible it may be. A fully closed model risks not having external expertise to guide alignment or risk mitigation. A hosted model can still be used to generate harmful disinformation but can gather user feedback. A fully open model can foster broader research but risks misuse. The complexity of risk tradeoffs along release methods is why safeguard research parallel to model development is necessary.\n\n- 4. Do the UK's regulators have sufficient expertise and resources to respond to large language models?[5] If not, what should be done to address this?\n\nExpertises throughout sectors must complement each other ; each sector and organization within provides insights that may not be represented in another. UK regulators can leverage the many available expertises by prioritizing regulatory mechanisms that use transparency to enable stakeholders to meaningfully engage with AI systems; investing in standardization mechanisms designed to work at the component level and are easily implementable; protecting open source and open science when naturally aligned with the requirements of more accountable and democratic technology; and consulting academic stakeholders and open source developers when designing exemption regimes. Government-provided resources for researchers to access infrastructure such as computing power can increase expertise and research on risks. This can take lessons from the U.S. National AI Research Resource.\n\n## 5. What are the non-regulatory and regulatory options to address risks and capitalise on opportunities?\n\nThere is no one panacea against all risks from AI; instead, safeguards should span regulatory, policy, legal, and technical tools and levers. Different pieces of legislation can address different aspects of AI systems, such as privacy legislation and IP law's impact on training data being separate but often complementary. Options can address many risks at once. Transparency requirements can be one vector of influence. Requiring model documentation, such as model cards, can address transparency and research reproducibility concerns.\n\nConcretely, non-regulatory options we recommend are: transparency guidance, researcher access and protections, public infrastructure for AI. Regulatory options include proportional requirements for systems by use (sector, risk, popularity) and use case.\n\nb) At what stage of the AI life cycle will interventions be most effective? As stated in question 2.a. of this response, since risk may arise along the life cycle, research and risk management processes should be applied across many system components. More research and evaluation tools are needed for examining training data, such as being able to examine attributes of large datasets. As we shared in our response to the U.S. Department of Commerce's National Telecommunications and Information Administration's Request for Comments, accountability mechanisms such as audits should focus on all stages of the development process by requiring transparency via good documentation and external access processes and inviting broad contribution across affected stakeholders\n\n## Conclusion\n\nThe LLM regulatory landscape requires many expertises. We thank the UK Parliament for the opportunity to provide our insights and look forward to supporting ongoing and future efforts.\n\nRespectfully,\n\nIrene Solaiman Policy Director Hugging Face\n\nMargaret Mitchell Chief Ethics Scientist Hugging Face\n\nSubmitted: 1 September 2023", + "start_char_id": 0, + "end_char_id": 9645 + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2023_UK_RFI_AI_Regulatory_Innovation_White_Paper.pdf", + "md_content": "\n\n## Hugging Face Comments on the UK AI Regulation White Paper\n\nHugging Face congratulates the UK government on its pro-innovation approach to AI regulation that recognizes the many benefits and opportunities of AI while controlling for risks. The following comments are informed by our experiences as an open platform for state-of-the-art (SotA) AI systems, working to make AI accessible and broadly available to researchers for responsible development. Comments are organized by questions listed in Annex C of the White Paper. If a section is not highlighted, we do not have specific, actionable feedback.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. Hugging Face is based in the U.S. and France, with an office in London and a global developer community.\n\n## The Revised Cross-Sectoral AI Principles\n\n- 1. Do you agree that requiring organisations to make it clear when they are using AI would adequately ensure transparency?\n\nMaking AI use clear requires guidelines and specificity on how to document and communicate aspects of an AI system and its use. Transparency and disclosure should address many AI system components throughout its lifecycle in addition to the contexts and applications into which the system is deployed.\n\nThis means not only robust documentation for models and datasets, but also for processes throughout systems development and testing. Guidance on how to document systems, such as through model cards, which we deploy widely at Hugging Face, in addition to tooling, can help lower the barrier for transparency. Approaches to transparency will necessarily differ by system. For large language models, their complex and multi-purpose capabilities in addition to large architecture make them difficult to make fully transparent.\n\n- 2. What other transparency measures would be appropriate, if any?\n\nMechanisms for transparency require many skill sets, not all of which are likely present in a developer organization. There is heavy overlap with accountability mechanisms such as audits and certifications. Both require access to AI systems and their components, meaning detailed model and dataset documentation at the very least. Openness can help improve transparency, and organizations dedicated to openness show exceptional transparency compliance. Pairing transparency requirements, such as documentation, with accountability, such as audits, can bolster trustworthiness by ensuring third-party validation .\n\n- 3. Do you agree that current routes to contestability or redress for AI-related harms are adequate? 4. How could routes to contestability or redress for AI-related harms be improved, if at all? 5. Do you agree that, when implemented effectively, the revised cross-sectoral principles will cover the risks posed by AI technologies?\n\nBetter evaluations and government-provided research resources are sorely needed for novel AI systems. In order to conduct adequate pre-deployment risk assessments, those evaluating a system need good tools and metrics. Foundation model evaluations, especially outside text and language modalities, face many pitfalls such as lack of standardization, lack of access to systems and necessary computing infrastructure, and lack of existing tools.\n\nApproaches by system, such as language models, can provide helpful insights across models and capabilities. Communicating evaluation findings should also be consumable to many audiences. Furthermore, evaluating inherently qualitative and social aspects of a system such as harmful biases, disinformation, and unsafe or violent content is difficult. More investment in evaluation is needed for models, datasets and other system components.\n\nA central resource for researchers to access infrastructure such as computing power can increase research on evaluations and safeguards. The many expertises needed to evaluate and mitigate risks may also require computer science and similar technical training in addition to low to no-code tooling. Lessons from the U.S. National AI Research Resource can strengthen global approaches to safe innovation.\n\n- 6. What, if anything, is missing from the revised principles?\n\nThe given principles can be overarching and inclusive to account for social impacts such as privacy and data protection and environmental impacts. Existing AI principles, such as the OECD's AI Principles, EU's requirements for Trustworthy AI, and the U.S. AI Bill of Rights should overlap with each other to bolster a global and allied approach to safe AI. These principles should be dynamic and updatable as new information about AI risks arise.\n\n## Monitoring and evaluation of the framework\n\n- 15. Do you agree with our overall approach to monitoring and evaluation? 16. What is the best way to measure the impact of our framework?\n\nMonitoring both the framework and the framework's effectiveness addressing AI risks should also invest in capabilities evaluations, risk taxonomies, and expertise across systems and high risk areas. We commend the given approach and emphasize interoperability with global frameworks to avoid patchwork legislation, while recognizing some tensions may arise in specific approaches such as legal definitions. Horizon scanning should be agile ; while some risks may be long-standing, such as disinformation from large language models, others may quickly arise, such as AI generations' impact on academic integrity.\n\nImpact should be measured iteratively and in conjunction with our understanding of AI capability and innovation and updating emergent risks .\n\n- 17. Do you agree that our approach strikes the right balance between supporting AI innovation; addressing known, prioritised risks; and future-proofing the AI regulation framework? 18. Do you agree that regulators are best placed to apply the principles and government is best placed to provide oversight and deliver central functions?\n\nWe agree with and applaud highlighting feedback loops, which should be inclusive of all stakeholders, such as regulators, industry, civil society, and academia, and global allies. Regulators should help guide innovation in a beneficial direction. The appropriate regulatory body and agency giving guidance will differ based on the type of system, the system's sectoral application and use case, and the urgency of attention/level of risk.\n\n## Tools for trustworthy AI\n\n21. Which non-regulatory tools for trustworthy AI would most help organisations to embed the AI regulation principles into existing business processes?\n\nThe most effective approaches to trustworthiness are injected throughout system development and address the system's context and application as it is deployed and affects users. For increasingly general-purpose systems, mechanisms can be applied by system and function; novel legal approaches such as Responsible AI Licenses (RAIL) can encourage innovation while preventing harmful uses.\n\n## Foundation models and the regulatory framework\n\n- F1. What specific challenges will foundation models such as large language models (LLMs) or open-source models pose for regulators trying to determine legal responsibility for AI outcomes?\n\nThe continually evolving landscape of foundation model development and deployment makes determining outcomes and ongoing processes. The options for releasing foundational models vary across a spectrum from fully closed to fully open, each option with its own challenges and tradeoffs. Open-source provides many opportunities for broader community research, including empowering researchers to create safeguards by being able to test on an accessible model. Ethical openness requires implementing many types of safeguards. Legal responsibilities will depend on the type of impact and legal precedent. Existing frameworks and risk considerations work should be examined and expanded.\n\nF2. Do you agree that measuring compute provides a potential tool that could be considered as part of the governance of foundation models? Compute needs differ vastly by researcher and type of research. Compute is more tangible than more complex infrastructural needs, such as clean and safe training data. Necessary research and development infrastructure, and gaps, extend beyond compute and governance mechanisms should encompass many system components. Furthermore, the development of more compute-efficient models as seen with LLaMA and Alpaca, point to innovation in compute-constrained environments. We also note that most of the risks inherent to AI are tied to the scale of impact on increasingly global populations; broader and more direct reach of a system requires stronger governance requirements than absolute compute needs.\n\n- F3. Are there other approaches to governing foundation models that would be more effective?\n\nBetter fora for developing community norms must be inclusive of the many developers, providers, researchers across academic disciplines, and stakeholders in AI. Community norms vary from appropriate release methods to shared safety protocols to best evaluations per modality. Fostering innovation requires supporting small and medium businesses and researchers' needs for access and technical infrastructure. Fostering safe innovation requires investing in technical, policy, and legal safeguards that preempt and protect from emerging risks.\n\n## Conclusion\n\nWe thank the UK government for the opportunity to provide feedback and acknowledge each of these proposed questions require continual input as the pace of AI continues to accelerate. We look forward to further discussions and to supporting a regulatory approach that will foster safe innovation.\n\n## Respectfully,\n\n\n\n\n\nIrene Solaiman\n\nYacine Jernite ML and Society Lead\n\nPolicy Director\n\nHugging Face\n\nHugging Face", + "chunks": [ + { + "headings": [ + "Hugging Face Comments on the UK AI Regulation White Paper" + ], + "markdown": "\n\n## Hugging Face Comments on the UK AI Regulation White Paper\n\nHugging Face congratulates the UK government on its pro-innovation approach to AI regulation that recognizes the many benefits and opportunities of AI while controlling for risks. The following comments are informed by our experiences as an open platform for state-of-the-art (SotA) AI systems, working to make AI accessible and broadly available to researchers for responsible development. Comments are organized by questions listed in Annex C of the White Paper. If a section is not highlighted, we do not have specific, actionable feedback.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. Hugging Face is based in the U.S. and France, with an office in London and a global developer community.\n\n## The Revised Cross-Sectoral AI Principles\n\n- 1. Do you agree that requiring organisations to make it clear when they are using AI would adequately ensure transparency?\n\nMaking AI use clear requires guidelines and specificity on how to document and communicate aspects of an AI system and its use. Transparency and disclosure should address many AI system components throughout its lifecycle in addition to the contexts and applications into which the system is deployed.\n\nThis means not only robust documentation for models and datasets, but also for processes throughout systems development and testing. Guidance on how to document systems, such as through model cards, which we deploy widely at Hugging Face, in addition to tooling, can help lower the barrier for transparency. Approaches to transparency will necessarily differ by system. For large language models, their complex and multi-purpose capabilities in addition to large architecture make them difficult to make fully transparent.\n\n- 2. What other transparency measures would be appropriate, if any?\n\nMechanisms for transparency require many skill sets, not all of which are likely present in a developer organization. There is heavy overlap with accountability mechanisms such as audits and certifications. Both require access to AI systems and their components, meaning detailed model and dataset documentation at the very least. Openness can help improve transparency, and organizations dedicated to openness show exceptional transparency compliance. Pairing transparency requirements, such as documentation, with accountability, such as audits, can bolster trustworthiness by ensuring third-party validation .\n\n- 3. Do you agree that current routes to contestability or redress for AI-related harms are adequate? 4. How could routes to contestability or redress for AI-related harms be improved, if at all? 5. Do you agree that, when implemented effectively, the revised cross-sectoral principles will cover the risks posed by AI technologies?\n\nBetter evaluations and government-provided research resources are sorely needed for novel AI systems. In order to conduct adequate pre-deployment risk assessments, those evaluating a system need good tools and metrics. Foundation model evaluations, especially outside text and language modalities, face many pitfalls such as lack of standardization, lack of access to systems and necessary computing infrastructure, and lack of existing tools.\n\nApproaches by system, such as language models, can provide helpful insights across models and capabilities. Communicating evaluation findings should also be consumable to many audiences. Furthermore, evaluating inherently qualitative and social aspects of a system such as harmful biases, disinformation, and unsafe or violent content is difficult. More investment in evaluation is needed for models, datasets and other system components.\n\nA central resource for researchers to access infrastructure such as computing power can increase research on evaluations and safeguards. The many expertises needed to evaluate and mitigate risks may also require computer science and similar technical training in addition to low to no-code tooling. Lessons from the U.S. National AI Research Resource can strengthen global approaches to safe innovation.\n\n- 6. What, if anything, is missing from the revised principles?\n\nThe given principles can be overarching and inclusive to account for social impacts such as privacy and data protection and environmental impacts. Existing AI principles, such as the OECD's AI Principles, EU's requirements for Trustworthy AI, and the U.S. AI Bill of Rights should overlap with each other to bolster a global and allied approach to safe AI. These principles should be dynamic and updatable as new information about AI risks arise.\n\n## Monitoring and evaluation of the framework\n\n- 15. Do you agree with our overall approach to monitoring and evaluation? 16. What is the best way to measure the impact of our framework?\n\nMonitoring both the framework and the framework's effectiveness addressing AI risks should also invest in capabilities evaluations, risk taxonomies, and expertise across systems and high risk areas. We commend the given approach and emphasize interoperability with global frameworks to avoid patchwork legislation, while recognizing some tensions may arise in specific approaches such as legal definitions. Horizon scanning should be agile ; while some risks may be long-standing, such as disinformation from large language models, others may quickly arise, such as AI generations' impact on academic integrity.\n\nImpact should be measured iteratively and in conjunction with our understanding of AI capability and innovation and updating emergent risks .\n\n- 17. Do you agree that our approach strikes the right balance between supporting AI innovation; addressing known, prioritised risks; and future-proofing the AI regulation framework? 18. Do you agree that regulators are best placed to apply the principles and government is best placed to provide oversight and deliver central functions?\n\nWe agree with and applaud highlighting feedback loops, which should be inclusive of all stakeholders, such as regulators, industry, civil society, and academia, and global allies. Regulators should help guide innovation in a beneficial direction. The appropriate regulatory body and agency giving guidance will differ based on the type of system, the system's sectoral application and use case, and the urgency of attention/level of risk.\n\n## Tools for trustworthy AI\n\n21. Which non-regulatory tools for trustworthy AI would most help organisations to embed the AI regulation principles into existing business processes?\n\nThe most effective approaches to trustworthiness are injected throughout system development and address the system's context and application as it is deployed and affects users. For increasingly general-purpose systems, mechanisms can be applied by system and function; novel legal approaches such as Responsible AI Licenses (RAIL) can encourage innovation while preventing harmful uses.\n\n## Foundation models and the regulatory framework\n\n- F1. What specific challenges will foundation models such as large language models (LLMs) or open-source models pose for regulators trying to determine legal responsibility for AI outcomes?\n\nThe continually evolving landscape of foundation model development and deployment makes determining outcomes and ongoing processes. The options for releasing foundational models vary across a spectrum from fully closed to fully open, each option with its own challenges and tradeoffs. Open-source provides many opportunities for broader community research, including empowering researchers to create safeguards by being able to test on an accessible model. Ethical openness requires implementing many types of safeguards. Legal responsibilities will depend on the type of impact and legal precedent. Existing frameworks and risk considerations work should be examined and expanded.\n\nF2. Do you agree that measuring compute provides a potential tool that could be considered as part of the governance of foundation models? Compute needs differ vastly by researcher and type of research. Compute is more tangible than more complex infrastructural needs, such as clean and safe training data. Necessary research and development infrastructure, and gaps, extend beyond compute and governance mechanisms should encompass many system components. Furthermore, the development of more compute-efficient models as seen with LLaMA and Alpaca, point to innovation in compute-constrained environments. We also note that most of the risks inherent to AI are tied to the scale of impact on increasingly global populations; broader and more direct reach of a system requires stronger governance requirements than absolute compute needs.\n\n- F3. Are there other approaches to governing foundation models that would be more effective?\n\nBetter fora for developing community norms must be inclusive of the many developers, providers, researchers across academic disciplines, and stakeholders in AI. Community norms vary from appropriate release methods to shared safety protocols to best evaluations per modality. Fostering innovation requires supporting small and medium businesses and researchers' needs for access and technical infrastructure. Fostering safe innovation requires investing in technical, policy, and legal safeguards that preempt and protect from emerging risks.\n\n## Conclusion\n\nWe thank the UK government for the opportunity to provide feedback and acknowledge each of these proposed questions require continual input as the pace of AI continues to accelerate. We look forward to further discussions and to supporting a regulatory approach that will foster safe innovation.\n\n## Respectfully,\n\n\n\n\n\nIrene Solaiman\n\nYacine Jernite ML and Society Lead\n\nPolicy Director\n\nHugging Face\n\nHugging Face", + "start_char_id": 0, + "end_char_id": 10232 + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2024_AISI_Dual_Use_Foundational_Models_Response.pdf", + "md_content": "\n\n## Hugging Face Comments on NIST AI 800-1: Managing Misuse Risk for Dual-Use Foundational Models\n\nHugging Face commends the US AI Safety Institute (AISI) on the AI 800-1 document: Managing Misuse Risk for Dual-Use Foundation Models. This comprehensive framework identifies key objectives and practices for managing risks associated with foundation models. We offer recommendations to strengthen this document based on our experiences in democratizing good AI and characterizing risks of systems as an open platform for state-of-the-art (SotA) AI systems. Our comments are organized by objectives and practices as outlined in the document. Where we do not have specific, actionable feedback on a section or practice, we have not highlighted it.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## Executive Summary\n\nHugging Face commends the US AI Safety Institute (AISI) for its comprehensive AI 800-1 document to address managing misuse risks for dual-use foundation models. Based on our experience as a leading open AI platform, we offer the following key recommendations to enhance the framework's effectiveness:\n\n- 1. Joint Management Across the AI Supply Chain : Risk management should be a shared responsibility among all stakeholders, including data providers, infrastructure providers, and end-users, rather than solely on individual model developers. Encourage open, collaborative approaches where diverse stakeholders contribute to defining risk thresholds and management strategies.\n- 2. Enhance Transparency, Accountability, and Ongoing Risk Identification : Implement regular transparency reporting on risk management practices, and establish mechanisms for meaningful accountability, including sharing information with\n\n\n\nindependent entities. Improve scientific and regulatory visibility on risk profiles by providing clear guidelines for reporting and categorizing misuse risks, and support external safety research through vulnerability disclosure policies and safe harbor provisions.\n\n- 3. Tailor Risk Interventions and Balance Security with Accessibility : Different risks require context-specific, flexible safeguards. Ensure that interventions are tailored to the nature of the risk, whether it be non-consensual intimate imagery or CBRN threats. When considering model access restrictions, balance the need for security with the benefits of openness to foster innovation and research.\n- 4. Recognize and Support Open Foundation Models : Explicitly acknowledge the unique characteristics and benefits of open foundation models, including their role in enabling external scrutiny, mitigating monoculture, and fostering innovation. Develop guidelines that support responsible open-source AI development while managing associated risks.\n\n## Objectives and Practices to Manage Misuse Risks\n\n## Anticipating and Measuring Risk\n\n(Objectives 1 and 4)\n\n## Holistic Approach to Risk Assessment\n\nTo effectively manage and safely deploy AI models, it is essential to adopt a holistic approach to risk assessment that encompasses both technical and societal factors. While this document primarily focuses on safety risks, it is important to recognize that mechanisms designed to address these risks can also be applied to broader societal concerns. Failing to integrate these considerations could lead to missed opportunities for creating more comprehensive and effective risk management strategies. For instance, the focus on preventing \"model theft\" might overlook the value of broad, inclusive participation in identifying model biases and other potential harms. A framework for measuring risks that comprehensively addresses foundational models' impacts on people and society would ensure that safety measures do not inadvertently neglect significant societal implications - or make addressing them through other means more difficult.\n\n## Collaborative Risk Definition\n\nThe current approach to risk definition, which relies on individual developers to define and assess risks, introduces several challenges. It creates disparities between corporate and collaborative developers, complicates comparison between models from different developers,\n\n\n\nand deviates from established scientific processes. Instead, we advocate for a more standardized, collaborative approach. This approach could involve a centralized system where diverse stakeholders contribute to and maintain a comprehensive list of potential malicious uses or unintended consequences of foundation models, akin to the Common Use Enumeration (CUE) component we have proposed for structured harm reporting. This collaborative approach offers several benefits:\n\n- \u25cf Broader Expertise: It leverages a wider range of expertise, including domain-specific knowledge that individual companies may lack.\n- \u25cf Resource Equity: It reduces disparities in resources and incentives for risk evaluation between corporate and collaborative developers.\n- \u25cf Consistent Assessment: It enables more consistent risk assessment across different models and developers, facilitating meaningful comparisons and industry-wide progress on safety.\n- \u25cf Scientific Alignment: It aligns more closely with established scientific processes for risk assessment.\n- \u25cf Comprehensive Risk Identification: It reduces duplicated efforts across organizations and is likely to identify a more comprehensive and nuanced set of potential risks than any single organization could alone.\n\nBy moving towards a more inclusive, standardized approach, the industry can establish a more robust, scientifically rigorous, and comprehensive system for anticipating and managing potential misuse risks in foundation models. Such an approach corresponds to Practice 1.1 by improving the standardization and inclusivity of risk assessments.\n\n## Transparency in Capability Estimation\n\nTransparency is crucial in estimating model capabilities before deployment (Practice 4.1). We recommend publicly documenting the methods used for capability estimation, including any limitations or uncertainties. Developing and using open benchmarks, leaderboards, and evaluation tools will enable more standardized and comparable capability assessments across the industry. Periodic measurement of capabilities throughout the development process is commendable, but this should be viewed as an opportunity to focus more on upstream development choices, such as dataset selection and model scaling, rather than merely increasing capabilities. For open models, leveraging their openness for collaborative risk identification is key.\n\n## More Effective Red Teaming\n\nRed teaming is a critical component in identifying vulnerabilities and strengthening defenses against potential misuse (Practice 4.2). To further enhance the effectiveness of red team exercises, we suggest establishing guidelines for open participatory processes that allow the\n\n\n\npublic and strategically selected experts to red team a model safely, including safe harbor clauses. Additionally, industry-wide mechanisms for sharing anonymized findings should be implemented. This collaborative effort would improve collective understanding of emerging threats, foster the development of more effective mitigation strategies, and create a culture of shared responsibility within the AI community. Red teaming should be used as part of a broader suite of AI accountability tools, including algorithmic impact assessments, external audits, and public consultations.\n\n## Establishing Plans for Managing Misuse Risk\n\n(Objective 2)\n\n## Standardized Risk Thresholds and Community Input\n\nTo enhance Practice 2.1, AISI should provide comprehensive guidelines for defining and assessing acceptable levels of misuse risk in various contexts. Drawing from collaborative governance approaches like the BigCode project, we recommend the following:\n\n- \u25cf Multi-stakeholder advisory mechanisms : Encourage the formation of advisory groups with representatives from academia, industry, and civil society to review and refine risk thresholds periodically.\n- \u25cf Tiered risk categorization systems : defining risk thresholds that accommodate varying acceptable risk levels across different contexts and stakeholder groups based on potential impact and likelihood of occurrence.\n- \u25cf Open participation processes : Outline best practices for mechanisms that enable diverse stakeholders to contribute insights on risk thresholds and management strategies.\n- \u25cf Structured community input : Recommend establishing regular feedback cycles where stakeholders can provide input on risk thresholds and mitigation strategies, modeled after collaborative processes like data inspection sprints.\n- \u25cf Public documentation standards : Encourage transparent documentation of risk threshold decisions and development processes to enhance accountability and trust.\n\n## Collaborative Risk Management Roadmaps\n\nTo improve Practice 2.2, AISI should provide guidelines for organizations to adopt iterative approaches to risk management, treating roadmaps as living documents that adapt to emerging threats. Recommendations include:\n\n\n\n- \u25cf Stakeholder impact assessment tools : Develop tools for stakeholders to assess their involvement or impact within AI systems, such as BigCode's 'Am I in The Stack' tool that can help address privacy and software security risks.\n- \u25cf Regular review cycles : Establish best practices for periodic reviews of risk management roadmaps, engaging both internal teams and external advisors.\n- \u25cf Lessons learned documentation : Maintain detailed logs of risk-related insights from each iteration or release, documenting improvements and changes in mitigation strategies. For example, the Starcoder project continuously improved its data curation process, enhancing personally identifiable information (PII) redaction and opt-out mechanisms based on community input and evolving best practices.\n- \u25cf Transparency reporting : Issue regular reports outlining updates to risk assessment methodologies, changes in identified risks, and strategies to mitigate them.\n- \u25cf Collaborative knowledge sharing : Promote the creation of platforms or processes for sharing best practices in risk management, enabling collaboration between developers, researchers, and other stakeholders.\n\n## Managing Risks and Ensuring Responsible Model Release\n\n(Objectives 3 and 5)\n\n## Reframing Model Theft and Managing Misuse Risks in Open Models\n\nEfforts to protect valuable AI assets must balance the need for open science and collaboration. For open foundation models, the traditional concept of \"model theft\" requires re-evaluation. Open-weight models hold minimal or nonexistent risk of theft, depending on license and permissive use. Instead, the focus should be on responsible sharing and usage. We recommend that AISI recognize that this objective should not inadvertently deter the development and utilization of open models, which benefit from transparency and community collaboration. Efforts to prevent misuse should be proportional to the actual risks posed by open models compared to closed ones. Emphasizing the development of clear usage guidelines, ethical frameworks, and community standards for responsible AI development and deployment will be more effective. This approach aligns with Practice 3.1 by advocating for a shift from theft prevention to responsible use.\n\n## Context-Specific Safety Measures\n\nA one-size-fits-all approach to risk management is inadequate for addressing the diverse range of threats associated with AI models. Different types of risks require tailored interventions. For instance, managing non-consensual intimate imagery involves distinct strategies compared to addressing Chemical, Biological, Radiological, and Nuclear (CBRN) risks. Implementing\n\n\n\nsafeguards proportionate to the model's misuse risk (Practice 5.2) necessitates flexible, context-specific measures. A tiered system of safeguards, adaptable based on the deployment context and model impact, allows organizations to tailor their risk management strategies effectively. Safeguards should be rigorously tested, with evidence of their effectiveness established before deployment. Clear and transparent criteria for what constitutes \"adequate\" risk management (Practice 5.3) are crucial, and these criteria should be regularly reviewed and updated to reflect new research and emerging threats.\n\n## Proactive and Contextual Risk Management\n\nPre-deployment risk management is crucial for the responsible development of foundation models. To strengthen this approach, we advocate for a holistic and inclusive strategy in assessing potential misuse risks (Practice 5.1). This assessment should involve cross-functional teams, including technical experts, legal professionals, ethicists, and communications specialists. By integrating diverse perspectives, these teams can comprehensively evaluate risks, considering technical vulnerabilities, ethical implications, public perception, and legal considerations. This comprehensive approach ensures that deployment risk assessments are robust and address all facets of potential AI incidents.\n\nManaging misuse risks in open foundation models requires proactive strategies, especially since the release of model weights enables a wide range of downstream applications. Effective risk management extends beyond the model itself to include platform-specific considerations. A \"safety by design\" approach is essential, where risks are assessed before broadening access, and staged releases-such as gated models -allow controlled distribution and user verification. Model distribution safety techniques such as SafeTensors enable secure dissemination of open models. Comprehensive documentation, such as governance cards, should outline anticipated risks and mitigation strategies, empowering users to adapt models responsibly. Community engagement through discussion forums and transparent content moderation guidelines, further supports responsible deployment. By combining these strategies, platforms can manage misuse risks effectively while fostering safe and responsible AI development.\n\n## Standards for Ongoing Risk Identification and Transparency (Objectives 6 and 7)\n\n## Distributed Responsibility for Misuse Identification and Reporting\n\nCurrent guidance places substantial responsibility for identifying and responding to misuse on model developers (Practice 6.1). This approach can be particularly challenging for developers of both open models and closed models with commercial APIs serving a large and diverse user\n\n\n\nbase, who might not even have all the information required about downstream systems and use cases to effectively and independently implement risk management measures. We recommend adopting a distributed responsibility model, involving all relevant stakeholders-data providers, infrastructure providers, individual model developers, downstream application developers, and end-users-in monitoring and reporting misuse. Clear definitions of roles and responsibilities for each stakeholder should be established to facilitate effective communication and collaboration in identifying and addressing misuse.\n\n## Coordinated Response Mechanisms\n\nTo enhance Practice 6.2, AISI could establish clear incident response protocols that incorporate a collaborative responsibility model, tiered harm classification, and distributed responsibility across the AI supply chain. Drawing lessons from the mature Common Vulnerabilities and Exposures (CVE) ecosystem in cybersecurity practices, these protocols should include comprehensive steps for initial assessment and triage of reported misuses, procedures for escalation based on severity and potential impact, and guidelines for timely and transparent communication with affected parties. AISI could additionally develop standardized documentation formats for incident response, including model metadata that can be pulled from standardized model cards, incident timelines, root cause analysis, mitigation measures, and lessons learned. Templates for post-incident reports should balance transparency with the protection of sensitive information, in accordance with responsible disclosure principles. Additionally, an independent adjudicator should be designated to resolve disputes between reporters and vendors, ensuring impartial issue resolution. Similar to established practices in cybersecurity vulnerability disclosure, AI harm reporting guidelines should address safe harbor protections for reporters disclosing potential misuses in good faith, providing legal protection and promoting a culture of transparency and cooperation.\n\n## Transparency Reporting Standards\n\nTo improve Practice 7.1 and Practice 7.2 , we recommend the development of a standardized template for AI transparency reports. These reports should include:\n\n- \u25cf Sections on identified misuse risks, implemented safeguards, and a summary of misuse incidents and responses (excluding sensitive details).\n- \u25cf Guidance on ongoing monitoring efforts, with recommendations on the appropriate frequency and level of detail based on the model's capabilities and deployment context.\n- \u25cf Examples that illustrate how to present complex technical information in an accessible manner for diverse stakeholders.\n- \u25cf Guidelines for supply chain transparency, covering the documentation of training data origins, model architectures, key algorithms, and third-party components.\n\n\n\nCurrently, Practice 7.3 does not adequately address the lack of standardization and monitoring in AI incident reporting websites. For example, the AI Incident Database predominantly lists news reports, leaving out crucial findings identified through red teaming, bug bounties, or independent research. Standardization and regular maintenance of these reporting systems are crucial to ensure comprehensive coverage and effective incident management.\n\n## Conclusion\n\nHugging Face remains committed to the responsible development and deployment of AI technologies. We greatly appreciate the opportunity to provide insights on this document and we look forward to ongoing collaboration with NIST and AISI, other industry partners, researchers, and policymakers to refine and implement best practices for managing the risks associated with dual-use foundation models.\n\nSubmitted by:\n\nAvijit Ghosh , Applied Policy Researcher, Hugging Face Yacine Jernite , ML and Society Lead, Hugging Face Irene Solaiman , Head of Global Policy, Hugging Face", + "chunks": [ + { + "headings": [ + "Hugging Face Comments on NIST AI 800-1: Managing Misuse Risk for Dual-Use Foundational Models" + ], + "markdown": "\n\n## Hugging Face Comments on NIST AI 800-1: Managing Misuse Risk for Dual-Use Foundational Models\n\nHugging Face commends the US AI Safety Institute (AISI) on the AI 800-1 document: Managing Misuse Risk for Dual-Use Foundation Models. This comprehensive framework identifies key objectives and practices for managing risks associated with foundation models. We offer recommendations to strengthen this document based on our experiences in democratizing good AI and characterizing risks of systems as an open platform for state-of-the-art (SotA) AI systems. Our comments are organized by objectives and practices as outlined in the document. Where we do not have specific, actionable feedback on a section or practice, we have not highlighted it.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## Executive Summary\n\nHugging Face commends the US AI Safety Institute (AISI) for its comprehensive AI 800-1 document to address managing misuse risks for dual-use foundation models. Based on our experience as a leading open AI platform, we offer the following key recommendations to enhance the framework's effectiveness:\n\n- 1. Joint Management Across the AI Supply Chain : Risk management should be a shared responsibility among all stakeholders, including data providers, infrastructure providers, and end-users, rather than solely on individual model developers. Encourage open, collaborative approaches where diverse stakeholders contribute to defining risk thresholds and management strategies.\n- 2. Enhance Transparency, Accountability, and Ongoing Risk Identification : Implement regular transparency reporting on risk management practices, and establish mechanisms for meaningful accountability, including sharing information with\n\n\n\nindependent entities. Improve scientific and regulatory visibility on risk profiles by providing clear guidelines for reporting and categorizing misuse risks, and support external safety research through vulnerability disclosure policies and safe harbor provisions.\n\n- 3. Tailor Risk Interventions and Balance Security with Accessibility : Different risks require context-specific, flexible safeguards. Ensure that interventions are tailored to the nature of the risk, whether it be non-consensual intimate imagery or CBRN threats. When considering model access restrictions, balance the need for security with the benefits of openness to foster innovation and research.\n- 4. Recognize and Support Open Foundation Models : Explicitly acknowledge the unique characteristics and benefits of open foundation models, including their role in enabling external scrutiny, mitigating monoculture, and fostering innovation. Develop guidelines that support responsible open-source AI development while managing associated risks.\n\n## Objectives and Practices to Manage Misuse Risks\n\n## Anticipating and Measuring Risk\n\n(Objectives 1 and 4)\n\n## Holistic Approach to Risk Assessment\n\nTo effectively manage and safely deploy AI models, it is essential to adopt a holistic approach to risk assessment that encompasses both technical and societal factors. While this document primarily focuses on safety risks, it is important to recognize that mechanisms designed to address these risks can also be applied to broader societal concerns. Failing to integrate these considerations could lead to missed opportunities for creating more comprehensive and effective risk management strategies. For instance, the focus on preventing \"model theft\" might overlook the value of broad, inclusive participation in identifying model biases and other potential harms. A framework for measuring risks that comprehensively addresses foundational models' impacts on people and society would ensure that safety measures do not inadvertently neglect significant societal implications - or make addressing them through other means more difficult.\n\n## Collaborative Risk Definition\n\nThe current approach to risk definition, which relies on individual developers to define and assess risks, introduces several challenges. It creates disparities between corporate and collaborative developers, complicates comparison between models from different developers,\n\n\n\nand deviates from established scientific processes. Instead, we advocate for a more standardized, collaborative approach. This approach could involve a centralized system where diverse stakeholders contribute to and maintain a comprehensive list of potential malicious uses or unintended consequences of foundation models, akin to the Common Use Enumeration (CUE) component we have proposed for structured harm reporting. This collaborative approach offers several benefits:\n\n- \u25cf Broader Expertise: It leverages a wider range of expertise, including domain-specific knowledge that individual companies may lack.\n- \u25cf Resource Equity: It reduces disparities in resources and incentives for risk evaluation between corporate and collaborative developers.\n- \u25cf Consistent Assessment: It enables more consistent risk assessment across different models and developers, facilitating meaningful comparisons and industry-wide progress on safety.\n- \u25cf Scientific Alignment: It aligns more closely with established scientific processes for risk assessment.\n- \u25cf Comprehensive Risk Identification: It reduces duplicated efforts across organizations and is likely to identify a more comprehensive and nuanced set of potential risks than any single organization could alone.\n\nBy moving towards a more inclusive, standardized approach, the industry can establish a more robust, scientifically rigorous, and comprehensive system for anticipating and managing potential misuse risks in foundation models. Such an approach corresponds to Practice 1.1 by improving the standardization and inclusivity of risk assessments.\n\n## Transparency in Capability Estimation\n\nTransparency is crucial in estimating model capabilities before deployment (Practice 4.1). We recommend publicly documenting the methods used for capability estimation, including any limitations or uncertainties. Developing and using open benchmarks, leaderboards, and evaluation tools will enable more standardized and comparable capability assessments across the industry. Periodic measurement of capabilities throughout the development process is commendable, but this should be viewed as an opportunity to focus more on upstream development choices, such as dataset selection and model scaling, rather than merely increasing capabilities. For open models, leveraging their openness for collaborative risk identification is key.\n\n## More Effective Red Teaming\n\nRed teaming is a critical component in identifying vulnerabilities and strengthening defenses against potential misuse (Practice 4.2). To further enhance the effectiveness of red team exercises, we suggest establishing guidelines for open participatory processes that allow the\n\n\n\npublic and strategically selected experts to red team a model safely, including safe harbor clauses. Additionally, industry-wide mechanisms for sharing anonymized findings should be implemented. This collaborative effort would improve collective understanding of emerging threats, foster the development of more effective mitigation strategies, and create a culture of shared responsibility within the AI community. Red teaming should be used as part of a broader suite of AI accountability tools, including algorithmic impact assessments, external audits, and public consultations.\n\n## Establishing Plans for Managing Misuse Risk\n\n(Objective 2)\n\n## Standardized Risk Thresholds and Community Input\n\nTo enhance Practice 2.1, AISI should provide comprehensive guidelines for defining and assessing acceptable levels of misuse risk in various contexts. Drawing from collaborative governance approaches like the BigCode project, we recommend the following:\n\n- \u25cf Multi-stakeholder advisory mechanisms : Encourage the formation of advisory groups with representatives from academia, industry, and civil society to review and refine risk thresholds periodically.\n- \u25cf Tiered risk categorization systems : defining risk thresholds that accommodate varying acceptable risk levels across different contexts and stakeholder groups based on potential impact and likelihood of occurrence.\n- \u25cf Open participation processes : Outline best practices for mechanisms that enable diverse stakeholders to contribute insights on risk thresholds and management strategies.\n- \u25cf Structured community input : Recommend establishing regular feedback cycles where stakeholders can provide input on risk thresholds and mitigation strategies, modeled after collaborative processes like data inspection sprints.\n- \u25cf Public documentation standards : Encourage transparent documentation of risk threshold decisions and development processes to enhance accountability and trust.\n\n## Collaborative Risk Management Roadmaps\n\nTo improve Practice 2.2, AISI should provide guidelines for organizations to adopt iterative approaches to risk management, treating roadmaps as living documents that adapt to emerging threats. Recommendations include:\n\n\n\n- \u25cf Stakeholder impact assessment tools : Develop tools for stakeholders to assess their involvement or impact within AI systems, such as BigCode's 'Am I in The Stack' tool that can help address privacy and software security risks.\n- \u25cf Regular review cycles : Establish best practices for periodic reviews of risk management roadmaps, engaging both internal teams and external advisors.\n- \u25cf Lessons learned documentation : Maintain detailed logs of risk-related insights from each iteration or release, documenting improvements and changes in mitigation strategies. For example, the Starcoder project continuously improved its data curation process, enhancing personally identifiable information (PII) redaction and opt-out mechanisms based on community input and evolving best practices.\n- \u25cf Transparency reporting : Issue regular reports outlining updates to risk assessment methodologies, changes in identified risks, and strategies to mitigate them.\n- \u25cf Collaborative knowledge sharing : Promote the creation of platforms or processes for sharing best practices in risk management, enabling collaboration between developers, researchers, and other stakeholders.\n\n## Managing Risks and Ensuring Responsible Model Release\n\n(Objectives 3 and 5)\n\n## Reframing Model Theft and Managing Misuse Risks in Open Models\n\nEfforts to protect valuable AI assets must balance the need for open science and collaboration. For open foundation models, the traditional concept of \"model theft\" requires re-evaluation. Open-weight models hold minimal or nonexistent risk of theft, depending on license and permissive use. Instead, the focus should be on responsible sharing and usage. We recommend that AISI recognize that this objective should not inadvertently deter the development and utilization of open models, which benefit from transparency and community collaboration. Efforts to prevent misuse should be proportional to the actual risks posed by open models compared to closed ones. Emphasizing the development of clear usage guidelines, ethical frameworks, and community standards for responsible AI development and deployment will be more effective. This approach align", + "char_slice": [ + 0, + 11808 + ] + }, + { + "headings": [ + "## Hugging Face Comments on NIST AI 800-1: Managing Misuse Risk for Dual-Use Foundational Models", + "## About Hugging Face", + "## Executive Summary", + "## Objectives and Practices to Manage Misuse Risks", + "## Anticipating and Measuring Risk", + "## Holistic Approach to Risk Assessment", + "## Collaborative Risk Definition", + "## Transparency in Capability Estimation", + "## More Effective Red Teaming", + "## Establishing Plans for Managing Misuse Risk", + "## Standardized Risk Thresholds and Community Input", + "## Collaborative Risk Management Roadmaps", + "## Managing Risks and Ensuring Responsible Model Release", + "## Reframing Model Theft and Managing Misuse Risks in Open Models" + ], + "markdown": ") redaction and opt-out mechanisms based on community input and evolving best practices.\n- \u25cf Transparency reporting : Issue regular reports outlining updates to risk assessment methodologies, changes in identified risks, and strategies to mitigate them.\n- \u25cf Collaborative knowledge sharing : Promote the creation of platforms or processes for sharing best practices in risk management, enabling collaboration between developers, researchers, and other stakeholders.\n\n## Managing Risks and Ensuring Responsible Model Release\n\n(Objectives 3 and 5)\n\n## Reframing Model Theft and Managing Misuse Risks in Open Models\n\nEfforts to protect valuable AI assets must balance the need for open science and collaboration. For open foundation models, the traditional concept of \"model theft\" requires re-evaluation. Open-weight models hold minimal or nonexistent risk of theft, depending on license and permissive use. Instead, the focus should be on responsible sharing and usage. We recommend that AISI recognize that this objective should not inadvertently deter the development and utilization of open models, which benefit from transparency and community collaboration. Efforts to prevent misuse should be proportional to the actual risks posed by open models compared to closed ones. Emphasizing the development of clear usage guidelines, ethical frameworks, and community standards for responsible AI development and deployment will be more effective. This approach aligns with Practice 3.1 by advocating for a shift from theft prevention to responsible use.\n\n## Context-Specific Safety Measures\n\nA one-size-fits-all approach to risk management is inadequate for addressing the diverse range of threats associated with AI models. Different types of risks require tailored interventions. For instance, managing non-consensual intimate imagery involves distinct strategies compared to addressing Chemical, Biological, Radiological, and Nuclear (CBRN) risks. Implementing\n\n\n\nsafeguards proportionate to the model's misuse risk (Practice 5.2) necessitates flexible, context-specific measures. A tiered system of safeguards, adaptable based on the deployment context and model impact, allows organizations to tailor their risk management strategies effectively. Safeguards should be rigorously tested, with evidence of their effectiveness established before deployment. Clear and transparent criteria for what constitutes \"adequate\" risk management (Practice 5.3) are crucial, and these criteria should be regularly reviewed and updated to reflect new research and emerging threats.\n\n## Proactive and Contextual Risk Management\n\nPre-deployment risk management is crucial for the responsible development of foundation models. To strengthen this approach, we advocate for a holistic and inclusive strategy in assessing potential misuse risks (Practice 5.1). This assessment should involve cross-functional teams, including technical experts, legal professionals, ethicists, and communications specialists. By integrating diverse perspectives, these teams can comprehensively evaluate risks, considering technical vulnerabilities, ethical implications, public perception, and legal considerations. This comprehensive approach ensures that deployment risk assessments are robust and address all facets of potential AI incidents.\n\nManaging misuse risks in open foundation models requires proactive strategies, especially since the release of model weights enables a wide range of downstream applications. Effective risk management extends beyond the model itself to include platform-specific considerations. A \"safety by design\" approach is essential, where risks are assessed before broadening access, and staged releases-such as gated models -allow controlled distribution and user verification. Model distribution safety techniques such as SafeTensors enable secure dissemination of open models. Comprehensive documentation, such as governance cards, should outline anticipated risks and mitigation strategies, empowering users to adapt models responsibly. Community engagement through discussion forums and transparent content moderation guidelines, further supports responsible deployment. By combining these strategies, platforms can manage misuse risks effectively while fostering safe and responsible AI development.\n\n## Standards for Ongoing Risk Identification and Transparency (Objectives 6 and 7)\n\n## Distributed Responsibility for Misuse Identification and Reporting\n\nCurrent guidance places substantial responsibility for identifying and responding to misuse on model developers (Practice 6.1). This approach can be particularly challenging for developers of both open models and closed models with commercial APIs serving a large and diverse user\n\n\n\nbase, who might not even have all the information required about downstream systems and use cases to effectively and independently implement risk management measures. We recommend adopting a distributed responsibility model, involving all relevant stakeholders-data providers, infrastructure providers, individual model developers, downstream application developers, and end-users-in monitoring and reporting misuse. Clear definitions of roles and responsibilities for each stakeholder should be established to facilitate effective communication and collaboration in identifying and addressing misuse.\n\n## Coordinated Response Mechanisms\n\nTo enhance Practice 6.2, AISI could establish clear incident response protocols that incorporate a collaborative responsibility model, tiered harm classification, and distributed responsibility across the AI supply chain. Drawing lessons from the mature Common Vulnerabilities and Exposures (CVE) ecosystem in cybersecurity practices, these protocols should include comprehensive steps for initial assessment and triage of reported misuses, procedures for escalation based on severity and potential impact, and guidelines for timely and transparent communication with affected parties. AISI could additionally develop standardized documentation formats for incident response, including model metadata that can be pulled from standardized model cards, incident timelines, root cause analysis, mitigation measures, and lessons learned. Templates for post-incident reports should balance transparency with the protection of sensitive information, in accordance with responsible disclosure principles. Additionally, an independent adjudicator should be designated to resolve disputes between reporters and vendors, ensuring impartial issue resolution. Similar to established practices in cybersecurity vulnerability disclosure, AI harm reporting guidelines should address safe harbor protections for reporters disclosing potential misuses in good faith, providing legal protection and promoting a culture of transparency and cooperation.\n\n## Transparency Reporting Standards\n\nTo improve Practice 7.1 and Practice 7.2 , we recommend the development of a standardized template for AI transparency reports. These reports should include:\n\n- \u25cf Sections on identified misuse risks, implemented safeguards, and a summary of misuse incidents and responses (excluding sensitive details).\n- \u25cf Guidance on ongoing monitoring efforts, with recommendations on the appropriate frequency and level of detail based on the model's capabilities and deployment context.\n- \u25cf Examples that illustrate how to present complex technical information in an accessible manner for diverse stakeholders.\n- \u25cf Guidelines for supply chain transparency, covering the documentation of training data origins, model architectures, key algorithms, and third-party components.\n\n\n\nCurrently, Practice 7.3 does not adequately address the lack of standardization and monitoring in AI incident reporting websites. For example, the AI Incident Database predominantly lists news reports, leaving out crucial findings identified through red teaming, bug bounties, or independent research. Standardization and regular maintenance of these reporting systems are crucial to ensure comprehensive coverage and effective incident management.\n\n## Conclusion\n\nHugging Face remains committed to the responsible development and deployment of AI technologies. We greatly appreciate the opportunity to provide insights on this document and we look forward to ongoing collaboration with NIST and AISI, other industry partners, researchers, and policymakers to refine and implement best practices for managing the risks associated with dual-use foundation models.\n\nSubmitted by:\n\nAvijit Ghosh , Applied Policy Researcher, Hugging Face Yacine Jernite , ML and Society Lead, Hugging Face Irene Solaiman , Head of Global Policy, Hugging Face", + "char_slice": [ + 10343, + 19048 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2024_BIS-2024-0047_Response.pdf", + "md_content": "\n\n## Hugging Face Comments on BIS-2024-0047 / RIN 0694-AJ55: Establishment of Reporting Requirements for the Development of Advanced Artificial Intelligence Models and Computing Clusters\n\nThe proposed rule regarding Establishment of Reporting Requirements for the Development of Advanced Artificial Intelligence Models and Computing Clusters covers important aspects of reporting; it sets clear goals of who should be required to report and excludes small actors who might not have the resources to fulfill regular reporting requirements. It further recognizes the changing nature of AI technology by ensuring that requirements and thresholds can be updated. We offer recommendations to strengthen the reporting requirements based on our experience with model documentation and work on social impact evaluations. We have organized our feedback by the questions highlighted in the Request for Comments and the definitions included in the proposed rule.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## General Comment: Safeguarding Research to Support Safer Technology\n\nHugging Face is a platform that supports open science, collaboration on new models and AI technology, and replicable and transparent research into the properties of AI models. As such, we feel the need to draw attention to the specific conditions of open and collaborative research and development of AI to ensure that regulatory requirements remain compatible with those conditions. In particular, as regards the reporting requirements for models that meet the proposed threshold, we argue that they should be designed with the full range of stakeholders in mind, including smaller actors such as start-ups, non-profit, and academic developers who do play a critical role in enabling sustainable evidence-based policymaking on large models despite having access to fewer computational resources than larger companies.\n\nScientific replication efforts of models that were at the 'frontier' of computational requirements at the time of their release in particular have been extremely valuable. Those efforts usually incur significantly decreased cost compared to the initial development due to several factors\n\n\n\nincluding the decreasing cost of computational resourced over time, volunteer participation, increased reliance on public or donated data; and contrary to commercial development, those efforts do not depend on expensive inference cost for at-scale commercial deployment to meet their goals. Examples of such efforts include the BigScience project, which created BLOOM, the largest open-source multilingual AI model, through collaboration between research institutions, startups, and companies. This project, supported by a public grant to support the computational costs, replicated the capabilities of the then-\"frontier\" GPT-3 model a year after its release. BLOOM not only supports research on large language models but also plays a key role in advancing the understanding of safety, as it is widely accessible to researchers. Another example of large open-source AI models include Pythia, a suite for analyzing LLMs, and the open models developed by Allen AI, such as MOLMo, which has been shown to reach competitive performance to some of the most advanced commercial alternatives.\n\nWhile such models may have limited direct economic impact, their value to research, innovation, and regulation is substantial. Among other contributions, these models have enabled research into vulnerabilities shared by commercial systems, including their susceptibility to jailbreaking, likelihood of memorizing and potentially leaking training information, and model extraction risks through API access. While efforts of these kinds have not yet trained a model involving more than 10^26 FlOps, we expect that access to sufficient public or private compute grants and funding will enable it in the future. Compliance with reporting requirements should therefore remain accessible to organizations with such unique needs as temporary coalitions of small actors or academic departments whose internal governance structure and legal representation differ from those of established commercial entities, especially in cases when the model development does not present consequent additional risk compared to existing commercial models. Additionally, much of the value of these efforts comes from their role in enabling external research on guardrails, evaluation, and more technically involved red-teaming, which complicates reporting of results. We make recommendations tailored for enabling compliance in those settings in the rest of this document.\n\n## Quarterly Notification Schedule\n\nWhile we acknowledge the need for timely notification, the proposed quarterly notification as it applies to 'any covered U.S. person', including ones who are not currently developing any covered models, seems excessive. We propose instead that model developers should be required to report on covered activities when they are planning a training run and before the start of that training run, and that the reporting requirement of no applicable activity be removed, considering in particular the risks of administrative oversights in organizations with changing structures or personnel, including academic and non-profit organizations and start-ups in earlier growth stages.\n\n\n\n## Collection Thresholds\n\nWhile the proposed collection thresholds based on computational operations and networking capabilities serve as a partial indicator of the resources required for reporting, they are not sufficient to adequately assess the risks associated with AI models. Compute thresholds, such as those outlined in E.O. 14110, offer insight into the scale of AI model training but fall short of capturing the broader risk landscape.\n\nThe risks posed by AI models cannot be understood in isolation from the context in which they are deployed, how they are used, and how accessible they are in terms of user interfaces and public access. Factors like the purpose of the model, the composition of its training data, its downstream applications, and its potential for misuse or harm are critical dimensions that go beyond raw computational power.\n\nIn order to reflect these considerations while preserving important open and collaborative research efforts, we propose a reporting scheme in two stages. We recommend that covered U.S. persons planning a training run that meets the given requirements should submit a broad description of their system that includes additional risk factors, including for example the inclusion of sources of data that contain information related to CBRN constraints, plans for deploying the model at scale, and plans for developing the model to enable entirely novel capabilities not available in existing widely available systems, but not additional information on system-level guardrails and testing occurring outside of the core model development activity (see our note on distinguishing definitions of AI models and AI systems ). Models that do not present particular additional risk factors along any of those dimensions should be dispensed from further reporting. Models that do could be requested to provide additional information on testing and developers could remain available for follow-up outreach.\n\nThis approach aligns with frameworks like the EU AI Act, where developers who meet a compute-based risk threshold are able to argue that their model does not present systemic risk if it has comparable capabilities to existing models. It also ensures that attention is focused on the models with the highest risk, while allowing open and collaborative research to keep pace with commercial development and provide an important scientific evidence basis to support further regulation.\n\nFurthermore, the inventory of models generated from this reporting could be coordinated with other disclosure processes, including for example systems that allow experts and users of AI systems to follow a process of coordinated disclosures when flaws or risks related to AI models and systems are identified.\n\n## Definitions\n\n## AI red-teaming\n\nThe current definition of AI red-teaming could benefit from greater clarity and depth, particularly around key areas like the scope of the red-teaming, how experts are selected, and who gets to contribute to the overall safety and reliability testing of AI systems.\n\nSafety testing should be applied to both the AI model as well as possible guardrails (see c(3)(ii)) to ensure knowledge about both the model capabilities and the guardrails employed on the model. Those guard rails can change faster than retraining a model and research has shown that these guard rails are not immune to jailbreaking.\n\nThese developments are most often publicly reported by external researchers. The current definition of AI red-teaming requires ' collaboration with developers of AI '. It should be clarified that the involvement of third party actors, who are not involved in the development of the AI system, is desirable. Encouraging open, collaborative approaches would allow diverse voices to contribute to defining risk thresholds and management strategies, fostering a more comprehensive risk management framework. Establishing guidelines for safe public and expert participation in red-teaming, along with safe harbor clauses, would expand its reach and effectiveness. Additionally, organizations would benefit from anonymized, shared findings, enabling collective understanding of emerging threats. This would foster a culture of shared responsibility, where the entire AI community contributes to safer, more reliable systems.\n\nWhile red-teaming is helpful for identifying flaws and vulnerabilities in AI systems, such as harmful outputs or unforeseen behaviors, it should be part of a broader, collaborative risk management strategy, integrating social impact evaluations.\n\nAny model evaluation framework needs to ensure that the testing of models is not replicating what is part of the training data, i.e., data leakage. A level of reporting on the training data is required to ensure that evaluation results are reflecting the required standards.\n\n## AI model and AI system\n\nThe current definitions of an AI model and system are currently too close to support regulatory efforts that have to rely on important distinctions between both. We recommend strengthening that distinction, taking inspiration for example from definitions of texts like the EU AI Act that make different requirements of models and systems.\n\nAn AI model is best understood as information: typically encoded as a set of model weights that represent statistics extracted from training data. An AI model cannot, by itself, produce outputs given inputs without additional software and hardware components.\n\n\n\n\n\nAn AI system is obtained by combining one or several models with different software components to process inputs and outputs and run information through the model weights, as well as with hardware (typically a Large-scale computing cluster ) to enable the required computation.\n\nThese differences are meaningful on several accounts. Firstly, guardrails are often integrated in a system as input or output processing outside of the core models, and are better discussed as a property of the system than of the models; in particular, model-level safety-finetuning type approaches have been shown to be brittle both for commercial APIs and for models with widely available weights. Secondly, the cost of deployment of a model is a significant factor in assessing the risks posed by its deployment. A model that is easily accessible to hundreds of millions of users through commercial deployment supported by a large cluster may pose risks on a different scale than a model released as a set of weights.\n\n## Dual-use foundation model\n\nThe definition of dual-use foundation model contains a requirement of size in parameters (' (c) Contains at least tens of billions of parameters '), which might be outdated as model sizes are changing over time. A definition of dual-use foundation model should focus on the application and context of the model rather than its size.\n\n## Submitted by:\n\nLucie-Aim\u00e9e Kaffee, Yacine Jernite, and Irene Solaiman; with input from Avijit Ghosh and Margaret Mitchell Hugging Face", + "chunks": [ + { + "headings": [ + "Hugging Face Comments on BIS-2024-0047 / RIN 0694-AJ55: Establishment of Reporting Requirements for the Development of Advanced Artificial Intelligence Models and Computing Clusters" + ], + "markdown": "\n\n## Hugging Face Comments on BIS-2024-0047 / RIN 0694-AJ55: Establishment of Reporting Requirements for the Development of Advanced Artificial Intelligence Models and Computing Clusters\n\nThe proposed rule regarding Establishment of Reporting Requirements for the Development of Advanced Artificial Intelligence Models and Computing Clusters covers important aspects of reporting; it sets clear goals of who should be required to report and excludes small actors who might not have the resources to fulfill regular reporting requirements. It further recognizes the changing nature of AI technology by ensuring that requirements and thresholds can be updated. We offer recommendations to strengthen the reporting requirements based on our experience with model documentation and work on social impact evaluations. We have organized our feedback by the questions highlighted in the Request for Comments and the definitions included in the proposed rule.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## General Comment: Safeguarding Research to Support Safer Technology\n\nHugging Face is a platform that supports open science, collaboration on new models and AI technology, and replicable and transparent research into the properties of AI models. As such, we feel the need to draw attention to the specific conditions of open and collaborative research and development of AI to ensure that regulatory requirements remain compatible with those conditions. In particular, as regards the reporting requirements for models that meet the proposed threshold, we argue that they should be designed with the full range of stakeholders in mind, including smaller actors such as start-ups, non-profit, and academic developers who do play a critical role in enabling sustainable evidence-based policymaking on large models despite having access to fewer computational resources than larger companies.\n\nScientific replication efforts of models that were at the 'frontier' of computational requirements at the time of their release in particular have been extremely valuable. Those efforts usually incur significantly decreased cost compared to the initial development due to several factors\n\n\n\nincluding the decreasing cost of computational resourced over time, volunteer participation, increased reliance on public or donated data; and contrary to commercial development, those efforts do not depend on expensive inference cost for at-scale commercial deployment to meet their goals. Examples of such efforts include the BigScience project, which created BLOOM, the largest open-source multilingual AI model, through collaboration between research institutions, startups, and companies. This project, supported by a public grant to support the computational costs, replicated the capabilities of the then-\"frontier\" GPT-3 model a year after its release. BLOOM not only supports research on large language models but also plays a key role in advancing the understanding of safety, as it is widely accessible to researchers. Another example of large open-source AI models include Pythia, a suite for analyzing LLMs, and the open models developed by Allen AI, such as MOLMo, which has been shown to reach competitive performance to some of the most advanced commercial alternatives.\n\nWhile such models may have limited direct economic impact, their value to research, innovation, and regulation is substantial. Among other contributions, these models have enabled research into vulnerabilities shared by commercial systems, including their susceptibility to jailbreaking, likelihood of memorizing and potentially leaking training information, and model extraction risks through API access. While efforts of these kinds have not yet trained a model involving more than 10^26 FlOps, we expect that access to sufficient public or private compute grants and funding will enable it in the future. Compliance with reporting requirements should therefore remain accessible to organizations with such unique needs as temporary coalitions of small actors or academic departments whose internal governance structure and legal representation differ from those of established commercial entities, especially in cases when the model development does not present consequent additional risk compared to existing commercial models. Additionally, much of the value of these efforts comes from their role in enabling external research on guardrails, evaluation, and more technically involved red-teaming, which complicates reporting of results. We make recommendations tailored for enabling compliance in those settings in the rest of this document.\n\n## Quarterly Notification Schedule\n\nWhile we acknowledge the need for timely notification, the proposed quarterly notification as it applies to 'any covered U.S. person', including ones who are not currently developing any covered models, seems excessive. We propose instead that model developers should be required to report on covered activities when they are planning a training run and before the start of that training run, and that the reporting requirement of no applicable activity be removed, considering in particular the risks of administrative oversights in organizations with changing structures or personnel, including academic and non-profit organizations and start-ups in earlier growth stages.\n\n\n\n## Collection Thresholds\n\nWhile the proposed collection thresholds based on computational operations and networking capabilities serve as a partial indicator of the resources required for reporting, they are not sufficient to adequately assess the risks associated with AI models. Compute thresholds, such as those outlined in E.O. 14110, offer insight into the scale of AI model training but fall short of capturing the broader risk landscape.\n\nThe risks posed by AI models cannot be understood in isolation from the context in which they are deployed, how they are used, and how accessible they are in terms of user interfaces and public access. Factors like the purpose of the model, the composition of its training data, its downstream applications, and its potential for misuse or harm are critical dimensions that go beyond raw computational power.\n\nIn order to reflect these considerations while preserving important open and collaborative research efforts, we propose a reporting scheme in two stages. We recommend that covered U.S. persons planning a training run that meets the given requirements should submit a broad description of their system that includes additional risk factors, including for example the inclusion of sources of data that contain information related to CBRN constraints, plans for deploying the model at scale, and plans for developing the model to enable entirely novel capabilities not available in existing widely available systems, but not additional information on system-level guardrails and testing occurring outside of the core model development activity (see our note on distinguishing definitions of AI models and AI systems ). Models that do not present particular additional risk factors along any of those dimensions should be dispensed from further reporting. Models that do could be requested to provide additional information on testing and developers could remain available for follow-up outreach.\n\nThis approach aligns with frameworks like the EU AI Act, where developers who meet a compute-based risk threshold are able to argue that their model does not present systemic risk if it has comparable capabilities to existing models. It also ensures that attention is focused on the models with the highest risk, while allowing open and collaborative research to keep pace with commercial development and provide an important scientific evidence basis to support further regulation.\n\nFurthermore, the inventory of models generated from this reporting could be coordinated with other disclosure processes, including for example systems that allow experts and users of AI systems to follow a process of coordinated disclosures when flaws or risks related to AI models and systems are identified.\n\n## Definitions\n\n## AI red-teaming\n\nThe current definition of AI red-teaming could benefit from greater clarity and depth, particularly around key areas like the scope of the red-teaming, how experts are selected, and who gets to contribute to the overall safety and reliability testing of AI systems.\n\nSafety testing should be applied to both the AI model as well as possible guardrails (see c(3)(ii)) to ensure knowledge about both the model capabilities and the guardrails employed on the model. Those guard rails can change faster than retraining a model and research has shown that these guard rails are not immune to jailbreaking.\n\nThese developments are most often publicly reported by external researchers. The current definition of AI red-teaming requires ' collaboration with developers of AI '. It should be clarified that the involvement of third party actors, who are not involved in the development of the AI system, is desirable. Encouraging open, collaborative approaches would allow diverse voices to contribute to defining risk thresholds and management strategies, fostering a more comprehensive risk management framework. Establishing guidelines for safe public and expert participation in red-teaming, along with safe harbor clauses, would expand its reach and effectiveness. Additionally, organizations would benefit from anonymized, shared findings, enabling collective understanding of emerging threats. This would foster a culture of shared responsibility, where the entire AI community contributes to safer, more reliable systems.\n\nWhile red-teaming is helpful for identifying flaws and vulnerabilities in AI systems, such as harmful outputs or unforeseen behaviors, it should be part of a broader, collaborative risk management strategy, integrating social impact evaluations.\n\nAny model evaluation framework needs to ensure that the testing of models is not replicating what is part of the training data, i.e., data leakage. A level of reporting on the training data is required to ensure that evaluation results are reflecting the required standards.\n\n## AI model and AI system\n\nThe current definitions of an AI model and system are currently too close to support regulatory efforts that have to rely on important distinctions between both. We recommend strengthening that distinction, taking inspiration for example from definitions of texts like the EU AI Act that make different requirements of models and systems.\n\nAn AI model is best understood as information: typically encoded as a set of model weights that represent statistics extracted from training data. An AI model cannot, by itself, produce outputs given inputs without additional software and hardware components.\n\n\n\n\n\nAn AI system is obtained by combining one or several models with different software components to process inputs and outputs and run information through the model", + "char_slice": [ + 0, + 11534 + ] + }, + { + "headings": [ + "## Hugging Face Comments on BIS-2024-0047 / RIN 0694-AJ55: Establishment of Reporting Requirements for the Development of Advanced Artificial Intelligence Models and Computing Clusters", + "## About Hugging Face", + "## General Comment: Safeguarding Research to Support Safer Technology", + "## Quarterly Notification Schedule", + "## Collection Thresholds", + "## Definitions", + "## AI red-teaming", + "## AI model and AI system" + ], + "markdown": "to safer, more reliable systems.\n\nWhile red-teaming is helpful for identifying flaws and vulnerabilities in AI systems, such as harmful outputs or unforeseen behaviors, it should be part of a broader, collaborative risk management strategy, integrating social impact evaluations.\n\nAny model evaluation framework needs to ensure that the testing of models is not replicating what is part of the training data, i.e., data leakage. A level of reporting on the training data is required to ensure that evaluation results are reflecting the required standards.\n\n## AI model and AI system\n\nThe current definitions of an AI model and system are currently too close to support regulatory efforts that have to rely on important distinctions between both. We recommend strengthening that distinction, taking inspiration for example from definitions of texts like the EU AI Act that make different requirements of models and systems.\n\nAn AI model is best understood as information: typically encoded as a set of model weights that represent statistics extracted from training data. An AI model cannot, by itself, produce outputs given inputs without additional software and hardware components.\n\n\n\n\n\nAn AI system is obtained by combining one or several models with different software components to process inputs and outputs and run information through the model weights, as well as with hardware (typically a Large-scale computing cluster ) to enable the required computation.\n\nThese differences are meaningful on several accounts. Firstly, guardrails are often integrated in a system as input or output processing outside of the core models, and are better discussed as a property of the system than of the models; in particular, model-level safety-finetuning type approaches have been shown to be brittle both for commercial APIs and for models with widely available weights. Secondly, the cost of deployment of a model is a significant factor in assessing the risks posed by its deployment. A model that is easily accessible to hundreds of millions of users through commercial deployment supported by a large cluster may pose risks on a different scale than a model released as a set of weights.\n\n## Dual-use foundation model\n\nThe definition of dual-use foundation model contains a requirement of size in parameters (' (c) Contains at least tens of billions of parameters '), which might be outdated as model sizes are changing over time. A definition of dual-use foundation model should focus on the application and context of the model rather than its size.\n\n## Submitted by:\n\nLucie-Aim\u00e9e Kaffee, Yacine Jernite, and Irene Solaiman; with input from Avijit Ghosh and Margaret Mitchell Hugging Face", + "char_slice": [ + 10155, + 12874 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2024_DSA_Response.pdf", + "md_content": "## Hugging Face S.A.S\n\n9 rue des colonnes,\n\n75002 Paris, France\n\nJanuary 24., 2023\n\n## Hugging Face Feedback on the Digital Services Acts Transparency Reports\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. As part of our activity, Hugging Face provides social features and communication platforms for people interacting with AI systems, including social posts and discussion threads, in addition to hosting the AI systems themselves.\n\n## Comments on 'Draft implementing regulation Ares(2023)8428591.pdf'\n\n## On adding further content:\n\nWe suggest that transparency reports provide readers with relevant information about the recommender systems that manage the content on their platform. Consider requiring documentation in the transparency reports for any relevant parameters of recommendation systems that affect content moderation, as well as information about how the moderation action may affect future recommendations. This could be, for example, at the same level of granularity as when applying recital 70 and article 27(2).\n\n## On 'machine readable' formats for reports:\n\nThe current framing explains, 'to ensure that the transparency reports are machine-readable', transparency reports are required to be in a CSV (comma-separated values) format. We agree this idea is good at a high-level, but in practice csv formats can have issues in appropriate rendering (for example if there is content with additional commas, non ASCII-characters, etc.) This also misses the option to have easily human-readable reports that enable people to interact with the content, such as by printing it out, highlighting, making notes, etc. much easier.\n\n\n\n\n\nOur Suggestion is to support this requirement by creating a script that renders an input .csv transparency report as a human-readable pdf. In a user interaction with this script, users and organizations would use an upload portal with this script embedded to upload their .csv transparency report, which then renders it in a human-readable pdf for the organization to check. If the rendering has errors, this is on the organization's side to fix before their final submission. This removes the burden from the government's side of trying to solve for another organization's errors, and makes it much easier for the organization to find issues. This also ensures human-readability as well as machine-readability.\n\n## Comments on Annex 1 - Templates\n\n## Regarding 2\\_categories\\_names\n\nThe categories bring up several issues. Two common themes emerge across them:\n\n- 1. The distinctions between illegal and legal content. Legal and illegal content are grouped together in this categorization. Sometimes 'illegal' is mentioned specifically; sometimes it is not. We recommend keeping these two distinct. The reasons for this include:\n- a. Not requiring service providers to make the decision as to whether something is technically illegal or not\n- b. Not creating reporting requirements for legal and appropriate content\n- c. Not creating (additional) stigma against legal activities\n\nFurther, for services that are offered in more than one country in the EU, the definition of 'illegal' may vary, causing potential confusion as to which definition of illegal should be considered in the report.\n\nOur Suggestion : If there is a goal for service providers to document illegal/unflawful content explicitly, consider revising the instructions to specify that service providers should report all illegal content they are aware of for each category.\n\n## 2. Domains and stakeholders .\n\n- a. Modality: A focus here is on image content. Consider removing the focus solely on images, or else explicitly including additional modalities where the harms of concern are also expressed, such as audio.\n- b. Stakeholders: In all categories, consider additionally grounding on EU 'protected grounds' and/or 'sensitive personal characteristics'. This includes categories such as those listed as sensitive personal data or subject to\n\n\n\nnon-discrimination by the European Commission or in European non-discrimination law. This would also mean updating the subcategory of 'Gender-based violence' to something like 'Violence based on protected grounds or sensitive characteristics' and the subcategory of 'Biometric data breach' to something like 'Protected personal data breach' .\n\nWe also note that these categories will have overlapping issues, which may make reporting a bit more confusing using this format. For example, 'Negative effects on civic discourse or elections' could include racist misinformation, which would also fall under 'Illegal or harmful speech' . Similarly, if the same content is considered in multiple categories, it seems this would inflate the number of notices: One report on a comment that has race and gender based discrimination could look like 2 notices in the report.\n\n## Category-specific suggestions:\n\n- \u25cf Animal Welfare\n- \u25cb Unlawful sale of animals: Consider 'Unauthorized'.\n- \u25cf Data protection and privacy violations\n- \u25cb Biometric data breach: As mentioned above, consider other categories of sensitive data per the GDPR. Consider specifying also genetic, health related data.\n- \u25cb Missing processing ground for data: Consider removing. Platform users are generally not aware of what data is collected and even less aware of how it is based on processing ground. We wonder whether this is likely to be a very low number compared to other categories, which will give the impression that data is generally collected lawfully because there are fewer reports.\n- \u25cb Also consider including doxxing as a specific subcategory.\n- \u25cf Risk for public security\n- \u25cb Risk for environmental damage: Consider making this its own superordinate category, with subcategories corresponding to carbon, energy, effects on air, effects on water, effects on animals and plant life, etc.\n- \u25cf Pornography or sexualized content: This category appears to refer to legal content that does not create harm. We disagree that this category fits within the context of the others. Going a step further, we disagree that the government should play a role in regulating legal expressions of sexuality on platforms that permit such content. If reporting of legal sexual content is desired, consider requiring it if it can be classified within the category 'Content in violation of the platform's terms and conditions' - for example, 'Pornography or sexualized content' might replace the 'Nudity' subcategory there.\n- \u25cb Image-based sexual abuse: Consider 'sexual abuse' as the subcategory, removing the requirement that it must be image-based to be reported.\n- \u25cf Self-harm\n- \u25cb This is a type of violence. Consider combining with the Violence category.\n- \u25cf Violence\n\n\n\n- \u25cb Gender-based violence: As mentioned above, consider updating to include violence based in any \"protected\" or 'sensitive' characteristic, such as would include religion, race, etc.\n\nRegarding 6\\_overall\\_figures and 8\\_by\\_language\\_and\\_country Templates\n\n## On 'accuracy rates' and 'error rates':\n\nIn statistical analysis, what you are referring to as 'accuracy rate' is more commonly called 'accuracy'. Reporting accuracy and error rates can be somewhat redundant; error rate = 1-accuracy .\n\nOur Suggestion is, in addition to Accuracy, consider the following metrics to be required, which should be defined based on are more sensitive to the harms of different kinds of errors and can be selected based on the application:\n\n- -sensitivity, recall, hit rate, or true positive rate (TPR)\n- -specificity, selectivity or true negative rate (TNR)\n- -precision or positive predictive value (PPV)\n- -negative predictive value (NPV)\n- -miss rate or false negative rate (FNR)\n- -fall-out or false positive rate (FPR)\n- -false discovery rate (FDR)\n- -false omission rate (FOR)\n\nThese should be disaggregated by various factors, such as via language and sensitive/protected demographic categories.\n\n## Comments on Annex II\n\n## Regarding Section 1.7\n\n## On 'linguistic expertise':\n\nAnnex II notes that 'Article 42(2)(b) prescribes that transparency report shall include the linguistic expertise of the persons carrying out the activities referred to in point (a).' Our Suggestion is to center on language fluency rather than linguistic expertise . The former deals with language proficiency as a function of a person's culture, upbringing, experience, etc. The latter is a field of study concerned with syntactic and morphological structures, etc. Further, for content that is not language-based, such as images, expertise such as cultural knowledge may be more relevant.\n\n\n\nOn 'Measures taken to provide training and assistance to persons in charge of content moderation':\n\nThis should perhaps have a \"quantitative\" complement, which would include compensation and working hours.\n\nOn 'Summary of the content moderation governance structure': This is fundamental to everything else the transparency reporting is looking for; it should perhaps be more highlighted, and with more explicit instructions on how to document this.\n\n## Regarding Section 1.8\n\n## On disaggregated by language:\n\nThis is a critical point that is often overlooked, and we are glad to see it in-place.", + "chunks": [ + { + "headings": [ + "Hugging Face S.A.S" + ], + "markdown": "## Hugging Face S.A.S\n\n9 rue des colonnes,\n\n75002 Paris, France\n\nJanuary 24., 2023\n\n## Hugging Face Feedback on the Digital Services Acts Transparency Reports\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. As part of our activity, Hugging Face provides social features and communication platforms for people interacting with AI systems, including social posts and discussion threads, in addition to hosting the AI systems themselves.\n\n## Comments on 'Draft implementing regulation Ares(2023)8428591.pdf'\n\n## On adding further content:\n\nWe suggest that transparency reports provide readers with relevant information about the recommender systems that manage the content on their platform. Consider requiring documentation in the transparency reports for any relevant parameters of recommendation systems that affect content moderation, as well as information about how the moderation action may affect future recommendations. This could be, for example, at the same level of granularity as when applying recital 70 and article 27(2).\n\n## On 'machine readable' formats for reports:\n\nThe current framing explains, 'to ensure that the transparency reports are machine-readable', transparency reports are required to be in a CSV (comma-separated values) format. We agree this idea is good at a high-level, but in practice csv formats can have issues in appropriate rendering (for example if there is content with additional commas, non ASCII-characters, etc.) This also misses the option to have easily human-readable reports that enable people to interact with the content, such as by printing it out, highlighting, making notes, etc. much easier.\n\n\n\n\n\nOur Suggestion is to support this requirement by creating a script that renders an input .csv transparency report as a human-readable pdf. In a user interaction with this script, users and organizations would use an upload portal with this script embedded to upload their .csv transparency report, which then renders it in a human-readable pdf for the organization to check. If the rendering has errors, this is on the organization's side to fix before their final submission. This removes the burden from the government's side of trying to solve for another organization's errors, and makes it much easier for the organization to find issues. This also ensures human-readability as well as machine-readability.\n\n## Comments on Annex 1 - Templates\n\n## Regarding 2\\_categories\\_names\n\nThe categories bring up several issues. Two common themes emerge across them:\n\n- 1. The distinctions between illegal and legal content. Legal and illegal content are grouped together in this categorization. Sometimes 'illegal' is mentioned specifically; sometimes it is not. We recommend keeping these two distinct. The reasons for this include:\n- a. Not requiring service providers to make the decision as to whether something is technically illegal or not\n- b. Not creating reporting requirements for legal and appropriate content\n- c. Not creating (additional) stigma against legal activities\n\nFurther, for services that are offered in more than one country in the EU, the definition of 'illegal' may vary, causing potential confusion as to which definition of illegal should be considered in the report.\n\nOur Suggestion : If there is a goal for service providers to document illegal/unflawful content explicitly, consider revising the instructions to specify that service providers should report all illegal content they are aware of for each category.\n\n## 2. Domains and stakeholders .\n\n- a. Modality: A focus here is on image content. Consider removing the focus solely on images, or else explicitly including additional modalities where the harms of concern are also expressed, such as audio.\n- b. Stakeholders: In all categories, consider additionally grounding on EU 'protected grounds' and/or 'sensitive personal characteristics'. This includes categories such as those listed as sensitive personal data or subject to\n\n\n\nnon-discrimination by the European Commission or in European non-discrimination law. This would also mean updating the subcategory of 'Gender-based violence' to something like 'Violence based on protected grounds or sensitive characteristics' and the subcategory of 'Biometric data breach' to something like 'Protected personal data breach' .\n\nWe also note that these categories will have overlapping issues, which may make reporting a bit more confusing using this format. For example, 'Negative effects on civic discourse or elections' could include racist misinformation, which would also fall under 'Illegal or harmful speech' . Similarly, if the same content is considered in multiple categories, it seems this would inflate the number of notices: One report on a comment that has race and gender based discrimination could look like 2 notices in the report.\n\n## Category-specific suggestions:\n\n- \u25cf Animal Welfare\n- \u25cb Unlawful sale of animals: Consider 'Unauthorized'.\n- \u25cf Data protection and privacy violations\n- \u25cb Biometric data breach: As mentioned above, consider other categories of sensitive data per the GDPR. Consider specifying also genetic, health related data.\n- \u25cb Missing processing ground for data: Consider removing. Platform users are generally not aware of what data is collected and even less aware of how it is based on processing ground. We wonder whether this is likely to be a very low number compared to other categories, which will give the impression that data is generally collected lawfully because there are fewer reports.\n- \u25cb Also consider including doxxing as a specific subcategory.\n- \u25cf Risk for public security\n- \u25cb Risk for environmental damage: Consider making this its own superordinate category, with subcategories corresponding to carbon, energy, effects on air, effects on water, effects on animals and plant life, etc.\n- \u25cf Pornography or sexualized content: This category appears to refer to legal content that does not create harm. We disagree that this category fits within the context of the others. Going a step further, we disagree that the government should play a role in regulating legal expressions of sexuality on platforms that permit such content. If reporting of legal sexual content is desired, consider requiring it if it can be classified within the category 'Content in violation of the platform's terms and conditions' - for example, 'Pornography or sexualized content' might replace the 'Nudity' subcategory there.\n- \u25cb Image-based sexual abuse: Consider 'sexual abuse' as the subcategory, removing the requirement that it must be image-based to be reported.\n- \u25cf Self-harm\n- \u25cb This is a type of violence. Consider combining with the Violence category.\n- \u25cf Violence\n\n\n\n- \u25cb Gender-based violence: As mentioned above, consider updating to include violence based in any \"protected\" or 'sensitive' characteristic, such as would include religion, race, etc.\n\nRegarding 6\\_overall\\_figures and 8\\_by\\_language\\_and\\_country Templates\n\n## On 'accuracy rates' and 'error rates':\n\nIn statistical analysis, what you are referring to as 'accuracy rate' is more commonly called 'accuracy'. Reporting accuracy and error rates can be somewhat redundant; error rate = 1-accuracy .\n\nOur Suggestion is, in addition to Accuracy, consider the following metrics to be required, which should be defined based on are more sensitive to the harms of different kinds of errors and can be selected based on the application:\n\n- -sensitivity, recall, hit rate, or true positive rate (TPR)\n- -specificity, selectivity or true negative rate (TNR)\n- -precision or positive predictive value (PPV)\n- -negative predictive value (NPV)\n- -miss rate or false negative rate (FNR)\n- -fall-out or false positive rate (FPR)\n- -false discovery rate (FDR)\n- -false omission rate (FOR)\n\nThese should be disaggregated by various factors, such as via language and sensitive/protected demographic categories.\n\n## Comments on Annex II\n\n## Regarding Section 1.7\n\n## On 'linguistic expertise':\n\nAnnex II notes that 'Article 42(2)(b) prescribes that transparency report shall include the linguistic expertise of the persons carrying out the activities referred to in point (a).' Our Suggestion is to center on language fluency rather than linguistic expertise . The former deals with language proficiency as a function of a person's culture, upbringing, experience, etc. The latter is a field of study concerned with syntactic and morphological structures, etc. Further, for content that is not language-based, such as images, expertise such as cultural knowledge may be more relevant.\n\n\n\nOn 'Measures taken to provide training and assistance to persons in charge of content moderation':\n\nThis should perhaps have a \"quantitative\" complement, which would include compensation and working hours.\n\nOn 'Summary of the content moderation governance structure': This is fundamental to everything else the transparency reporting is looking for; it should perhaps be more highlighted, and with more explicit instructions on how to document this.\n\n## Regarding Section 1.8\n\n## On disaggregated by language:\n\nThis is a critical point that is often overlooked, and we are glad to see it in-place.", + "start_char_id": 0, + "end_char_id": 9740 + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2024_EU-Stakeholder-Consultation_Free_Text_Submission.pdf", + "md_content": "\n\n## Template for Submissions to the Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI\n\nAuthors: Yacine Jernite, Lucie-Aim\u00e9e Kaffee\n\n## Working Group 1 - Transparency and copyright-related rules\n\nThe first Working Group focuses on detailing out documentation to downstream providers and the AI Office on the basis of Annexes XI and XII to the AI Act, policies to be put in place to comply with Union law on copyright and related rights, and making publicly available a summary about the training content . Please enter your comments below. This section is ideal for providing insights that may not be captured by the structured input above or provide any contextual information that may be relevant to this Working Group.\n\nTransparency requirements are particularly apt to contribute to more trustworthy GPAI while fostering innovation from a broad set of actors, especially in research institutions, start-ups, and SMEs.\n\nThe content of the data used during the training of a GPAI model data plays a substantial role in ensuring that the technology is robust, rights-respecting, and trustworthy. Information about training data provides necessary context for interpreting a model's results on performance benchmarks, identifying risks related to privacy, understanding the role and contribution of content under copyright in the technology, and making informed choices about deployment in specific contexts; as well as shaping which uses of a model, benign or malicious, will have better or worse performance. At the same time, templates for developers to provide this information need to contend with the scale and diversity of datasets involved. Commercial developers of GPAIs have also argued that excessive details on their full data pipelines could encroach on their trade secrets.\n\nIn order to manage this tension, transparency requirements should focus on the boundaries between the activities of the GPAI developers and external stakeholders. These would include:\n\n- \u25cf The sources of the data used in the various pre-training, fine-tuning, and evaluation datasets - including publicly available data, data obtained through licensing deals, and data elicited from crowd workers or users of the system. The initial conditions in which external data are acquired determine privacy and intellectual property risks, market dynamics around the value of creative works and data, and the ability of external stakeholders to broadly know when they might be entitled to exercise data rights.\n- \u25cf Extensive details on the evaluation datasets for publicly released performance results and other measures of model behaviors to qualify the scope, possible limitations, and\n\n\n\nconstruct validity of the evaluations. Reproducible evaluations on broadly accessible benchmarks should be prioritised whenever possible.\n\n- \u25cf Details on risk mitigation strategies, especially through techniques such as training data filtering or content guidelines-based fine-tuning (sometimes called Constitutional AI) that raise many of the same questions as content moderation at scale, and thus require similar levels of transparency to enable proper governance as are requested of especially VLOPs in the European Union.\n- \u25cf Clarity on the uses of data obtained while running the model, such as documents used as inputs for a served model or user queries.\n\nA focus on the above categories of information helps tailor transparency requirements to specific risks to EU citizens' rights and legitimate interest. It also preserves trade secrets that are more closely tied to the developer's exclusive handling of data. Notably, recent models have been reported to reach increasingly higher performances on benchmark thanks to more elaborate uses of 'self-play' synthetic data and reinforcement learning, which would remain beyond the scope of the proposed transparency requirements.\n\nTransparency-based approaches to GPAI governance present the unique advantage of being well aligned with the ethos of open-source and open science development, and the thriving ecosystem of start-ups and SMEs they support. Indeed, good documentation of proper uses and trust building through maintenance and transparency are instrumental to the success of OSS software. Free and Open Source AI developers and start-ups and SMEs also have unique constraints that should be acknowledged when formulating transparency requirements; by focusing on information the developers have access to and minimising the burden of engaging external certification or maintaining several versions of documentation, among others. For open models that are put on the EU market and shared on open repositories such as GitHub or Hugging Face, requirements should be manageable simply by publishing sufficient documentation alongside the model code or weights, so as to ensure that less-resourced organisations, or organisations that only exist for a limited times - e.g. punctual collaborations such as BigScience - can sustainably meet their demands.\n\nGPAI developers, especially in academic institutions and SMEs, also commonly make use of publicly accessible datasets; since those are accessible and may be analysed by external stakeholders, they should be understood to comply with transparency requirements by default, as long as the developers disclose their use.\n\nWorking Group 2 - Risk identification and assessment measures for systemic risks\n\nThe Code of Practice should help to establish a risk taxonomy of the type and nature of the systemic risks at Union level, including their sources. The second Working Group will focus on\n\n\n\ndetailing the risk taxonomy based on a proposal by the AI Office and identifying and detailing relevant technical risk assessment measures, including model evaluation and adversarial testing. Please enter your comments below. This section is ideal for providing insights that may not be captured by the structured input above or provide any contextual information that may be relevant to this Working Group.\n\nRisk identification and assessment measures are an essential part of enabling foresight and mitigating risks of harms. In order to play that role, evaluations need to be properly scoped, scientifically validated, reproducible, and accessible to all categories of actors, including developers of open models, start-ups, and SMEs developing GPAI models and systems. In particular, the risks of GPAI systems should be defined primarily by the people most likely to be affected with sufficient involvement of external stakeholders to ensure construct validity.\n\nIn order to improve the state of the art in and adoption of risk assessment practices while avoiding fragmentation, the Code of Practice should focus on having developers:\n\n- \u25cf Participate in public efforts involving civil society and public organisations in establishing consensus-driven and scientifically validated risk evaluations, with a view to contributing to international standards. Risks depend on the context of an AI model and its users, therefore it is crucial to include a variety of perspectives in their definition [1].\n- \u25cf Contribute to the development of public benchmarks and risk evaluation methodology according to their size and capacity.\n- \u25cf Collaborate on joint infrastructure for running those evaluations, leveraging open source software and open models as appropriate to facilitate broad participation.\n\nModels that are released on a Free and Open Source (FOS) basis, in particular, should benefit from particular support in meeting these requirements. Requirements for FOS GPAI models should prioritise evaluations that can be run without significant extra computation budgets or involvement of third-party organisations to acknowledge both their default increased transparency (more expensive or involved evaluations may be run by external parties as needed pre-deployment in specific commercial products) and different operational constraints (organisational dynamics for academic actors or collaborations between open developers may not allow for costly external audits).\n\n[1] Wachter: Limitations and Loopholes in the EU AI Act and AI Liability Directives: What This\n\nMeans for the European Union, the United States, and Beyond https://ora.ox.ac.uk/objects/uuid:0525099f-88c6-4690-abfa-741a8c057e00/files/sht24wm314\n\nWorking Group 3 - Risk mitigation measures for systemic risks\n\nThe Code of Practice should be focused on specific risk assessment and mitigation measures. The third Working Group will focus on Identifying and detailing relevant technical risk mitigation\n\n\n\nmeasures, including cybersecurity protection for the general-purpose AI model and the physical infrastructure of the model. Please enter your comments below. This section is ideal for providing insights that may not be captured by the structured input above or provide any contextual information that may be relevant to this Working Group.\n\nRisk mitigation measures for systemic risks should focus on the entire development process, from project design to data curation to final fine-tuning and deployment. While measures based primarily on fine-tuning and deployment guardrails, such as input or output filtering, can be a part of lowering overall risks of especially unintentional misuses, they have been shown to be too brittle to constitute a complete solution, subject to e.g. jailbreaking and unintentional removal through fine-tuning (including through closed APIs) [1,2].\n\nMitigation strategies that focus upstream on the development process, including through training data curation or early-development considerations on the trade-offs inherent in different capabilities, on the other hand, present the dual advantage of being more robust and more accessible to well-intentioned actors training models to be released openly.\n\nIn order to facilitate the development of more robust systemic risk mitigation strategies while preserving the ability of responsible developers to share open models that support research and innovations, we recommend that the Code of Practice focus on:\n\n- \u25cf Prioritising safety by design approaches along the full development chain\n- \u25cf Providing clarity on which risk mitigation approaches should be leveraged by developers or by deployers\n- \u25cf Contributing to open-source tools for risk mitigation that can easily be adopted by actors of all sizes, and can be externally validated, especially where they require trade-offs between different values\n- \u25cf Facilitate skill-sharing and constitution of a set of best practices by encouraging transparent reporting on safety strategies - risk mitigation is in the public interest and should not be treated as an exclusive competitive advantage, sharing information between actors helps spread good practices and identify unforeseen effects faster.\n\n[1] Zou et al.: Universal and Transferable Adversarial Attacks on Aligned Language Models https://llm-attacks.org/ and https://arxiv.org/abs/2307.15043\n\n[2] Lu et al.: Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models https://openaccess.thecvf.com/content/ICCV2023/papers/Lu\\_Set-level\\_Guidance\\_Attack\\_Boos ting\\_Adversarial\\_Transferability\\_of\\_Vision-Language\\_Pre-training\\_Models\\_ICCV\\_2023\\_paper.pdf\n\n\n\n## General Considerations for the drawing-up of the Code of Practice\n\nPlease provide any general considerations that should be taken into account during the drawing-up of the first Code of Practice for providers of general-purpose AI models.\n\nThe Code of Practice represents a unique opportunity to catalyse progress toward more trustworthy GPAI technology across actors. In order to meet those goals while realising the aims outlined in the AI Act of supporting research and innovation, including through free and open source AI, its design process needs to fully take the needs and strengths of all actors into consideration. This includes the most well-resourced developers deploying GPAI systems at scale, start-ups and SMEs who increasingly have access to the resources needed to train their own versions of GPAI that may be better suited to their own use cases (including different languages, domains, modalities), and non-profit and academic researchers who develop open GPAI models that enable much of the research. This research is needed to better understand those systems and ensure that regulation keeps pace with the latest technical developments including e.g. the BigScience [1] and BigCode [2] efforts, AI2 work on the DolMA dataset [3] and OLMO language models [4], and EleutherAI's Pythia models [5].\n\nIn order to ensure that the Code of Practice prioritises measures that are both accessible to all of the above stakeholders and foster more robustness and accountability, we recommend that all working groups:\n\n- \u25cf Prioritise open collaboration with external experts to ensure that measures are driven by needs expressed across stakeholder groups\n- \u25cf Prioritise documentation and transparency as significant contributors to robust and reliable technology\n- \u25cf Focus on measures upstream in the development chain supported by open tools and methods\n- \u25cf Ensure that all measures are properly documented, and that evaluations in particular are reproducible and subject to external scrutiny\n- \u25cf Ensure that start-up and SMEs have a path to compliance that does not depend on extensive third-party certification or audits\n- \u25cf Ensure that requirements for developers of open models are suited to the part of the development chain they have direct control over\n\nSuch an approach will not only support innovation and diverse participation in the evolution of GPAI technology in the EU, but also ensure that the Code of Practice remains relevant as technical conditions evolve over the coming years.\n\n[1] https://bigscience.huggingface.co/\n\n[2] https://www.bigcode-project.org/\n\n[3] https://allenai.github.io/dolma/\n\n[4] https://allenai.org/olmo\n\n\n\n[5] https://huggingface.co/EleutherAI/pythia-6.9b", + "chunks": [ + { + "headings": [ + "Template for Submissions to the Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI" + ], + "markdown": "\n\n## Template for Submissions to the Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI\n\nAuthors: Yacine Jernite, Lucie-Aim\u00e9e Kaffee\n\n## Working Group 1 - Transparency and copyright-related rules\n\nThe first Working Group focuses on detailing out documentation to downstream providers and the AI Office on the basis of Annexes XI and XII to the AI Act, policies to be put in place to comply with Union law on copyright and related rights, and making publicly available a summary about the training content . Please enter your comments below. This section is ideal for providing insights that may not be captured by the structured input above or provide any contextual information that may be relevant to this Working Group.\n\nTransparency requirements are particularly apt to contribute to more trustworthy GPAI while fostering innovation from a broad set of actors, especially in research institutions, start-ups, and SMEs.\n\nThe content of the data used during the training of a GPAI model data plays a substantial role in ensuring that the technology is robust, rights-respecting, and trustworthy. Information about training data provides necessary context for interpreting a model's results on performance benchmarks, identifying risks related to privacy, understanding the role and contribution of content under copyright in the technology, and making informed choices about deployment in specific contexts; as well as shaping which uses of a model, benign or malicious, will have better or worse performance. At the same time, templates for developers to provide this information need to contend with the scale and diversity of datasets involved. Commercial developers of GPAIs have also argued that excessive details on their full data pipelines could encroach on their trade secrets.\n\nIn order to manage this tension, transparency requirements should focus on the boundaries between the activities of the GPAI developers and external stakeholders. These would include:\n\n- \u25cf The sources of the data used in the various pre-training, fine-tuning, and evaluation datasets - including publicly available data, data obtained through licensing deals, and data elicited from crowd workers or users of the system. The initial conditions in which external data are acquired determine privacy and intellectual property risks, market dynamics around the value of creative works and data, and the ability of external stakeholders to broadly know when they might be entitled to exercise data rights.\n- \u25cf Extensive details on the evaluation datasets for publicly released performance results and other measures of model behaviors to qualify the scope, possible limitations, and\n\n\n\nconstruct validity of the evaluations. Reproducible evaluations on broadly accessible benchmarks should be prioritised whenever possible.\n\n- \u25cf Details on risk mitigation strategies, especially through techniques such as training data filtering or content guidelines-based fine-tuning (sometimes called Constitutional AI) that raise many of the same questions as content moderation at scale, and thus require similar levels of transparency to enable proper governance as are requested of especially VLOPs in the European Union.\n- \u25cf Clarity on the uses of data obtained while running the model, such as documents used as inputs for a served model or user queries.\n\nA focus on the above categories of information helps tailor transparency requirements to specific risks to EU citizens' rights and legitimate interest. It also preserves trade secrets that are more closely tied to the developer's exclusive handling of data. Notably, recent models have been reported to reach increasingly higher performances on benchmark thanks to more elaborate uses of 'self-play' synthetic data and reinforcement learning, which would remain beyond the scope of the proposed transparency requirements.\n\nTransparency-based approaches to GPAI governance present the unique advantage of being well aligned with the ethos of open-source and open science development, and the thriving ecosystem of start-ups and SMEs they support. Indeed, good documentation of proper uses and trust building through maintenance and transparency are instrumental to the success of OSS software. Free and Open Source AI developers and start-ups and SMEs also have unique constraints that should be acknowledged when formulating transparency requirements; by focusing on information the developers have access to and minimising the burden of engaging external certification or maintaining several versions of documentation, among others. For open models that are put on the EU market and shared on open repositories such as GitHub or Hugging Face, requirements should be manageable simply by publishing sufficient documentation alongside the model code or weights, so as to ensure that less-resourced organisations, or organisations that only exist for a limited times - e.g. punctual collaborations such as BigScience - can sustainably meet their demands.\n\nGPAI developers, especially in academic institutions and SMEs, also commonly make use of publicly accessible datasets; since those are accessible and may be analysed by external stakeholders, they should be understood to comply with transparency requirements by default, as long as the developers disclose their use.\n\nWorking Group 2 - Risk identification and assessment measures for systemic risks\n\nThe Code of Practice should help to establish a risk taxonomy of the type and nature of the systemic risks at Union level, including their sources. The second Working Group will focus on\n\n\n\ndetailing the risk taxonomy based on a proposal by the AI Office and identifying and detailing relevant technical risk assessment measures, including model evaluation and adversarial testing. Please enter your comments below. This section is ideal for providing insights that may not be captured by the structured input above or provide any contextual information that may be relevant to this Working Group.\n\nRisk identification and assessment measures are an essential part of enabling foresight and mitigating risks of harms. In order to play that role, evaluations need to be properly scoped, scientifically validated, reproducible, and accessible to all categories of actors, including developers of open models, start-ups, and SMEs developing GPAI models and systems. In particular, the risks of GPAI systems should be defined primarily by the people most likely to be affected with sufficient involvement of external stakeholders to ensure construct validity.\n\nIn order to improve the state of the art in and adoption of risk assessment practices while avoiding fragmentation, the Code of Practice should focus on having developers:\n\n- \u25cf Participate in public efforts involving civil society and public organisations in establishing consensus-driven and scientifically validated risk evaluations, with a view to contributing to international standards. Risks depend on the context of an AI model and its users, therefore it is crucial to include a variety of perspectives in their definition [1].\n- \u25cf Contribute to the development of public benchmarks and risk evaluation methodology according to their size and capacity.\n- \u25cf Collaborate on joint infrastructure for running those evaluations, leveraging open source software and open models as appropriate to facilitate broad participation.\n\nModels that are released on a Free and Open Source (FOS) basis, in particular, should benefit from particular support in meeting these requirements. Requirements for FOS GPAI models should prioritise evaluations that can be run without significant extra computation budgets or involvement of third-party organisations to acknowledge both their default increased transparency (more expensive or involved evaluations may be run by external parties as needed pre-deployment in specific commercial products) and different operational constraints (organisational dynamics for academic actors or collaborations between open developers may not allow for costly external audits).\n\n[1] Wachter: Limitations and Loopholes in the EU AI Act and AI Liability Directives: What This\n\nMeans for the European Union, the United States, and Beyond https://ora.ox.ac.uk/objects/uuid:0525099f-88c6-4690-abfa-741a8c057e00/files/sht24wm314\n\nWorking Group 3 - Risk mitigation measures for systemic risks\n\nThe Code of Practice should be focused on specific risk assessment and mitigation measures. The third Working Group will focus on Identifying and detailing relevant technical risk mitigation\n\n\n\nmeasures, including cybersecurity protection for the general-purpose AI model and the physical infrastructure of the model. Please enter your comments below. This section is ideal for providing insights that may not be captured by the structured input above or provide any contextual information that may be relevant to this Working Group.\n\nRisk mitigation measures for systemic risks should focus on the entire development process, from project design to data curation to final fine-tuning and deployment. While measures based primarily on fine-tuning and deployment guardrails, such as input or output filtering, can be a part of lowering overall risks of especially unintentional misuses, they have been shown to be too brittle to constitute a complete solution, subject to e.g. jailbreaking and unintentional removal through fine-tuning (including through closed APIs) [1,2].\n\nMitigation strategies that focus upstream on the development process, including through training data curation or early-development considerations on the trade-offs inherent in different capabilities, on the other hand, present the dual advantage of being more robust and more accessible to well-intentioned actors training models to be released openly.\n\nIn order to facilitate the development of more robust systemic risk mitigation strategies while preserving the ability of responsible developers to share open models that support research and innovations, we recommend that the Code of Practice focus on:\n\n- \u25cf Prioritising safety by design approaches along the full development chain\n- \u25cf Providing clarity on which risk mitigation approaches should be leveraged by developers or by deployers\n- \u25cf Contributing to open-source tools for risk mitigation that can easily be adopted by actors of all sizes, and can be externally validated, especially where they require trade-offs between different values\n- \u25cf Facilitate skill-sharing and constitution of a set of best practices by encouraging transparent reporting on safety strategies - risk mitigation is in the public interest and should not be treated as an exclusive competitive advantage, sharing information between actors helps spread good practices and identify unforeseen effects faster.\n\n[1] Zou et al.: Universal", + "char_slice": [ + 0, + 10894 + ] + }, + { + "headings": [ + "## Template for Submissions to the Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI", + "## Working Group 1 - Transparency and copyright-related rules" + ], + "markdown": "s) [1,2].\n\nMitigation strategies that focus upstream on the development process, including through training data curation or early-development considerations on the trade-offs inherent in different capabilities, on the other hand, present the dual advantage of being more robust and more accessible to well-intentioned actors training models to be released openly.\n\nIn order to facilitate the development of more robust systemic risk mitigation strategies while preserving the ability of responsible developers to share open models that support research and innovations, we recommend that the Code of Practice focus on:\n\n- \u25cf Prioritising safety by design approaches along the full development chain\n- \u25cf Providing clarity on which risk mitigation approaches should be leveraged by developers or by deployers\n- \u25cf Contributing to open-source tools for risk mitigation that can easily be adopted by actors of all sizes, and can be externally validated, especially where they require trade-offs between different values\n- \u25cf Facilitate skill-sharing and constitution of a set of best practices by encouraging transparent reporting on safety strategies - risk mitigation is in the public interest and should not be treated as an exclusive competitive advantage, sharing information between actors helps spread good practices and identify unforeseen effects faster.\n\n[1] Zou et al.: Universal and Transferable Adversarial Attacks on Aligned Language Models https://llm-attacks.org/ and https://arxiv.org/abs/2307.15043\n\n[2] Lu et al.: Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models https://openaccess.thecvf.com/content/ICCV2023/papers/Lu\\_Set-level\\_Guidance\\_Attack\\_Boos ting\\_Adversarial\\_Transferability\\_of\\_Vision-Language\\_Pre-training\\_Models\\_ICCV\\_2023\\_paper.pdf\n\n\n\n## General Considerations for the drawing-up of the Code of Practice\n\nPlease provide any general considerations that should be taken into account during the drawing-up of the first Code of Practice for providers of general-purpose AI models.\n\nThe Code of Practice represents a unique opportunity to catalyse progress toward more trustworthy GPAI technology across actors. In order to meet those goals while realising the aims outlined in the AI Act of supporting research and innovation, including through free and open source AI, its design process needs to fully take the needs and strengths of all actors into consideration. This includes the most well-resourced developers deploying GPAI systems at scale, start-ups and SMEs who increasingly have access to the resources needed to train their own versions of GPAI that may be better suited to their own use cases (including different languages, domains, modalities), and non-profit and academic researchers who develop open GPAI models that enable much of the research. This research is needed to better understand those systems and ensure that regulation keeps pace with the latest technical developments including e.g. the BigScience [1] and BigCode [2] efforts, AI2 work on the DolMA dataset [3] and OLMO language models [4], and EleutherAI's Pythia models [5].\n\nIn order to ensure that the Code of Practice prioritises measures that are both accessible to all of the above stakeholders and foster more robustness and accountability, we recommend that all working groups:\n\n- \u25cf Prioritise open collaboration with external experts to ensure that measures are driven by needs expressed across stakeholder groups\n- \u25cf Prioritise documentation and transparency as significant contributors to robust and reliable technology\n- \u25cf Focus on measures upstream in the development chain supported by open tools and methods\n- \u25cf Ensure that all measures are properly documented, and that evaluations in particular are reproducible and subject to external scrutiny\n- \u25cf Ensure that start-up and SMEs have a path to compliance that does not depend on extensive third-party certification or audits\n- \u25cf Ensure that requirements for developers of open models are suited to the part of the development chain they have direct control over\n\nSuch an approach will not only support innovation and diverse participation in the evolution of GPAI technology in the EU, but also ensure that the Code of Practice remains relevant as technical conditions evolve over the coming years.\n\n[1] https://bigscience.huggingface.co/\n\n[2] https://www.bigcode-project.org/\n\n[3] https://allenai.github.io/dolma/\n\n[4] https://allenai.org/olmo\n\n\n\n[5] https://huggingface.co/EleutherAI/pythia-6.9b", + "char_slice": [ + 9510, + 14071 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2024_EU-Stakeholder-Consultation_Survey_Questions.pdf", + "md_content": "\n\n## Hugging Face Comments on the EU AI Office's Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI\n\nResponses by: Lucie-Aim\u00e9e Kaffee, Yacine Jernite, Bruna Trevelin\n\n## About Hugging Face\n\nHugging Face is a community-oriented company working to democratise good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analysing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. Part of those activities include releasing open-source tools to support other actors developing their own models, as well as collaborating on new state-of-the-art GPAI models such as StarCoder2, the highest-performing fully-open code model to date. Its training dataset, The Stack, exemplifies a working opt-out methodology, and its governance card provides an implementation of the goals of the Code of Practice: https://shorturl.at/lBbAa.\n\n## Section 1. General-purpose AI models: transparency and copyright-related rules\n\nA. Information and documentation by general-purpose AI model providers to providers of AI systems\n\nProviders of general-purpose AI models have a particular role and responsibility along the AI value chain, as the models they provide may form the basis for a range of downstream systems, often provided by downstream providers that necessitate a good understanding of the models and their capabilities, both to enable the integration of such models into their products, and to fulfil their obligations under the AI Act or other regulations. Therefore, model providers should draw up, keep up-to-date and make available information and documentation to providers of AI systems who intend to integrate the general-purpose AI model into their AI system. Widely adopted documentation practices include model cards and data sheets.\n\n\n\nA minimal set of elements of information and documentation by general-purpose AI model providers to providers of AI systems is already set out in AI Act Annex XII.\n\n- 1. In the current state of the art , for which elements of information and documentation by general-purpose AI model providers to providers of AI systems do practices exist that, in your view, achieve the above-mentioned purpose ?\n\nFrom the list below following AI Act Annex XII, please select all relevant elements. If such practices exist, please provide links to relevant material substantiating your reply, such as model cards, data sheets or templates.\n\nA general description of the general-purpose AI model including:\n\nThe tasks that the model is intended to perform and the type and nature of AI systems into which it can be integrated;\n\nThe acceptable use policies applicable;\n\nThe date of release and methods of distribution;\n\nHow the model interacts, or can be used to interact, with hardware or software that is not part of the model itself, where applicable;\n\nThe versions of relevant software related to the use of the general-purpose AI model, where applicable;\n\nThe architecture and number of parameters;\n\nThe modality (e.g., text, image) and format of inputs and outputs;\n\nThe licence for the model.\n\nA description of the elements of the model and of the process for its development, including:\n\nThe technical means (e.g., instructions for use, infrastructure, tools) required for the general-purpose AI model to be integrated into AI systems;\n\nThe modality (e.g., text, image, etc.) and format of the inputs and outputs and their maximum size (e.g., context window length, etc.);\n\nInformation on the data used for training, testing and validation, where applicable, including the type and provenance of data and curation methodologies.\n\nLinks to relevant material\n\nModel cards on Hugging Face: https://huggingface.co/docs/hub/en/model-cards Example of usage of model cards: https://huggingface.co/meta-llama/Meta-Llama-3-8B (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means)\n\n\n\nhttps://huggingface.co/CohereForAI/c4ai-command-r-v01 (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means)\n\nExample usage of the model cards: https://huggingface.co/HuggingFaceM4/idefics2-8b (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means, dataset for training (see below as OBELICS)) Usage policy (RAIL and OpenRAIL):\n\nhttps://www.licenses.ai/blog/2022/8/18/naming-convention-of-responsible-ai-licenses https://huggingface.co/spaces/bigscience/license\n\nUse of RAIL: https://huggingface.co/bigscience/bloom\n\nTool Use With Command R: https://cohere.com/blog/tool-use-with-command-r Dataset cards on Hugging Face: https://huggingface.co/docs/hub/en/datasets-cards https://github.com/huggingface/datasets/blob/main/templates/README\\_guide.md\n\nExample of usage of dataset cards:\n\nhttps://huggingface.co/datasets/HuggingFaceM4/OBELICS\n\n2. Beyond the minimal set of elements listed in the previous question, are there other elements that should be included in information and documentation by general-purpose AI model providers to providers of AI systems to achieve the above-mentioned purpose?\n\nYes\n\nNo\n\nI don't know\n\nLinks to relevant material\n\nWhere applicable, ethical charters and other considerations made in the creation of the model: https://bigscience.huggingface.co/blog/bigscience-ethical-charter https://huggingface.co/blog/ethical-charter-multimodal Governance cards: https://arxiv.org/abs/2312.03872\n\nhttps://huggingface.co/datasets/bigcode/governance-card\n\n## B. Technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities\n\nIn addition to the provision of information on the general-purpose AI model for its usage by the downstream providers, technical documentation should be prepared and kept up to date by the general-purpose AI model provider for the purpose of making it available, upon request, to the AI Office and the national competent authorities.\n\nA minimal set of elements of such technical documentation of the general-purpose AI model to be made available by providers, upon request, to the AI Office and the national competent authorities is already set out in AI Act Annex XI.\n\n\n\n- 3. In the current state of the art , for which elements of documentation by general-purpose AI model providers do practices exist that, in your view, provide a necessary level of information for the above-mentioned purpose ?\n\nFrom the list below following AI Act Annex XI, please select all relevant elements. If such practices exist, please provide links to relevant material substantiating your reply, such as model cards, data sheets or templates.\n\nA general description of the general-purpose AI model including:\n\nThe tasks that the model is intended to perform and the type and nature of AI systems into which it can be integrated;\n\nThe acceptable use policies applicable;\n\nThe date of release and methods of distribution;\n\nThe architecture and number of parameters;\n\nThe modality (e.g., text, image) and format of inputs and outputs;\n\nThe licence.\n\nA description of the elements of the model, and relevant information of the process for the development, including:\n\nThe technical means (e.g., instructions for use, infrastructure, tools) required for the general-purpose AI model to be integrated into AI systems;\n\nThe design specifications of the model and training process, including training methodologies and techniques, the key design choices including the rationale and assumptions made; what the model is designed to optimise for and the relevance of the different parameters, as applicable;\n\nInformation on the data used for training, testing and validation, where applicable, including the type and provenance of data and curation methodologies (e.g. cleaning, filtering etc), the number of data points, their scope and main characteristics; how the data was obtained and selected as well as all other measures to detect the unsuitability of data sources and methods to detect identifiable biases, where applicable;\n\nthe computational resources used to train the model (e.g. number of floating point operations), training time, and other relevant details related to the training;\n\nknown or estimated energy consumption of the model.\n\nAdditional information to be provided by providers of general-purpose AI models with systemic risk:\n\n\n\nA detailed description of the evaluation strategies, including evaluation results, on the basis of available public evaluation protocols and tools or otherwise of other evaluation methodologies. Evaluation strategies shall include evaluation criteria, metrics and the methodology on the identification of limitations;\n\nWhere applicable, a detailed description of the measures put in place for the purpose of conducting internal and/or external adversarial testing (e.g., red teaming), model adaptations, including alignment and fine-tuning;\n\nWhere applicable, a detailed description of the system architecture explaining how software components build or feed into each other and integrate into the overall processing;\n\nLinks to relevant material\n\nExample of usage of model cards: https://huggingface.co/meta-llama/Meta-Llama-3-8B (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means)\n\nhttps://huggingface.co/CohereForAI/c4ai-command-r-v01 (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means)\n\nExample usage of the model cards: https://huggingface.co/HuggingFaceM4/idefics2-8b (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means, dataset for training (see below as IDEFICS)) Documentation of SmolLM, including experiments, evaluation, and description of the training of the model: https://huggingface.co/blog/smollm\n\nOpenLLM leaderboard, evaluation on standard evaluation metrics:\n\nhttps://huggingface.co/spaces/open-llm-leaderboard/open\\_llm\\_leaderboard Existing events for red teaming:\n\nhttps://aivillage.org/generative%20red%20team/generative-red-team-2/\n\nThe Environmental Impacts of AI - Primer:\n\nhttps://huggingface.co/blog/sasha/ai-environment-primer\n\nEnergy Star Ratings for AI Models:\n\nhttps://huggingface.co/blog/sasha/energy-star-ai-proposal\n\n4. Beyond the minimal set of elements listed in the previous question, are there other elements that should, in your view, be included in technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities?\n\nYes\n\nNo\n\nI don't know\n\nLinks to relevant material\n\n\n\nSocial impact evaluations, see Evaluating the Social Impact of Generative AI Systems in Systems and Society https://arxiv.org/abs/2306.05949\n\n## C. Policy to respect Union copyright law\n\nThe AI Act requires providers of general-purpose AI models to put in place a policy to comply with Union law on copyright and related rights, and in particular to identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3) of Directive (EU) 2019/790.\n\n5. What are, in your view, the main elements that need to be included in the policy that providers of general-purpose AI models have to put in place to comply with Union law on copyright and related rights, as required by the AI Act?\n\nPlease select all relevant options from the list of options suggested below. If selected, please elaborate further on the content of the measures and provide links to any good practices you are aware of.\n\nAllocation of responsibility within the organisation for the implementation and monitoring of compliance with the policy and the measures therein;\n\nMeasures to identify and comply with the rights reservation from the text and data mining exception pursuant to Article 4(3) of Directive (EU) 2019/790;\n\nMeasures to obtain the authorisation from right holders, where applicable;\n\nMeasures to detect and remove collected copyright protected content for which rights reservation from the text and data mining exception has been expressed pursuant to Article 4(3) of Directive (EU) 2019/790;\n\nMeasures to prevent the generation, in the outputs of the model, of copyright infringing content;\n\nMeans for contact with rightsholders;\n\nMeasures for complaint handling from rightsholders;\n\nOther\n\nI don't know\n\nYour comments\n\n700 character(s) maximum\n\nMeasures to comply with rights reservation need to be accessible to SMEs and developers of open models, including actors leveraging public datasets. This requires ensuring that\n\n\n\nopt-outs are publicly accessible, machine-readable with a known protocol. Such an approach also benefits rights holders more than fragmented protocols from GPAI developers. Measures to prevent the generation of copyright infringing content should also be cognizant of open developers and of the current lack of accepted definition of 'substantial similarity' for model generations. A sufficiently detailed training data summary template is more accessible to open developers than under-defined output filtering requirements.\n\nLinks to relevant material\n\nOpen Future: Considerations For Implementing Rightholder Opt-Outs By AI Model Developers https://openfuture.eu/wp-content/uploads/2024/05/240516considerations\\_of\\_opt-out\\_com pliance\\_policies.pdf\n\n- 6. How can, in your view, the policy to be put in place by providers of general-purpose AI models to comply with Union copyright law ensure that providers of those models comply with the existing solutions for the expression of the text and data mining rights reservation , pursuant to Article 4(3) of Directive (EU) 2019/790?\n\nPlease explain how this can be achieved and specify from the list below the state-of-the-art technologies you are aware of to identify and comply with the right reservations expressed by rightsholders, providing further information and examples.\n\nTechnologies/tools that identify right reservations at the website/domain level\n\nTechnologies/tools that identify right reservations at work level\n\nTechnologies/tools that aggregate the expression of right reservations\n\nOther\n\nI don't know\n\nYour comments\n\n700 character(s) maximum\n\nThe policy should ensure that the technologies and standards used for expressing opt-outs are widely and freely accessible, allowing rights holders and AI developers to engage without barriers that could adversely impact competition across developers. The opt-out process must be straightforward and user-friendly, avoiding the need for legal expertise and supporting automated processing to streamline compliance with the text and data mining rights reservation under Article 4(3) of Directive (EU) 2019/790. Additionally, tools to aggregate and manage opt-out requests are needed, as current solutions are insufficient and fail to prevent data from being crawled despite opt-outs.\n\nLinks to relevant material\n\n\n\nDomain-level\n\nAmI in The Stack? App https://huggingface.co/spaces/bigcode/in-the-stack\n\nHugging Face has integrated the Spawning HaveIBeenTrained API with the Hub for datasets\n\nthat have an image\\_url field, e.g. see the report widget on the right in\n\nhttps://huggingface.co/datasets/kakaobrain/coyo-700m\n\nCommonCrawl, which is a dataset of web crawl data, often used as the base for training data for models, respects opt-outs via robot.txt: https://commoncrawl.org/ccbot\n\nrobots.txt; ai.txt https://spawning.ai/ai-txt\n\nTDM Reservation protocol (TDMRep),\n\nhttps://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240202/\n\nWork-level\n\nCoalition for Content Provenance and Authenticity (C2PA)), https://c2pa.org/\n\nInternational Standard Content Code (ISCC), http://iscc.codes/ Spawning.ai's Have I been trained?, haveibeentrained.com\n\nOther relevant material\n\nAnalysis of opt-outs: https://www.dataprovenance.org/Consent\\_in\\_Crisis.pdf\n\n- D. Summary about content used for the training of general-purpose AI models The AI Act requires providers to draw up and make publicly available a sufficiently detailed summary about the content used for training of the general-purpose AI model, according to a template provided by the AI Office. While due account should be taken of the need to protect trade secrets and confidential business information, the summary is to be generally comprehensive in its scope instead of technically detailed to facilitate parties with legitimate interests, including copyright holders, to exercise and enforce their rights under Union law. The template that should be drafted by the AI Office for the sufficiently detailed summary should be simple, effective, and allow providers to provide the required summary in narrative form.\n- 7. What are in your view the categories of information sources that should be presented in the summary to ensure that it comprehensively describes the main sources of data used for the training of the general-purpose AI model?\n\nFrom the list below, please select all options that you consider relevant.\n\nPublic/ open data repositories\n\nContent/data publicly available online (e.g. scraped from the internet)\n\nProprietary data generated by the provider\n\nUser-generated data obtained through the services or products provided by the provider\n\nCopyright protected content licensed by rightsholders\n\n\n\nOther data/content or data sets acquired from third parties (e.g. licensed proprietary databases, data acquired from datahubs, public interest institutions such as libraries etc.)\n\nSynthetically generated data\n\nOther\n\nI don't know\n\nIf selected, please specify the level of granularity/detail for each of the selected options , keeping in mind that AI Act requires the summary to be comprehensive instead of technically detailed and provided in a narrative form to facilitate parties with legitimate interests, including rightsholders, to exercise and enforce their rights under Union law, while taking due account of the need to protect providers' trade secrets and confidential business information. If additional categories should be considered, please specify them and the level of granularity/detail. You can motivate your choice and provide links to any good practices.\n\n700 character(s) maximum\n\nInformation would ideally be detailed enough for any rights holder to understand how their data contributes to the GPAI system. Information needs to be detailed enough to ensure that broad trends in the use of personal data, creative works, and representations of specific groups affect the system are legible to external stakeholders including journalists and policymakers. This level of granularity requires different definitions for different categories, e.g. a list of URLs for public data repositories, the top domains by content in a web crawl, or the respective sizes and status of intermediaries (e.g. publisher, platform, archive) for different modalities of licensed content.\n\nLinks to relevant material\n\nOpen Future's Towards robust training data transparency:\n\nhttps://openfuture.eu/publication/towards-robust-training-data-transparency/ What's in my big data: https://arxiv.org/abs/2310.20707 https://wimbd.apps.allenai.org/\n\nOn training data memorisation and PII data leakage:\n\nLukas et al.: Analyzing Leakage of Personally Identifiable Information in Language Models https://arxiv.org/pdf/2302.00539\n\nCarlini et al.: Extracting Training Data from Large Language Models\n\nhttps://arxiv.org/abs/2012.07805\n\n- 8. In your view, should the summary include one or more of the following characteristics/information about the data used for the training/of the general-purpose AI model in order to facilitate parties with legitimate interests, including copyright holders, to enforce their rights under Union law?\n\n\n\nPlease select all relevant options from the list of options suggested below. If selected, please explain your choice and provide links to any good practices.\n\nModalities / type of data (text, images, videos, music, etc);\n\nNature of the data (personal, non-personal or mixed);\n\nTime of acquisition/collection of the data;\n\nData range of the data (e.g. time span), including date cutoffs\n\nIn case of data scraped from the internet, information about the crawlers used;\n\nInformation about diversity of the data (for example linguistic, geographical, demographic diversity);\n\nPercentage of each of the main data sources to the overall training/fine-tuning;\n\nLegal basis for the processing under Union copyright law and data protection law, as applicable;\n\nMeasures taken to address risks to parties with legitimate interests (e.g. measures to identify and respect opt-out from the text and data mining exception, respect data protection and address privacy risks, bias, generation of illegal or harmful content;\n\nOther\n\nI don't know\n\nYour comments\n\n700 character(s) maximum\n\nAll categories outlined have direct bearing on EU regulations including intellectual property, anti-discrimination, and personal data protections. While they can easily be documented for newly curated datasets, retroactive information gathering is more challenging. Requirements should reflect this by permitting less detailed documentation on pre-existing datasets, as long as they are properly identified and updates are fully documented. Requirements should be more limited for open access datasets and SMEs to reflect their inherent transparency and different resources respectively. The legal basis dimension can be the most difficult to assess and would benefit from harmonised interpretations.\n\nLinks to relevant material\n\nOpen Future's Towards robust training data transparency:\n\nhttps://openfuture.eu/publication/towards-robust-training-data-transparency/\n\n- 9. Considering the purpose of the summary to provide meaningful information to facilitate the exercise of the rights of parties with legitimate interests under Union law, while taking due account of the need to respect business confidentiality and trade secrets of providers, what types of information in your view are justified not to be disclosed in the summary as being not necessary or disproportionate for its purpose described above?\n\n\n\n## 700 character(s) maximum\n\nThe performance of a GPAI system depends not only on the source of the data but on the specific learning objectives, data processing, and increasingly on techniques of 'self-play' and RLHF whose specifics remain beyond the scope of the proposed summary. The latest release of the OpenAI o1 system in particular exemplifies the role of these advanced techniques in gaining a competitive advantage. Therefore, focusing information requirements on data that has external rights holders can help support the rights of EU citizens and support a healthy and sustainable data ecosystem without requiring GPAI developers to reveal their unique contributions to the performance of their products.\n\n## Section 2. General-purpose AI models with systemic risk: risk taxonomy, assessment and mitigation\n\n## A. Risk taxonomy\n\nSome general-purpose AI models could pose systemic risks, which should be understood to increase with model capabilities and model reach and can arise along the entire lifecycle of the model. 'Systemic risks' refer to risks that are specific to the high-impact capabilities of general-purpose AI models (matching or exceeding the capabilities of the most advanced general-purpose AI models); have a significant impact on the Union market due to their reach; or are due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or society as a whole, that can be propagated at scale across the value chain (AI Act Article 3(65)). Systemic risks are influenced by conditions of misuse, model reliability, model fairness and model security, the level of autonomy of the model, its access to tools, novel or combined modalities, release and distribution strategies, the potential to remove guardrails and other factors. The Code of Practice should help to establish a risk taxonomy of the type and nature of the systemic risks at Union level, including their sources. The Code should take into account international approaches.\n\n10. Do you consider the following list of systemic risks based on AI Act Recital 110 and international approaches to be comprehensive to inform a taxonomy of systemic risks from general-purpose AI models? If additional risks should be considered in your view, please specify.\n\n## Systemic risk from model malfunctions\n\n- \u25cf Harmful bias and discrimination: The ways in which models can give rise to harmful bias and discrimination with risks to individuals, communities or societies.\n- \u25cf Misinformation and harming privacy: The dissemination of illegal or false content and facilitation of harming privacy with threats to democratic values and human rights.\n\n\n\n- \u25cf Major accidents: Risks in relation to major accidents and disruptions of critical sectors, that a particular event could lead to a chain reaction with considerable negative effects that could affect up to an entire city, an entire domain activity or an entire community.\n- \u25cf Loss of control: Unintended issues of control relating to alignment with human intent, the effects of interaction and tool use, including for example the capacity to control physical systems, 'self-replicating' or training other models.\n\n## Systemic risk from malicious use\n\n- \u25cf Disinformation: The facilitation of disinformation and manipulation of public opinion with threats to democratic values and human rights.\n- \u25cf Chemical, biological, radiological, and nuclear risks: Dual-use science risks related to ways in which barriers to entry can be lowered, including for weapons development, design acquisition, or use.\n- \u25cf Cyber offence: Risks related to offensive cyber capabilities such as the ways in which vulnerability discovery, exploitation, or operational use can be enabled.\n\n## Other systemic risks, with reasonably foreseeable negative effects on\n\n- \u25cf public health\n- \u25cf safety\n- \u25cf democratic processes\n- \u25cf public and economic security\n- \u25cf fundamental rights\n- \u25cf the society as a whole.\n\nYes, this list of systemic risks is comprehensive.\n\nFurther or more specific systemic risks should be considered.\n\nThe list is comprehensive, as long as the risks currently listed under other systemic risks are prioritised alongside the other two categories; in particular safety, democratic processes, public and economic security, fundamental rights have been at the forefront of recent discussions such as the use of AI to decide whether unemployed workers get benefits [1] or access to information about democratic processes [2].\n\n[1]\n\nhttps://gizmodo.com/googles-ai-will-help-decide-whether-unemployed-workers-get-benefits-2 000496215\n\n[2]\n\nhttps://www.latimes.com/opinion/story/2024-03-08/primaries-voting-elections-ai-misinforma tion-plaforms-chatgpt\n\n\n\n- 11. What are in your view sources of systemic risks that may stem from the development, the placing on the market, or the use of general-purpose AI models? Systemic risks should be understood to increase with model capabilities and model reach.\n\nPlease select all relevant elements from the list. If additional sources should be considered, please specify. You can also provide details on any of the sources or other considerations.\n\nLevel of autonomy of the model: The degree to which a general-purpose AI model has the capability to autonomously interact with the world, plan ahead, and pursue goals.\n\nAdaptability to learn new, distinct tasks: The capability of a model to independently acquire skills for different types of tasks.\n\nAccess to tools: A model gaining access to tools, such as databases or web browsers, and other affordances in its environment.\n\nNovel or combined modalities: Modalities a model can process as input and generate as output, such as text, images, video, audio or robotic actions.\n\nRelease and distribution strategies: The way a model is released, such as under free and open-source license, or otherwise made available on the market.\n\nPotential to remove guardrails: The ability to bypass or disable pre-defined safety constraints or boundaries set up to ensure a model operates within desired parameters and avoids unintended or harmful outcomes.\n\nAmount of computation used for training the model: Cumulative amount of computation ('compute') used for model training measured in floating point operations as one of the relevant approximations for model capabilities.\n\nData set used for training the model: Quality or size of the data set used for training the model as a factor influencing model capabilities.\n\nOther\n\nPlease specify\n\n700 character(s) maximum\n\nThe content of the training datasets is a strong factor for most of the systemic risks considered. The prevalence of personal data, hate speech, information pertaining to capabilities of concern for the model, or known misinformation in the training dataset, for example, have direct bearing on the model behaviour; often more so than the results of a model on general performance benchmarks.\n\nYour comments\n\n700 character(s) maximum\n\nDistribution strategies are important but equating open-source with higher risk is misguided. There is no evidence suggesting that open-source models are riskier than closed-source ones\n\n\n\n[https://crfm.stanford.edu/open-fms/]. In fact, closed models are often more accessible due to user-friendly interfaces and cheap commercial access for a large user base [https://tinyurl.com/yf2yu8zv]; which compounds with the frailty of deployment-level guardrails [https://llm-attacks.org/].\n\nOpen-source releases offer unique benefits and risk strategies especially at the ecosystem level [https://huggingface.co/blog/ethics-soc-3] Limitations of computation thresholds should be acknowledged [https://tinyurl.com/mrb4sjzc ]\n\n## B. Risk identification and assessment measures\n\nIn light of potential systemic risks, the AI Act puts in place effective rules and oversight. Providers of general-purpose AI models with systemic risks should continuously assess and mitigate systemic risks. The Code of Practice should be focused on specific risk assessment measures for general-purpose AI models with systemic risk. Following the risk taxonomy, appropriate measures could be applied to assess different systemic risks, tailored to each specific type and nature of risk, including their sources. In addition to further risk assessment measures which will be detailed out in the Code of Practice, the AI Act requires providers to perform the necessary model evaluations, in particular prior to its first placing on the market, including conducting and documenting adversarial testing of the model, also, as appropriate, through internal or independent external testing. The following concerns technical risk assessment measures, including model evaluation and adversarial testing. This is in line with the focus of the Code of Practice Working Group 2 'Risk identification and assessment measures for systemic risks'.\n\nQuestion12. How can the effective implementation of risk assessment measures reflect differences in size and capacity between various providers such as SMEs and start-ups?\n\n700 character(s) maximum\n\nRisk assessment measures should be developed in collaboration between actors of all sizes and allowing participation of affected stakeholders outside of the largest developers. This both ensures that the more meaningful risks are prioritised and allows smaller companies to follow established and scientifically validated standards, rather than having each actor reproduce work or prioritise risks at their own discretion without input from the most affected parties. This ensures that SMEs and start-ups can fully participate in responsible development.\n\n- 13. In the current state of the art , which specific risk assessment measures should, in your view, general-purpose AI model providers take to effectively assess systemic risks along the entire model lifecycle, in addition to evaluation and testing?\n\nPlease indicate to what extent you agree that providers should take the risk assessment\n\n\n\nmeasures from the list. You can add additional measures and provide details on any of the measures, such as what is required for measures to be effective in practice.\n\n| Potential risk assessment measures | Strongly agree | Somewh at agree | Neither agree nor disagree | Disagree | I don't know |\n|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------------|------------------------------|------------|----------------|\n| Determining risk thresholds and risk tolerance , incl. acceptable levels of risks and capabilities for model development and deployment, and respective quantification of risk severity and probability | | X | | | |\n| Forecasting model capabilities and risks before and during model development | | | X | | |\n| Continuous monitoring for emergence of risks , including data from users, relevant stakeholders, incident databases or similar | | | X | | |\n| Determining effectiveness of risk mitigation measures | | X | | | |\n| Safety cases to demonstrate that the model does not exceed maximum risk thresholds | | X | | | |\n| Aggregate risk assessment before model development | | X | | | |\n| Aggregate risk assessment before model deployment | | X | | | |\n| Aggregate risk assessment along the entire model lifecycle | | X | | | |\n\n\n\n\n\nThird-party involvement in risk assessment, for example, related to inspections of training data, models or internal governance\n\nX\n\nYour comments\n\n700 character(s) maximum\n\nThird-party involvement in risk assessment is essential, particularly for inspections of training data, models, or internal governance. In order for this feedback to be meaningful, the entire development cycle of GPAIs should be sufficiently meaningfully documented, allowing external stakeholders to direct their attention to the most relevant development choices. This transparency may be supported through documents like governance cards [https://arxiv.org/abs/2312.03872] to allow continuous external input.\n\n- 14. Please provide links to relevant material on state-of-the-art risk assessment measures, such as model cards, data sheets, templates or other publications.\n\nEvaluating the Social Impact of Generative AI Systems in Systems and Society https://arxiv.org/abs/2306.05949\n\nBias Preservation in Machine Learning: The Legality of Fairness Metrics Under EU\n\nNon-Discrimination Law https://researchrepository.wvu.edu/wvlr/vol123/iss3/4/ The BigCode Project Governance Card: https://arxiv.org/abs/2312.03872\n\n15. In the current state of the art, which specific practices related to model evaluations should, in your view, general-purpose AI model providers take with a view to identifying and mitigating systemic risks?\n\nModel evaluations can include various techniques, such as benchmarks and automated tests, red teaming and adversarial testing including stress testing and boundary testing, white-box evaluations with model explanation and interpretability techniques, and sociotechnical evaluations like field testing, user studies or uplift studies.\n\nPlease indicate to what extent you agree that providers should implement the practice from the list. You can add additional practices and provide details on any of the practices. You can also indicate which model evaluation techniques listed above or which other techniques can reliably assess which specific systemic risks.\n\n| Potential evaluation practices | Stron | Some what | Neither agree nor | Disagr | I don't know |\n|----------------------------------|---------|-------------|---------------------|----------|----------------|\n| | gly | | | ee | |\n| | agree | agree | | | |\n| | | | disagree | | |\n\n\n\n| Performing evaluations at several checkpoints throughout the model lifecycle, in particular during development and prior to internal deployment | X | | |\n|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----|----|----|\n| Performing evaluations at several checkpoints throughout the model lifecycle, in particular when the model risk profile changes such as with access to tools or with different release strategies | | X | |\n| Ensuring evaluations inform model deployment in real-world conditions | X | | |\n| Ensuring evaluations provide insights into the degree to which a model introduces or exacerbates risks | | | X |\n| Using non-public model evaluations , as appropriate | | X | |\n| Involve independent external evaluators , including with appropriate levels of access to the model and related information | | X | |\n| Involve affected persons , to understand effects of human interactions with a particular model over time | | X | |\n| Documenting evaluation strategies and results | X | | |\n\n\n\n| Reporting evaluation strategies and results publicly , as appropriate | X | |\n|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----|----|\n| Reporting evaluation strategies and results to selected authorities and administrative bodies , as appropriate, including sensitive evaluation results | | X |\n| Continuously evaluate and improve evaluation strategies based on information from risk assessment and mitigation measures, including from incidents and near-misses | X | |\n\nYour comments 700 character(s) maximum\n\nDocumentation/Reporting are core to ensuring accountability, the appropriateness of evaluations, and the development of new evaluation strategies.\n\nOrganisations who release open models, especially academic and SMEs, have different constraints and might not be able to hire external evaluators or report results privately. However, as models are de facto accessible to third parties, open source models as defined in the AI Act should be assumed to comply with these requirements.\n\nNon-public evaluations should only be used to avoid data contamination, as transparency is necessary to build trust and accountability. Involving affected persons should be done with care.\n\n- 16. Please provide links to relevant material on state-of-the-art risk assessment measures, such as model cards, data sheets, templates or other publications.\n\nOn the importance of reporting evaluation results in detail: Burnell et al.: Rethink reporting of evaluation results in AI; Aggregate metrics and lack of access to results limit understanding https://www.science.org/doi/10.1126/science.adf6369 Summary: https://montrealethics.ai/rethink-reporting-of-evaluation-results-in-ai/\n\nOn data contamination in evaluation data: Balloccu et al.: Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in\n\n\n\nClosed-Source LLMs https://arxiv.org/abs/2402.03927\n\nJacovi et al.: Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks https://aclanthology.org/2023.emnlp-main.308.pdf\n\nOn involving affected persons:\n\nSloane et al.: Participation Is not a Design Fix for Machine Learning\n\nhttps://dl.acm.org/doi/pdf/10.1145/3551624.3555285\n\nBirhane et al.: Power to the People? Opportunities and Challenges for Participatory AI https://arxiv.org/pdf/2209.07572\n\n17. What are the greatest challenges that a general-purpose AI model provider could face in implementing risk assessment measures, including model evaluations? 700 character(s) maximum\n\nWhile general-purpose AI model providers have access to significant technical knowledge and computational resources, they typically lack the expertise necessary to address social and economic risks of deploying models at scale - especially for risks that depend not just on properties of the model, but on its deployment and commercialisation formats.\n\nEngaging external stakeholders and affected communities is essential for identifying many of the most current and acute risks of the technology, but care must be taken to ensure that their involvement is not exploitative and that they benefit from the process rather than being used solely for the improvement of AI models and systems.\n\n## C. Technical risk mitigation\n\nCodes of Practice should also be focused on specific risk mitigation measures for general-purpose AI models with systemic risk. Following the risk taxonomy, appropriate measures could be applied to mitigate different systemic risks, tailored to each specific type and nature of risk, including their sources.\n\nThe following concerns technical risk mitigation measures, including cybersecurity protection for the general-purpose AI model and the physical infrastructure of the model. Measures can relate to model design, development or deployment.\n\nThis is in line with the focus of the Code of Practice Working Group 3 'Risk mitigation measures for systemic risks'.\n\nQuestion18. How can the effective implementation of technical risk mitigation measures reflect differences in size and capacity between various providers such as SMEs and start-ups? 700 character(s) maximum\n\nSMEs, start-ups, and open-source developers can implement technical risk mitigation by adhering to established good practices in AI model development. These interventions can be both more robust and more accessible than post-hoc interventions based on, e.g., safety fine-tuning or input filtering. Prioritising safety by design [1] and emphasising transparency\n\n\n\nrequirements help ensure that risk management is more accessible and achievable for providers of all sizes.\n\n- [1] https://info.thorn.org/hubfs/thorn-safety-by-design-for-generative-AI.pdf\n- 19. In the current state of the art, which specific technical risk mitigation measures should, in your view, general-purpose AI model providers take to effectively mitigate systemic risks along the entire model lifecycle, in addition to cybersecurity protection?\n\nPlease indicate to what extent you agree that providers should take the measures from the list. You can add additional measures and provide details on any of the measures, such as what is required for measures to be effective in practice.\n\n| Potential technical risk assessment measures | Strongly agree | Somewh at agree | Neither agree nor disagree | Disa gree | I don't know |\n|-------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------------|------------------------------|-------------|----------------|\n| Data governance such as data selection, cleaning, quality control | X | | | | |\n| Model design and development to achieve an appropriate level of trustworthiness characteristics such as model reliability, fairness or security | | | X | | |\n| Fine-tuning for trustworthiness and alignment such as through Reinforcement Learning from Human Feedback (RLHF) or Constitutional AI | | | X | | |\n| Unlearning techniques such as to remove specific harmful capabilities from models | | | | X | |\n\n\n\n| Technical deployment guardrails , such as content and other filters, capability restrictions, fine-tuning restrictions or monitoring-based restrictions in case of misuse by users | | | X |\n|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----|----|-----|\n| Mitigation measures relating to the model architecture, components, access to tools or model autonomy | X | | |\n| Detection, labelling and other measures related to AI-generated or manipulated content | | X | |\n| Regular model updates , including security updates | | | X |\n| Measuring model performance on an ongoing basis | | | X |\n| Identification and mitigation of model misuse | | X | |\n| Access control to tools and levels of model autonomy | | | X |\n\nYour comments 700 character(s) maximum\n\nReinforcement fine-tuning reduces unwanted model responses, but is brittle towards intentional misuse. Technical deployment guardrails, while helpful, have inherent limitations and must be thoughtfully implemented, taking into account deployment context. For open models, guardrails are often controlled by the deployer, not the developer. Dataset-level measures, like data governance, consistently mitigate risks and apply across both API deployment and open source. We also broadly caution against interventions that are beyond the state of the art and are not scientifically validated, such as unlearning or requiring fully reliable watermarking across modalities.\n\n\n\n- 20. Please provide links to relevant material on state-of-the-art technical risk mitigation practices, such as model cards, data sheets, templates or other publications.\n- 21. What are the greatest challenges that a general-purpose AI provider could face in implementing technical risk mitigation measures?\n\nPreventing misuse of models when developing them:\n\nKaffee et al.: Thorny Roses: Investigating the Dual Use Dilemma in Natural Language Processing https://aclanthology.org/2023.findings-emnlp.932/\n\nJailbreaking of closed models:\n\nZou et al.: Universal and Transferable Adversarial Attacks on Aligned Language Models https://llm-attacks.org/ and https://arxiv.org/abs/2307.15043\n\nLu et al.: Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models\n\nhttps://openaccess.thecvf.com/content/ICCV2023/papers/Lu\\_Set-level\\_Guidance\\_Attack\\_Bo osting\\_Adversarial\\_Transferability\\_of\\_Vision-Language\\_Pre-training\\_Models\\_ICCV\\_2023\\_pape r.pdf\n\n700 character(s) maximum\n\nWhen deploying novel GPAI models, it is impossible to predict all possible misuses of the models. Therefore, adapting models for specific settings makes it easier to develop context-specific risk mitigation techniques\n\n[https://ieeexplore.ieee.org/abstract/document/1507050]. With wider access to the models through open models, domain experts are able to test for specific risks. It also requires updating of the risk mitigation strategies as they're developed.\n\nRequiring open model providers to track all model reuse is technically infeasible as well as undesirable, as it restricts reuse possibilities for startups and research initiatives, limiting innovation and further development.\n\n## D. Internal risk management and governance for general-purpose AI model providers\n\nThe following concerns policies and procedures to operationalise risk management in internal governance of general-purpose AI model providers, including keeping track of, documenting, and reporting serious incidents and possible corrective measures.\n\nThis is in line with the focus of the Code of Practice Working Group 4 'Internal risk management and governance for general-purpose AI model providers'.\n\n- 22. How can the effective implementation of internal risk management and governance measures reflect differences in size and capacity between various providers such as SMEs and start-ups?\n\n700 character(s) maximum\n\n\n\nClear open and accessible standards that can be followed without involving external organisations are much more accessible to SMEs and start-ups. Risk evaluation should avoid over-relying on mandatory external audits besides the one conducted through the AI Office or similar entities, and instead focus on clear, straightforward guidelines that small teams or individual developers can easily follow. The Code of Practice should not impose requirements that necessitate dedicated personnel, recognising the constraints faced by smaller entities.\n\n## Links to relevant material\n\nMarino et al.: Compliance Cards: Automated EU AI Act Compliance Analyses amidst a Complex AI Supply Chain https://arxiv.org/abs/2406.14758\n\nOn the limitations of AI audits Birhane et al.: AI auditing: The Broken Bus on the Road to AI Accountability https://arxiv.org/abs/2401.14462\n\nRaji et al.: Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance https://dl.acm.org/doi/pdf/10.1145/3514094.3534181\n\nCostanza-Chock et al.: Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem https://arxiv.org/pdf/2310.02521\n\n23. In the current state of the art, which specific internal risk management and governance measures should, in your view, general-purpose AI model providers take to effectively mitigate systemic risks along the entire model lifecycle, in addition to serious incident reporting?\n\nPlease indicate to what extent you agree that providers should take the measures from the list. You can add additional measures and provide details on any of the measures, such as what is required for measures to be effective in practice.\n\n| Potential internal risk management and governance measures | Strongl y agree | Somew hat agree | Neither agree nor disagree | Disa gree | I don't know |\n|---------------------------------------------------------------------------------------------------------------------|-------------------|-------------------|------------------------------|-------------|----------------|\n| Risk management framework across the model lifecycle | | X | | | |\n| Internal independent oversight functions in a transparent governance structure, such as related to risks and ethics | | | X | | |\n\n\n\n| Traceability in relation to datasets, processes, and decisions made during model development | X | |\n|-----------------------------------------------------------------------------------------------------------------------------|-----|----|\n| Ensuring that staff are familiar with their duties and the organisation's risk management practices | | X |\n| Responsible scaling policies | | X |\n| Acceptable use policies | X | |\n| Whistleblower protections | X | |\n| Internal resource allocation towards risk assessment and mitigation measures as well as research to mitigate systemic risks | | X |\n| Robust security controls including physical security, cyber security and information security | | X |\n| External accountability measures such as third-party audits, model or aggregated data access for researchers | X | |\n| Other collaborations and involvements of a diverse set of stakeholders , including impacted communities | | X |\n| Responsible release practices including staged release, particularly before open-sourcing a model with systemic risk | | X |\n\n\n\n| Transparency reports such as model cards, system cards or data sheets | X | | |\n|----------------------------------------------------------------------------------------------------------|-----|----|----|\n| Human oversight mechanisms | | X | |\n| Know-your-customer practices | | X | |\n| Logging, reporting and follow-up of near-misses along the lifecycle | | | X |\n| Measures to mitigate and remediate deployment issues and vulnerabilities | | X | |\n| Complaints handling and redress mechanisms, such as bug bounty programs | X | | |\n| Mandatory model updating policies and limit on maximum model availability | | | X |\n| Third-party and user discovery mechanisms and reporting related to deployment issues and vulnerabilities | | X | |\n\n- 24. Please provide links to relevant material on state-of-the-art governance risk mitigation practices, such as model cards, data sheets, templates or other publications.\n\nComplaints handling and redress mechanisms:\n\nCattell et al.: Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities https://arxiv.org/abs/2402.07039\n\nStaged releases:\n\nSolaiman: The Gradient of Generative AI Release: Methods and Considerations https://arxiv.org/abs/2302.04844\n\nOn involving affected communities:\n\nSloane et al.: Participation Is not a Design Fix for Machine Learning\n\n\n\nhttps://dl.acm.org/doi/pdf/10.1145/3551624.3555285\n\nBirhane et al.: Power to the People? Opportunities and Challenges for Participatory AI https://arxiv.org/pdf/2209.07572\n\n25. What are the greatest challenges that a general-purpose AI provider could face in implementing governance risk mitigation measures? 700 character(s) maximum\n\nSmaller providers, such as SMEs and startups, often lack the financial and human resources needed to establish dedicated governance structures and devise their own risk management frameworks from scratch. The rapidly changing nature of AI and its applications requires governance measures to continuously adapt to new risks and developments. Meaningfully involving diverse stakeholders, including impacted communities, in governance practices requires effective communication and collaboration. Ease of implementation by smaller actors and of adaptation to changing conditions for all can be greatly facilitated by direct collaboration between actors.\n\n## Section 3. Reviewing and monitoring of the General-Purpose AI Code of Practice\n\nThe process of drawing-up the first Code of Practice will start immediately after the AI Act enters into force and will last for 9 months, in view of enabling providers of general-purpose AI models to demonstrate compliance on time. The AI Office shall aim to ensure that the Code of Practice clearly sets out their specific objectives and contains commitments or measures, including key performance indicators as appropriate, to ensure the achievement of those objectives. The AI Office shall aim to ensure that participants to the Code of Practice report regularly to the AI Office on the implementation of the commitments and the measures taken and their outcomes, including as measured against the key performance indicators as appropriate. Key performance indicators and reporting commitments shall reflect differences in size and capacity between various participants. The AI Office and the Board shall regularly monitor and evaluate the achievement of the objectives of the Code of Practice by the participants and their contribution to the proper application of this Regulation. The AI Office shall, as appropriate, encourage and facilitate the review and adaptation of the Code of Practice.\n\nQuestion26. What are examples of key performance indicators which are, in your view, effective to measure the compliance of participants with the objectives and measures which will be established by the Code of Practice?\n\n700 character(s) maximum\n\nThe Code of Practice aims to ensure that developers demonstrate adequate foresight and\n\n\n\nconsideration. This can be validated in collaboration with external bodies, such as the scientific panel. When alerts arise regarding the use of GPAI models, the panel should evaluate the extent to which the Code of Practice was followed and whether it adequately covered and addressed the issues.\n\nThe Code of Practice is considered successful if it meets the dual goals of seeing extensive and transparent adoption from all categories of developers and of serving as an effective support for assessing practices in the light of stakeholder complaints to support more reliable and trustworthy technology.\n\n- 27. How can key performance indicators and reporting commitments for providers reflect differences in size and capacity between various providers such as SMEs and start-ups? 700 character(s) maximum\n\nKey performance indicators and reporting commitments for providers should reflect differences in size by tailoring complexity and resource requirements to the provider's scale. SMEs and startups could have simpler reporting processes and more flexible timelines, with access to open-source tools to reduce costs. For example, smaller entities may use internal evaluations or peer reviews instead of costly third-party audits. KPIs should focus on core issues like data governance, allowing startups to gradually adopt standards while remaining competitive, fostering innovation, and encouraging diverse ecosystem participation.\n\n28. Which aspects should inform the timing of review and adaptation of the content of the Code of Practice for general-purpose AI models in order to ensure that the state of the art is reflected? This does not necessarily imply a complete review, but can only involve pertinent parts.\n\nPlease rank all relevant aspects from the following list from 1 to 4 (1 being the most important). You can add additional aspects and provide details on any of the aspects or other considerations under \"Specify\".\n\nPre-planned intervals to assess the need for revision: Assessments of whether the content of the Code of Practice for general-purpose AI models needs to be revised or adapted should be pre-planned for specific time intervals.\n\nRank 2\n\nAlerts by independent experts or other stakeholders: Alerts by selected independent experts, such as by the Scientific Panel which will be set up in the AI Act governance structure, or by other stakeholders such as\n\nRank 1\n\n\n\ndownstream providers, academia or civil society should inform a revision of the content of the Code of Practice.\n\nMonitoring and foresight: Independent monitoring and foresight related to the AI ecosystem, technological and market developments, emergence of systemic risks and any other relevant trends, such as related to sources of risks like model autonomy, should inform a revision of the content of the Code of Practice\n\n## Your comments\n\n700 character(s) maximum\n\nThe AI Office should incorporate expert feedback on the Code of Practice and make necessary adjustments based on this input. Nevertheless, regular reviews are essential to determine if updates are needed. While foresight can be a useful tool to analyse current trends, in the context of the fast developing technology of AI it is not a proven tool to implement policy changes.\n\nLink to relevant material\n\nA way of enabling the general public to alert, e.g., the AI Office of model flaws and risks are coordinated flaw disclosures: Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities https://arxiv.org/abs/2402.07039\n\nRank 3", + "chunks": [ + { + "headings": [ + "Hugging Face Comments on the EU AI Office's Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI" + ], + "markdown": "\n\n## Hugging Face Comments on the EU AI Office's Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI\n\nResponses by: Lucie-Aim\u00e9e Kaffee, Yacine Jernite, Bruna Trevelin\n\n## About Hugging Face\n\nHugging Face is a community-oriented company working to democratise good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analysing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. Part of those activities include releasing open-source tools to support other actors developing their own models, as well as collaborating on new state-of-the-art GPAI models such as StarCoder2, the highest-performing fully-open code model to date. Its training dataset, The Stack, exemplifies a working opt-out methodology, and its governance card provides an implementation of the goals of the Code of Practice: https://shorturl.at/lBbAa.\n\n## Section 1. General-purpose AI models: transparency and copyright-related rules\n\nA. Information and documentation by general-purpose AI model providers to providers of AI systems\n\nProviders of general-purpose AI models have a particular role and responsibility along the AI value chain, as the models they provide may form the basis for a range of downstream systems, often provided by downstream providers that necessitate a good understanding of the models and their capabilities, both to enable the integration of such models into their products, and to fulfil their obligations under the AI Act or other regulations. Therefore, model providers should draw up, keep up-to-date and make available information and documentation to providers of AI systems who intend to integrate the general-purpose AI model into their AI system. Widely adopted documentation practices include model cards and data sheets.\n\n\n\nA minimal set of elements of information and documentation by general-purpose AI model providers to providers of AI systems is already set out in AI Act Annex XII.\n\n- 1. In the current state of the art , for which elements of information and documentation by general-purpose AI model providers to providers of AI systems do practices exist that, in your view, achieve the above-mentioned purpose ?\n\nFrom the list below following AI Act Annex XII, please select all relevant elements. If such practices exist, please provide links to relevant material substantiating your reply, such as model cards, data sheets or templates.\n\nA general description of the general-purpose AI model including:\n\nThe tasks that the model is intended to perform and the type and nature of AI systems into which it can be integrated;\n\nThe acceptable use policies applicable;\n\nThe date of release and methods of distribution;\n\nHow the model interacts, or can be used to interact, with hardware or software that is not part of the model itself, where applicable;\n\nThe versions of relevant software related to the use of the general-purpose AI model, where applicable;\n\nThe architecture and number of parameters;\n\nThe modality (e.g., text, image) and format of inputs and outputs;\n\nThe licence for the model.\n\nA description of the elements of the model and of the process for its development, including:\n\nThe technical means (e.g., instructions for use, infrastructure, tools) required for the general-purpose AI model to be integrated into AI systems;\n\nThe modality (e.g., text, image, etc.) and format of the inputs and outputs and their maximum size (e.g., context window length, etc.);\n\nInformation on the data used for training, testing and validation, where applicable, including the type and provenance of data and curation methodologies.\n\nLinks to relevant material\n\nModel cards on Hugging Face: https://huggingface.co/docs/hub/en/model-cards Example of usage of model cards: https://huggingface.co/meta-llama/Meta-Llama-3-8B (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means)\n\n\n\nhttps://huggingface.co/CohereForAI/c4ai-command-r-v01 (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means)\n\nExample usage of the model cards: https://huggingface.co/HuggingFaceM4/idefics2-8b (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means, dataset for training (see below as OBELICS)) Usage policy (RAIL and OpenRAIL):\n\nhttps://www.licenses.ai/blog/2022/8/18/naming-convention-of-responsible-ai-licenses https://huggingface.co/spaces/bigscience/license\n\nUse of RAIL: https://huggingface.co/bigscience/bloom\n\nTool Use With Command R: https://cohere.com/blog/tool-use-with-command-r Dataset cards on Hugging Face: https://huggingface.co/docs/hub/en/datasets-cards https://github.com/huggingface/datasets/blob/main/templates/README\\_guide.md\n\nExample of usage of dataset cards:\n\nhttps://huggingface.co/datasets/HuggingFaceM4/OBELICS\n\n2. Beyond the minimal set of elements listed in the previous question, are there other elements that should be included in information and documentation by general-purpose AI model providers to providers of AI systems to achieve the above-mentioned purpose?\n\nYes\n\nNo\n\nI don't know\n\nLinks to relevant material\n\nWhere applicable, ethical charters and other considerations made in the creation of the model: https://bigscience.huggingface.co/blog/bigscience-ethical-charter https://huggingface.co/blog/ethical-charter-multimodal Governance cards: https://arxiv.org/abs/2312.03872\n\nhttps://huggingface.co/datasets/bigcode/governance-card\n\n## B. Technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities\n\nIn addition to the provision of information on the general-purpose AI model for its usage by the downstream providers, technical documentation should be prepared and kept up to date by the general-purpose AI model provider for the purpose of making it available, upon request, to the AI Office and the national competent authorities.\n\nA minimal set of elements of such technical documentation of the general-purpose AI model to be made available by providers, upon request, to the AI Office and the national competent authorities is already set out in AI Act Annex XI.\n\n\n\n- 3. In the current state of the art , for which elements of documentation by general-purpose AI model providers do practices exist that, in your view, provide a necessary level of information for the above-mentioned purpose ?\n\nFrom the list below following AI Act Annex XI, please select all relevant elements. If such practices exist, please provide links to relevant material substantiating your reply, such as model cards, data sheets or templates.\n\nA general description of the general-purpose AI model including:\n\nThe tasks that the model is intended to perform and the type and nature of AI systems into which it can be integrated;\n\nThe acceptable use policies applicable;\n\nThe date of release and methods of distribution;\n\nThe architecture and number of parameters;\n\nThe modality (e.g., text, image) and format of inputs and outputs;\n\nThe licence.\n\nA description of the elements of the model, and relevant information of the process for the development, including:\n\nThe technical means (e.g., instructions for use, infrastructure, tools) required for the general-purpose AI model to be integrated into AI systems;\n\nThe design specifications of the model and training process, including training methodologies and techniques, the key design choices including the rationale and assumptions made; what the model is designed to optimise for and the relevance of the different parameters, as applicable;\n\nInformation on the data used for training, testing and validation, where applicable, including the type and provenance of data and curation methodologies (e.g. cleaning, filtering etc), the number of data points, their scope and main characteristics; how the data was obtained and selected as well as all other measures to detect the unsuitability of data sources and methods to detect identifiable biases, where applicable;\n\nthe computational resources used to train the model (e.g. number of floating point operations), training time, and other relevant details related to the training;\n\nknown or estimated energy consumption of the model.\n\nAdditional information to be provided by providers of general-purpose AI models with systemic risk:\n\n\n\nA detailed description of the evaluation strategies, including evaluation results, on the basis of available public evaluation protocols and tools or otherwise of other evaluation methodologies. Evaluation strategies shall include evaluation criteria, metrics and the methodology on the identification of limitations;\n\nWhere applicable, a detailed description of the measures put in place for the purpose of conducting internal and/or external adversarial testing (e.g., red teaming), model adaptations, including alignment and fine-tuning;\n\nWhere applicable, a detailed description of the system architecture explaining how software components build or feed into each other and integrate into the overall processing;", + "char_slice": [ + 0, + 9503 + ] + }, + { + "headings": [ + "## Hugging Face Comments on the EU AI Office's Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI", + "## About Hugging Face", + "## Section 1. General-purpose AI models: transparency and copyright-related rules", + "## B. Technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities" + ], + "markdown": "including the type and provenance of data and curation methodologies (e.g. cleaning, filtering etc), the number of data points, their scope and main characteristics; how the data was obtained and selected as well as all other measures to detect the unsuitability of data sources and methods to detect identifiable biases, where applicable;\n\nthe computational resources used to train the model (e.g. number of floating point operations), training time, and other relevant details related to the training;\n\nknown or estimated energy consumption of the model.\n\nAdditional information to be provided by providers of general-purpose AI models with systemic risk:\n\n\n\nA detailed description of the evaluation strategies, including evaluation results, on the basis of available public evaluation protocols and tools or otherwise of other evaluation methodologies. Evaluation strategies shall include evaluation criteria, metrics and the methodology on the identification of limitations;\n\nWhere applicable, a detailed description of the measures put in place for the purpose of conducting internal and/or external adversarial testing (e.g., red teaming), model adaptations, including alignment and fine-tuning;\n\nWhere applicable, a detailed description of the system architecture explaining how software components build or feed into each other and integrate into the overall processing;\n\nLinks to relevant material\n\nExample of usage of model cards: https://huggingface.co/meta-llama/Meta-Llama-3-8B (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means)\n\nhttps://huggingface.co/CohereForAI/c4ai-command-r-v01 (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means)\n\nExample usage of the model cards: https://huggingface.co/HuggingFaceM4/idefics2-8b (includes intended tasks, use policy, release date, version, architecture and number of parameters, modality, license, technical means, dataset for training (see below as IDEFICS)) Documentation of SmolLM, including experiments, evaluation, and description of the training of the model: https://huggingface.co/blog/smollm\n\nOpenLLM leaderboard, evaluation on standard evaluation metrics:\n\nhttps://huggingface.co/spaces/open-llm-leaderboard/open\\_llm\\_leaderboard Existing events for red teaming:\n\nhttps://aivillage.org/generative%20red%20team/generative-red-team-2/\n\nThe Environmental Impacts of AI - Primer:\n\nhttps://huggingface.co/blog/sasha/ai-environment-primer\n\nEnergy Star Ratings for AI Models:\n\nhttps://huggingface.co/blog/sasha/energy-star-ai-proposal\n\n4. Beyond the minimal set of elements listed in the previous question, are there other elements that should, in your view, be included in technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities?\n\nYes\n\nNo\n\nI don't know\n\nLinks to relevant material\n\n\n\nSocial impact evaluations, see Evaluating the Social Impact of Generative AI Systems in Systems and Society https://arxiv.org/abs/2306.05949\n\n## C. Policy to respect Union copyright law\n\nThe AI Act requires providers of general-purpose AI models to put in place a policy to comply with Union law on copyright and related rights, and in particular to identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3) of Directive (EU) 2019/790.\n\n5. What are, in your view, the main elements that need to be included in the policy that providers of general-purpose AI models have to put in place to comply with Union law on copyright and related rights, as required by the AI Act?\n\nPlease select all relevant options from the list of options suggested below. If selected, please elaborate further on the content of the measures and provide links to any good practices you are aware of.\n\nAllocation of responsibility within the organisation for the implementation and monitoring of compliance with the policy and the measures therein;\n\nMeasures to identify and comply with the rights reservation from the text and data mining exception pursuant to Article 4(3) of Directive (EU) 2019/790;\n\nMeasures to obtain the authorisation from right holders, where applicable;\n\nMeasures to detect and remove collected copyright protected content for which rights reservation from the text and data mining exception has been expressed pursuant to Article 4(3) of Directive (EU) 2019/790;\n\nMeasures to prevent the generation, in the outputs of the model, of copyright infringing content;\n\nMeans for contact with rightsholders;\n\nMeasures for complaint handling from rightsholders;\n\nOther\n\nI don't know\n\nYour comments\n\n700 character(s) maximum\n\nMeasures to comply with rights reservation need to be accessible to SMEs and developers of open models, including actors leveraging public datasets. This requires ensuring that\n\n\n\nopt-outs are publicly accessible, machine-readable with a known protocol. Such an approach also benefits rights holders more than fragmented protocols from GPAI developers. Measures to prevent the generation of copyright infringing content should also be cognizant of open developers and of the current lack of accepted definition of 'substantial similarity' for model generations. A sufficiently detailed training data summary template is more accessible to open developers than under-defined output filtering requirements.\n\nLinks to relevant material\n\nOpen Future: Considerations For Implementing Rightholder Opt-Outs By AI Model Developers https://openfuture.eu/wp-content/uploads/2024/05/240516considerations\\_of\\_opt-out\\_com pliance\\_policies.pdf\n\n- 6. How can, in your view, the policy to be put in place by providers of general-purpose AI models to comply with Union copyright law ensure that providers of those models comply with the existing solutions for the expression of the text and data mining rights reservation , pursuant to Article 4(3) of Directive (EU) 2019/790?\n\nPlease explain how this can be achieved and specify from the list below the state-of-the-art technologies you are aware of to identify and comply with the right reservations expressed by rightsholders, providing further information and examples.\n\nTechnologies/tools that identify right reservations at the website/domain level\n\nTechnologies/tools that identify right reservations at work level\n\nTechnologies/tools that aggregate the expression of right reservations\n\nOther\n\nI don't know\n\nYour comments\n\n700 character(s) maximum\n\nThe policy should ensure that the technologies and standards used for expressing opt-outs are widely and freely accessible, allowing rights holders and AI developers to engage without barriers that could adversely impact competition across developers. The opt-out process must be straightforward and user-friendly, avoiding the need for legal expertise and supporting automated processing to streamline compliance with the text and data mining rights reservation under Article 4(3) of Directive (EU) 2019/790. Additionally, tools to aggregate and manage opt-out requests are needed, as current solutions are insufficient and fail to prevent data from being crawled despite opt-outs.\n\nLinks to relevant material\n\n\n\nDomain-level\n\nAmI in The Stack? App https://huggingface.co/spaces/bigcode/in-the-stack\n\nHugging Face has integrated the Spawning HaveIBeenTrained API with the Hub for datasets\n\nthat have an image\\_url field, e.g. see the report widget on the right in\n\nhttps://huggingface.co/datasets/kakaobrain/coyo-700m\n\nCommonCrawl, which is a dataset of web crawl data, often used as the base for training data for models, respects opt-outs via robot.txt: https://commoncrawl.org/ccbot\n\nrobots.txt; ai.txt https://spawning.ai/ai-txt\n\nTDM Reservation protocol (TDMRep),\n\nhttps://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240202/\n\nWork-level\n\nCoalition for Content Provenance and Authenticity (C2PA)), https://c2pa.org/\n\nInternational Standard Content Code (ISCC), http://iscc.codes/ Spawning.ai's Have I been trained?, haveibeentrained.com\n\nOther relevant material\n\nAnalysis of opt-outs: https://www.dataprovenance.org/Consent\\_in\\_Crisis.pdf\n\n- D. Summary about content used for the training of general-purpose AI models The AI Act requires providers to draw up and make publicly available a sufficiently detailed summary about the content used for training of the general-purpose AI model, according to a template provided by the AI Office. While due account should be taken of the need to protect trade secrets and confidential business information, the summary is to be generally comprehensive in its scope instead of technically detailed to facilitate parties with legitimate interests, including copyright holders, to exercise and enforce their rights under Union law. The template that should be drafted by the AI Office for the sufficiently detailed summary should be simple, effective, and allow providers to provide the required summary in narrative form.\n- 7. What are in your view the categories of information sources that should be presented in the summary to ensure that it comprehensively describes the main sources of data used for the training of the general-purpose AI model?\n\nFrom the list below, please select all options that you consider relevant.\n\nPublic/ open data repositories\n\nContent/data publicly available online (e.g. scraped from the internet", + "char_slice": [ + 8111, + 17614 + ] + }, + { + "headings": [ + "## Hugging Face Comments on the EU AI Office's Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI", + "## About Hugging Face", + "## Section 1. General-purpose AI models: transparency and copyright-related rules", + "## B. Technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities", + "## C. Policy to respect Union copyright law" + ], + "markdown": "rained.com\n\nOther relevant material\n\nAnalysis of opt-outs: https://www.dataprovenance.org/Consent\\_in\\_Crisis.pdf\n\n- D. Summary about content used for the training of general-purpose AI models The AI Act requires providers to draw up and make publicly available a sufficiently detailed summary about the content used for training of the general-purpose AI model, according to a template provided by the AI Office. While due account should be taken of the need to protect trade secrets and confidential business information, the summary is to be generally comprehensive in its scope instead of technically detailed to facilitate parties with legitimate interests, including copyright holders, to exercise and enforce their rights under Union law. The template that should be drafted by the AI Office for the sufficiently detailed summary should be simple, effective, and allow providers to provide the required summary in narrative form.\n- 7. What are in your view the categories of information sources that should be presented in the summary to ensure that it comprehensively describes the main sources of data used for the training of the general-purpose AI model?\n\nFrom the list below, please select all options that you consider relevant.\n\nPublic/ open data repositories\n\nContent/data publicly available online (e.g. scraped from the internet)\n\nProprietary data generated by the provider\n\nUser-generated data obtained through the services or products provided by the provider\n\nCopyright protected content licensed by rightsholders\n\n\n\nOther data/content or data sets acquired from third parties (e.g. licensed proprietary databases, data acquired from datahubs, public interest institutions such as libraries etc.)\n\nSynthetically generated data\n\nOther\n\nI don't know\n\nIf selected, please specify the level of granularity/detail for each of the selected options , keeping in mind that AI Act requires the summary to be comprehensive instead of technically detailed and provided in a narrative form to facilitate parties with legitimate interests, including rightsholders, to exercise and enforce their rights under Union law, while taking due account of the need to protect providers' trade secrets and confidential business information. If additional categories should be considered, please specify them and the level of granularity/detail. You can motivate your choice and provide links to any good practices.\n\n700 character(s) maximum\n\nInformation would ideally be detailed enough for any rights holder to understand how their data contributes to the GPAI system. Information needs to be detailed enough to ensure that broad trends in the use of personal data, creative works, and representations of specific groups affect the system are legible to external stakeholders including journalists and policymakers. This level of granularity requires different definitions for different categories, e.g. a list of URLs for public data repositories, the top domains by content in a web crawl, or the respective sizes and status of intermediaries (e.g. publisher, platform, archive) for different modalities of licensed content.\n\nLinks to relevant material\n\nOpen Future's Towards robust training data transparency:\n\nhttps://openfuture.eu/publication/towards-robust-training-data-transparency/ What's in my big data: https://arxiv.org/abs/2310.20707 https://wimbd.apps.allenai.org/\n\nOn training data memorisation and PII data leakage:\n\nLukas et al.: Analyzing Leakage of Personally Identifiable Information in Language Models https://arxiv.org/pdf/2302.00539\n\nCarlini et al.: Extracting Training Data from Large Language Models\n\nhttps://arxiv.org/abs/2012.07805\n\n- 8. In your view, should the summary include one or more of the following characteristics/information about the data used for the training/of the general-purpose AI model in order to facilitate parties with legitimate interests, including copyright holders, to enforce their rights under Union law?\n\n\n\nPlease select all relevant options from the list of options suggested below. If selected, please explain your choice and provide links to any good practices.\n\nModalities / type of data (text, images, videos, music, etc);\n\nNature of the data (personal, non-personal or mixed);\n\nTime of acquisition/collection of the data;\n\nData range of the data (e.g. time span), including date cutoffs\n\nIn case of data scraped from the internet, information about the crawlers used;\n\nInformation about diversity of the data (for example linguistic, geographical, demographic diversity);\n\nPercentage of each of the main data sources to the overall training/fine-tuning;\n\nLegal basis for the processing under Union copyright law and data protection law, as applicable;\n\nMeasures taken to address risks to parties with legitimate interests (e.g. measures to identify and respect opt-out from the text and data mining exception, respect data protection and address privacy risks, bias, generation of illegal or harmful content;\n\nOther\n\nI don't know\n\nYour comments\n\n700 character(s) maximum\n\nAll categories outlined have direct bearing on EU regulations including intellectual property, anti-discrimination, and personal data protections. While they can easily be documented for newly curated datasets, retroactive information gathering is more challenging. Requirements should reflect this by permitting less detailed documentation on pre-existing datasets, as long as they are properly identified and updates are fully documented. Requirements should be more limited for open access datasets and SMEs to reflect their inherent transparency and different resources respectively. The legal basis dimension can be the most difficult to assess and would benefit from harmonised interpretations.\n\nLinks to relevant material\n\nOpen Future's Towards robust training data transparency:\n\nhttps://openfuture.eu/publication/towards-robust-training-data-transparency/\n\n- 9. Considering the purpose of the summary to provide meaningful information to facilitate the exercise of the rights of parties with legitimate interests under Union law, while taking due account of the need to respect business confidentiality and trade secrets of providers, what types of information in your view are justified not to be disclosed in the summary as being not necessary or disproportionate for its purpose described above?\n\n\n\n## 700 character(s) maximum\n\nThe performance of a GPAI system depends not only on the source of the data but on the specific learning objectives, data processing, and increasingly on techniques of 'self-play' and RLHF whose specifics remain beyond the scope of the proposed summary. The latest release of the OpenAI o1 system in particular exemplifies the role of these advanced techniques in gaining a competitive advantage. Therefore, focusing information requirements on data that has external rights holders can help support the rights of EU citizens and support a healthy and sustainable data ecosystem without requiring GPAI developers to reveal their unique contributions to the performance of their products.\n\n## Section 2. General-purpose AI models with systemic risk: risk taxonomy, assessment and mitigation\n\n## A. Risk taxonomy\n\nSome general-purpose AI models could pose systemic risks, which should be understood to increase with model capabilities and model reach and can arise along the entire lifecycle of the model. 'Systemic risks' refer to risks that are specific to the high-impact capabilities of general-purpose AI models (matching or exceeding the capabilities of the most advanced general-purpose AI models); have a significant impact on the Union market due to their reach; or are due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or society as a whole, that can be propagated at scale across the value chain (AI Act Article 3(65)). Systemic risks are influenced by conditions of misuse, model reliability, model fairness and model security, the level of autonomy of the model, its access to tools, novel or combined modalities, release and distribution strategies, the potential to remove guardrails and other factors. The Code of Practice should help to establish a risk taxonomy of the type and nature of the systemic risks at Union level, including their sources. The Code should take into account international approaches.\n\n10. Do you consider the following list of systemic risks based on AI Act Recital 110 and international approaches to be comprehensive to inform a taxonomy of systemic risks from general-purpose AI models? If additional risks should be considered in your view, please specify.\n\n## Systemic risk from model malfunctions\n\n- \u25cf Harmful bias and discrimination: The ways in which models can give rise to harmful bias and discrimination with risks to individuals, communities or societies.\n- \u25cf Misinformation and harming privacy: The dissemination of illegal or false content and facilitation of harming privacy with threats to democratic values and human rights.\n\n\n\n- \u25cf Major accidents: Risks in relation to major accidents and disruptions of critical sectors, that a particular event could lead to a chain reaction with considerable negative effects that could affect up to an entire city, an entire domain activity or an entire community.\n- \u25cf Loss of control: Unintended issues of control relating to alignment with human intent, the effects of interaction and tool use, including for example the capacity to control physical systems, 'self-replicating' or training other models.\n\n## Systemic risk from malicious use\n\n- \u25cf Disinformation: The facilitation of disinformation and manipulation of public opinion with threats to democratic values and human rights.\n- \u25cf Chemical, biological, radiological, and nuclear risks: Dual-use science risks related to ways in which barriers to entry can be lowered, including for weapons development, design acquisition, or use.\n- \u25cf Cyber offence: Risks related to offensive cyber capabilities such as the ways in which vulnerability discovery, exploitation, or operational use can be enabled.\n\n## Other systemic risks, with reasonably foreseeable negative effects on\n\n- \u25cf public health\n- \u25cf safety\n- \u25cf democratic processes\n- \u25cf public and economic security\n- \u25cf fundamental rights\n- \u25cf the society as a whole.\n\nYes, this list of", + "char_slice": [ + 16269, + 26648 + ] + }, + { + "headings": [ + "## Hugging Face Comments on the EU AI Office's Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI", + "## About Hugging Face", + "## Section 1. General-purpose AI models: transparency and copyright-related rules", + "## B. Technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities", + "## C. Policy to respect Union copyright law", + "## 700 character(s) maximum", + "## Section 2. General-purpose AI models with systemic risk: risk taxonomy, assessment and mitigation", + "## A. Risk taxonomy", + "## Systemic risk from model malfunctions", + "## Systemic risk from malicious use", + "## Other systemic risks, with reasonably foreseeable negative effects on" + ], + "markdown": "-->\n\n- \u25cf Major accidents: Risks in relation to major accidents and disruptions of critical sectors, that a particular event could lead to a chain reaction with considerable negative effects that could affect up to an entire city, an entire domain activity or an entire community.\n- \u25cf Loss of control: Unintended issues of control relating to alignment with human intent, the effects of interaction and tool use, including for example the capacity to control physical systems, 'self-replicating' or training other models.\n\n## Systemic risk from malicious use\n\n- \u25cf Disinformation: The facilitation of disinformation and manipulation of public opinion with threats to democratic values and human rights.\n- \u25cf Chemical, biological, radiological, and nuclear risks: Dual-use science risks related to ways in which barriers to entry can be lowered, including for weapons development, design acquisition, or use.\n- \u25cf Cyber offence: Risks related to offensive cyber capabilities such as the ways in which vulnerability discovery, exploitation, or operational use can be enabled.\n\n## Other systemic risks, with reasonably foreseeable negative effects on\n\n- \u25cf public health\n- \u25cf safety\n- \u25cf democratic processes\n- \u25cf public and economic security\n- \u25cf fundamental rights\n- \u25cf the society as a whole.\n\nYes, this list of systemic risks is comprehensive.\n\nFurther or more specific systemic risks should be considered.\n\nThe list is comprehensive, as long as the risks currently listed under other systemic risks are prioritised alongside the other two categories; in particular safety, democratic processes, public and economic security, fundamental rights have been at the forefront of recent discussions such as the use of AI to decide whether unemployed workers get benefits [1] or access to information about democratic processes [2].\n\n[1]\n\nhttps://gizmodo.com/googles-ai-will-help-decide-whether-unemployed-workers-get-benefits-2 000496215\n\n[2]\n\nhttps://www.latimes.com/opinion/story/2024-03-08/primaries-voting-elections-ai-misinforma tion-plaforms-chatgpt\n\n\n\n- 11. What are in your view sources of systemic risks that may stem from the development, the placing on the market, or the use of general-purpose AI models? Systemic risks should be understood to increase with model capabilities and model reach.\n\nPlease select all relevant elements from the list. If additional sources should be considered, please specify. You can also provide details on any of the sources or other considerations.\n\nLevel of autonomy of the model: The degree to which a general-purpose AI model has the capability to autonomously interact with the world, plan ahead, and pursue goals.\n\nAdaptability to learn new, distinct tasks: The capability of a model to independently acquire skills for different types of tasks.\n\nAccess to tools: A model gaining access to tools, such as databases or web browsers, and other affordances in its environment.\n\nNovel or combined modalities: Modalities a model can process as input and generate as output, such as text, images, video, audio or robotic actions.\n\nRelease and distribution strategies: The way a model is released, such as under free and open-source license, or otherwise made available on the market.\n\nPotential to remove guardrails: The ability to bypass or disable pre-defined safety constraints or boundaries set up to ensure a model operates within desired parameters and avoids unintended or harmful outcomes.\n\nAmount of computation used for training the model: Cumulative amount of computation ('compute') used for model training measured in floating point operations as one of the relevant approximations for model capabilities.\n\nData set used for training the model: Quality or size of the data set used for training the model as a factor influencing model capabilities.\n\nOther\n\nPlease specify\n\n700 character(s) maximum\n\nThe content of the training datasets is a strong factor for most of the systemic risks considered. The prevalence of personal data, hate speech, information pertaining to capabilities of concern for the model, or known misinformation in the training dataset, for example, have direct bearing on the model behaviour; often more so than the results of a model on general performance benchmarks.\n\nYour comments\n\n700 character(s) maximum\n\nDistribution strategies are important but equating open-source with higher risk is misguided. There is no evidence suggesting that open-source models are riskier than closed-source ones\n\n\n\n[https://crfm.stanford.edu/open-fms/]. In fact, closed models are often more accessible due to user-friendly interfaces and cheap commercial access for a large user base [https://tinyurl.com/yf2yu8zv]; which compounds with the frailty of deployment-level guardrails [https://llm-attacks.org/].\n\nOpen-source releases offer unique benefits and risk strategies especially at the ecosystem level [https://huggingface.co/blog/ethics-soc-3] Limitations of computation thresholds should be acknowledged [https://tinyurl.com/mrb4sjzc ]\n\n## B. Risk identification and assessment measures\n\nIn light of potential systemic risks, the AI Act puts in place effective rules and oversight. Providers of general-purpose AI models with systemic risks should continuously assess and mitigate systemic risks. The Code of Practice should be focused on specific risk assessment measures for general-purpose AI models with systemic risk. Following the risk taxonomy, appropriate measures could be applied to assess different systemic risks, tailored to each specific type and nature of risk, including their sources. In addition to further risk assessment measures which will be detailed out in the Code of Practice, the AI Act requires providers to perform the necessary model evaluations, in particular prior to its first placing on the market, including conducting and documenting adversarial testing of the model, also, as appropriate, through internal or independent external testing. The following concerns technical risk assessment measures, including model evaluation and adversarial testing. This is in line with the focus of the Code of Practice Working Group 2 'Risk identification and assessment measures for systemic risks'.\n\nQuestion12. How can the effective implementation of risk assessment measures reflect differences in size and capacity between various providers such as SMEs and start-ups?\n\n700 character(s) maximum\n\nRisk assessment measures should be developed in collaboration between actors of all sizes and allowing participation of affected stakeholders outside of the largest developers. This both ensures that the more meaningful risks are prioritised and allows smaller companies to follow established and scientifically validated standards, rather than having each actor reproduce work or prioritise risks at their own discretion without input from the most affected parties. This ensures that SMEs and start-ups can fully participate in responsible development.\n\n- 13. In the current state of the art , which specific risk assessment measures should, in your view, general-purpose AI model providers take to effectively assess systemic risks along the entire model lifecycle, in addition to evaluation and testing?\n\nPlease indicate to what extent you agree that providers should take the risk assessment\n\n\n\nmeasures from the list. You can add additional measures and provide details on any of the measures, such as what is required for measures to be effective in practice.\n\n| Potential risk assessment measures | Strongly agree | Somewh at agree | Neither agree nor disagree | Disagree | I don't know |\n|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------------|------------------------------|------------|----------------|\n| Determining risk thresholds and risk tolerance , incl. acceptable levels of risks and capabilities for model development and deployment, and respective quantification of risk severity and probability | | X | | | |\n| Forecasting model capabilities and risks before and during model development | | | X | | |\n| Continuous monitoring for emergence of risks , including data from users, relevant stakeholders, incident databases or similar | | | X | | |\n| Determining effectiveness of risk mitigation measures | | X | | | |\n| Safety cases to demonstrate that the model does not exceed maximum risk thresholds | | X | | | |\n| Aggregate risk assessment before model development | | X | | | |\n| Aggregate risk assessment before model deployment | | X | | | |\n| Aggregate risk assessment along the entire model lifecycle | | X | | | |\n\n\n\n\n\nThird-party involvement in risk assessment, for example, related to inspections of training data, models or internal governance", + "char_slice": [ + 25347, + 36038 + ] + }, + { + "headings": [ + "## Hugging Face Comments on the EU AI Office's Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI", + "## About Hugging Face", + "## Section 1. General-purpose AI models: transparency and copyright-related rules", + "## B. Technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities", + "## C. Policy to respect Union copyright law", + "## 700 character(s) maximum", + "## Section 2. General-purpose AI models with systemic risk: risk taxonomy, assessment and mitigation", + "## A. Risk taxonomy", + "## Systemic risk from model malfunctions", + "## Systemic risk from malicious use", + "## Other systemic risks, with reasonably foreseeable negative effects on", + "## B. Risk identification and assessment measures" + ], + "markdown": "------------------|------------|----------------|\n| Determining risk thresholds and risk tolerance , incl. acceptable levels of risks and capabilities for model development and deployment, and respective quantification of risk severity and probability | | X | | | |\n| Forecasting model capabilities and risks before and during model development | | | X | | |\n| Continuous monitoring for emergence of risks , including data from users, relevant stakeholders, incident databases or similar | | | X | | |\n| Determining effectiveness of risk mitigation measures | | X | | | |\n| Safety cases to demonstrate that the model does not exceed maximum risk thresholds | | X | | | |\n| Aggregate risk assessment before model development | | X | | | |\n| Aggregate risk assessment before model deployment | | X | | | |\n| Aggregate risk assessment along the entire model lifecycle | | X | | | |\n\n\n\n\n\nThird-party involvement in risk assessment, for example, related to inspections of training data, models or internal governance\n\nX\n\nYour comments\n\n700 character(s) maximum\n\nThird-party involvement in risk assessment is essential, particularly for inspections of training data, models, or internal governance. In order for this feedback to be meaningful, the entire development cycle of GPAIs should be sufficiently meaningfully documented, allowing external stakeholders to direct their attention to the most relevant development choices. This transparency may be supported through documents like governance cards [https://arxiv.org/abs/2312.03872] to allow continuous external input.\n\n- 14. Please provide links to relevant material on state-of-the-art risk assessment measures, such as model cards, data sheets, templates or other publications.\n\nEvaluating the Social Impact of Generative AI Systems in Systems and Society https://arxiv.org/abs/2306.05949\n\nBias Preservation in Machine Learning: The Legality of Fairness Metrics Under EU\n\nNon-Discrimination Law https://researchrepository.wvu.edu/wvlr/vol123/iss3/4/ The BigCode Project Governance Card: https://arxiv.org/abs/2312.03872\n\n15. In the current state of the art, which specific practices related to model evaluations should, in your view, general-purpose AI model providers take with a view to identifying and mitigating systemic risks?\n\nModel evaluations can include various techniques, such as benchmarks and automated tests, red teaming and adversarial testing including stress testing and boundary testing, white-box evaluations with model explanation and interpretability techniques, and sociotechnical evaluations like field testing, user studies or uplift studies.\n\nPlease indicate to what extent you agree that providers should implement the practice from the list. You can add additional practices and provide details on any of the practices. You can also indicate which model evaluation techniques listed above or which other techniques can reliably assess which specific systemic risks.\n\n| Potential evaluation practices | Stron | Some what | Neither agree nor | Disagr | I don't know |\n|----------------------------------|---------|-------------|---------------------|----------|----------------|\n| | gly | | | ee | |\n| | agree | agree | | | |\n| | | | disagree | | |\n\n\n\n| Performing evaluations at several checkpoints throughout the model lifecycle, in particular during development and prior to internal deployment | X | | |\n|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----|----|----|\n| Performing evaluations at several checkpoints throughout the model lifecycle, in particular when the model risk profile changes such as with access to tools or with different release strategies | | X | |\n| Ensuring evaluations inform model deployment in real-world conditions | X | | |\n| Ensuring evaluations provide insights into the degree to which a model introduces or exacerbates risks | | | X |\n| Using non-public model evaluations , as appropriate | | X | |\n| Involve independent external evaluators , including with appropriate levels of access to the model and related information | | X | |\n| Involve affected persons , to understand effects of human interactions with a particular model over time | | X | |\n| Documenting evaluation strategies and results | X | | |\n\n\n\n| Reporting evaluation strategies and results publicly , as appropriate | X | |\n|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----|----|\n| Reporting evaluation strategies and results to selected authorities and administrative bodies , as appropriate, including sensitive evaluation results | | X |\n| Continuously evaluate and improve evaluation strategies based on information from risk assessment and mitigation measures, including from incidents and near-misses | X | |\n\nYour comments 700 character(s) maximum\n\nDocumentation/Reporting are core to ensuring accountability, the appropriateness of evaluations, and the development of new evaluation strategies.\n\nOrganisations who release open models, especially academic and SMEs, have different constraints and might not be able to hire external evaluators or report results privately. However, as models are de facto accessible to third parties, open source models as defined in the AI Act should be assumed to comply with these requirements.\n\nNon-public evaluations should only be used to avoid data contamination, as transparency is necessary to build trust and accountability. Involving affected persons should be done with care.\n\n- 16. Please provide links to relevant material on state-of-the-art risk assessment measures, such as model cards, data sheets, templates or other publications.\n\nOn the importance of reporting evaluation results in detail: Burnell et al.: Rethink reporting of evaluation results in AI; Aggregate metrics and lack of access to results limit understanding https://www.science.org/doi/10.1126/science.adf6369 Summary: https://montrealethics.ai/rethink-reporting-of-evaluation-results-in-ai/\n\nOn data contamination in evaluation data: Balloccu et al.: Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in\n\n\n\nClosed-Source LLMs https://arxiv.org/abs/2402.03927\n\nJacovi et al.: Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks https://aclanthology.org/2023.emnlp-main.308.pdf\n\nOn involving affected persons:\n\nSloane et al.: Participation Is not a Design Fix for Machine Learning\n\nhttps://dl.acm.org/doi/pdf/10.1145/3551624.3555285\n\nBirhane et al.: Power to the People? Opportunities and Challenges for Participatory AI https://arxiv.org/pdf/2209.07572\n\n17. What are the greatest challenges that a general-purpose AI model provider could face in implementing risk assessment measures, including model evaluations? 700 character(s) maximum\n\nWhile general-purpose AI model providers have access to significant technical knowledge and computational resources, they typically lack the expertise necessary to address social and economic risks of deploying models at scale - especially for risks that depend not just on properties of the model, but on its deployment and commercialisation formats.\n\nEngaging external stakeholders and affected communities is essential for identifying many of the most current and acute risks of the technology, but care must be taken to ensure that", + "char_slice": [ + 33396, + 43789 + ] + }, + { + "headings": [ + "## Hugging Face Comments on the EU AI Office's Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI", + "## About Hugging Face", + "## Section 1. General-purpose AI models: transparency and copyright-related rules", + "## B. Technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities", + "## C. Policy to respect Union copyright law", + "## 700 character(s) maximum", + "## Section 2. General-purpose AI models with systemic risk: risk taxonomy, assessment and mitigation", + "## A. Risk taxonomy", + "## Systemic risk from model malfunctions", + "## Systemic risk from malicious use", + "## Other systemic risks, with reasonably foreseeable negative effects on", + "## B. Risk identification and assessment measures" + ], + "markdown": "covi et al.: Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks https://aclanthology.org/2023.emnlp-main.308.pdf\n\nOn involving affected persons:\n\nSloane et al.: Participation Is not a Design Fix for Machine Learning\n\nhttps://dl.acm.org/doi/pdf/10.1145/3551624.3555285\n\nBirhane et al.: Power to the People? Opportunities and Challenges for Participatory AI https://arxiv.org/pdf/2209.07572\n\n17. What are the greatest challenges that a general-purpose AI model provider could face in implementing risk assessment measures, including model evaluations? 700 character(s) maximum\n\nWhile general-purpose AI model providers have access to significant technical knowledge and computational resources, they typically lack the expertise necessary to address social and economic risks of deploying models at scale - especially for risks that depend not just on properties of the model, but on its deployment and commercialisation formats.\n\nEngaging external stakeholders and affected communities is essential for identifying many of the most current and acute risks of the technology, but care must be taken to ensure that their involvement is not exploitative and that they benefit from the process rather than being used solely for the improvement of AI models and systems.\n\n## C. Technical risk mitigation\n\nCodes of Practice should also be focused on specific risk mitigation measures for general-purpose AI models with systemic risk. Following the risk taxonomy, appropriate measures could be applied to mitigate different systemic risks, tailored to each specific type and nature of risk, including their sources.\n\nThe following concerns technical risk mitigation measures, including cybersecurity protection for the general-purpose AI model and the physical infrastructure of the model. Measures can relate to model design, development or deployment.\n\nThis is in line with the focus of the Code of Practice Working Group 3 'Risk mitigation measures for systemic risks'.\n\nQuestion18. How can the effective implementation of technical risk mitigation measures reflect differences in size and capacity between various providers such as SMEs and start-ups? 700 character(s) maximum\n\nSMEs, start-ups, and open-source developers can implement technical risk mitigation by adhering to established good practices in AI model development. These interventions can be both more robust and more accessible than post-hoc interventions based on, e.g., safety fine-tuning or input filtering. Prioritising safety by design [1] and emphasising transparency\n\n\n\nrequirements help ensure that risk management is more accessible and achievable for providers of all sizes.\n\n- [1] https://info.thorn.org/hubfs/thorn-safety-by-design-for-generative-AI.pdf\n- 19. In the current state of the art, which specific technical risk mitigation measures should, in your view, general-purpose AI model providers take to effectively mitigate systemic risks along the entire model lifecycle, in addition to cybersecurity protection?\n\nPlease indicate to what extent you agree that providers should take the measures from the list. You can add additional measures and provide details on any of the measures, such as what is required for measures to be effective in practice.\n\n| Potential technical risk assessment measures | Strongly agree | Somewh at agree | Neither agree nor disagree | Disa gree | I don't know |\n|-------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------------|------------------------------|-------------|----------------|\n| Data governance such as data selection, cleaning, quality control | X | | | | |\n| Model design and development to achieve an appropriate level of trustworthiness characteristics such as model reliability, fairness or security | | | X | | |\n| Fine-tuning for trustworthiness and alignment such as through Reinforcement Learning from Human Feedback (RLHF) or Constitutional AI | | | X | | |\n| Unlearning techniques such as to remove specific harmful capabilities from models | | | | X | |\n\n\n\n| Technical deployment guardrails , such as content and other filters, capability restrictions, fine-tuning restrictions or monitoring-based restrictions in case of misuse by users | | | X |\n|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----|----|-----|\n| Mitigation measures relating to the model architecture, components, access to tools or model autonomy | X | | |\n| Detection, labelling and other measures related to AI-generated or manipulated content | | X | |\n| Regular model updates , including security updates | | | X |\n| Measuring model performance on an ongoing basis | | | X |\n| Identification and mitigation of model misuse | | X | |\n| Access control to tools and levels of model autonomy | | | X |\n\nYour comments 700 character(s) maximum\n\nReinforcement fine-tuning reduces unwanted model responses, but is brittle towards intentional misuse. Technical deployment guardrails, while helpful, have inherent limitations and must be thoughtfully implemented, taking into account deployment context. For open models, guardrails are often controlled by the deployer, not the developer. Dataset-level measures, like data governance, consistently mitigate risks and apply across both API deployment and open source. We also broadly caution against interventions that are beyond the state of the art and are not scientifically validated, such as unlearning or requiring fully reliable watermarking across modalities.\n\n\n\n- 20. Please provide links to relevant material on state-of-the-art technical risk mitigation practices, such as model cards, data sheets, templates or other publications.\n- 21. What are the greatest challenges that a general-purpose AI provider could face in implementing technical risk mitigation measures?\n\nPreventing misuse of models when developing them:\n\nKaffee et al.: Thorny Roses: Investigating the Dual Use Dilemma in Natural Language Processing https://aclanthology.org/2023.findings-emnlp.932/\n\nJailbreaking of closed models:\n\nZou et al.: Universal and Transferable Adversarial Attacks on Aligned Language Models https://llm-attacks.org/ and https://arxiv.org/abs/2307.15043\n\nLu et al.: Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models\n\nhttps://openaccess.thecvf.com/content/ICCV2023/papers/Lu\\_Set-level\\_Guidance\\_Attack\\_Bo osting\\_Adversarial\\_Transferability\\_of\\_Vision-Language\\_Pre-training\\_Models\\_ICCV\\_2023\\_pape r.pdf\n\n700 character(s) maximum\n\nWhen deploying novel GPAI models, it is impossible to predict all possible misuses of the models. Therefore, adapting models for specific settings makes it easier to develop context-specific risk mitigation techniques\n\n[https://ieeexplore.ieee.org/abstract/document/1507050]. With wider access to the models through open models, domain experts are able to test for specific risks. It also requires updating of the risk mitigation strategies as they're developed.\n\nRequiring open model providers to track all model reuse is technically infeasible as well as undesirable, as it restricts reuse possibilities for startups and research initiatives, limiting innovation and further development.\n\n## D. Internal risk management and governance for general-purpose AI model providers\n\nThe following concerns policies and procedures to operationalise risk management in internal governance of general-purpose AI model providers, including keeping track of, documenting, and reporting serious incidents and possible corrective measures.", + "char_slice": [ + 42609, + 51820 + ] + }, + { + "headings": [ + "## Hugging Face Comments on the EU AI Office's Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI", + "## About Hugging Face", + "## Section 1. General-purpose AI models: transparency and copyright-related rules", + "## B. Technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities", + "## C. Policy to respect Union copyright law", + "## 700 character(s) maximum", + "## Section 2. General-purpose AI models with systemic risk: risk taxonomy, assessment and mitigation", + "## A. Risk taxonomy", + "## Systemic risk from model malfunctions", + "## Systemic risk from malicious use", + "## Other systemic risks, with reasonably foreseeable negative effects on", + "## B. Risk identification and assessment measures", + "## C. Technical risk mitigation", + "## D. Internal risk management and governance for general-purpose AI model providers" + ], + "markdown": "Guidance\\_Attack\\_Bo osting\\_Adversarial\\_Transferability\\_of\\_Vision-Language\\_Pre-training\\_Models\\_ICCV\\_2023\\_pape r.pdf\n\n700 character(s) maximum\n\nWhen deploying novel GPAI models, it is impossible to predict all possible misuses of the models. Therefore, adapting models for specific settings makes it easier to develop context-specific risk mitigation techniques\n\n[https://ieeexplore.ieee.org/abstract/document/1507050]. With wider access to the models through open models, domain experts are able to test for specific risks. It also requires updating of the risk mitigation strategies as they're developed.\n\nRequiring open model providers to track all model reuse is technically infeasible as well as undesirable, as it restricts reuse possibilities for startups and research initiatives, limiting innovation and further development.\n\n## D. Internal risk management and governance for general-purpose AI model providers\n\nThe following concerns policies and procedures to operationalise risk management in internal governance of general-purpose AI model providers, including keeping track of, documenting, and reporting serious incidents and possible corrective measures.\n\nThis is in line with the focus of the Code of Practice Working Group 4 'Internal risk management and governance for general-purpose AI model providers'.\n\n- 22. How can the effective implementation of internal risk management and governance measures reflect differences in size and capacity between various providers such as SMEs and start-ups?\n\n700 character(s) maximum\n\n\n\nClear open and accessible standards that can be followed without involving external organisations are much more accessible to SMEs and start-ups. Risk evaluation should avoid over-relying on mandatory external audits besides the one conducted through the AI Office or similar entities, and instead focus on clear, straightforward guidelines that small teams or individual developers can easily follow. The Code of Practice should not impose requirements that necessitate dedicated personnel, recognising the constraints faced by smaller entities.\n\n## Links to relevant material\n\nMarino et al.: Compliance Cards: Automated EU AI Act Compliance Analyses amidst a Complex AI Supply Chain https://arxiv.org/abs/2406.14758\n\nOn the limitations of AI audits Birhane et al.: AI auditing: The Broken Bus on the Road to AI Accountability https://arxiv.org/abs/2401.14462\n\nRaji et al.: Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance https://dl.acm.org/doi/pdf/10.1145/3514094.3534181\n\nCostanza-Chock et al.: Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem https://arxiv.org/pdf/2310.02521\n\n23. In the current state of the art, which specific internal risk management and governance measures should, in your view, general-purpose AI model providers take to effectively mitigate systemic risks along the entire model lifecycle, in addition to serious incident reporting?\n\nPlease indicate to what extent you agree that providers should take the measures from the list. You can add additional measures and provide details on any of the measures, such as what is required for measures to be effective in practice.\n\n| Potential internal risk management and governance measures | Strongl y agree | Somew hat agree | Neither agree nor disagree | Disa gree | I don't know |\n|---------------------------------------------------------------------------------------------------------------------|-------------------|-------------------|------------------------------|-------------|----------------|\n| Risk management framework across the model lifecycle | | X | | | |\n| Internal independent oversight functions in a transparent governance structure, such as related to risks and ethics | | | X | | |\n\n\n\n| Traceability in relation to datasets, processes, and decisions made during model development | X | |\n|-----------------------------------------------------------------------------------------------------------------------------|-----|----|\n| Ensuring that staff are familiar with their duties and the organisation's risk management practices | | X |\n| Responsible scaling policies | | X |\n| Acceptable use policies | X | |\n| Whistleblower protections | X | |\n| Internal resource allocation towards risk assessment and mitigation measures as well as research to mitigate systemic risks | | X |\n| Robust security controls including physical security, cyber security and information security | | X |\n| External accountability measures such as third-party audits, model or aggregated data access for researchers | X | |\n| Other collaborations and involvements of a diverse set of stakeholders , including impacted communities | | X |\n| Responsible release practices including staged release, particularly before open-sourcing a model with systemic risk | | X |\n\n\n\n| Transparency reports such as model cards, system cards or data sheets | X | | |\n|----------------------------------------------------------------------------------------------------------|-----|----|----|\n| Human oversight mechanisms | | X | |\n| Know-your-customer practices | | X | |\n| Logging, reporting and follow-up of near-misses along the lifecycle | | | X |\n| Measures to mitigate and remediate deployment issues and vulnerabilities | | X | |\n| Complaints handling and redress mechanisms, such as bug bounty programs | X | | |\n| Mandatory model updating policies and limit on maximum model availability | | | X |\n| Third-party and user discovery mechanisms and reporting related to deployment issues and vulnerabilities | | X | |\n\n- 24. Please provide links to relevant material on state-of-the-art governance risk mitigation practices, such as model cards, data sheets, templates or other publications.\n\nComplaints handling and redress mechanisms:\n\nCattell et al.: Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities https://arxiv.org/abs/2402.07039\n\nStaged releases:\n\nSolaiman: The Gradient of Generative AI Release: Methods and Considerations https://arxiv.org/abs/2302.04844\n\nOn involving affected communities:\n\nSloane et al.: Participation Is not a Design Fix for Machine Learning\n\n\n\nhttps://dl.acm.org/doi/pdf/10.1145/3551624.3555285\n\nBirhane et al.: Power to the People? Opportunities and Challenges for Participatory AI https://arxiv.org/pdf/2209.07572\n\n25. What are the greatest challenges that a general-purpose AI provider could face in implementing governance risk mitigation measures? 700 character(s) maximum\n\nSmaller providers, such as SMEs and startups, often lack the financial and human resources needed to establish dedicated governance structures and devise their own risk management frameworks from scratch. The rapidly changing nature of AI and its applications requires governance measures to continuously adapt to new risks and developments. Meaningfully involving diverse stakeholders, including impacted communities, in governance practices requires effective communication and collaboration. Ease of implementation by smaller actors and of adaptation to changing conditions for all can be greatly facilitated by direct collaboration between actors.\n\n## Section 3. Reviewing and monitoring of the General-Purpose AI Code of Practice\n\nThe process of drawing-up the first Code of Practice will start immediately after the AI Act enters into force and will last for 9 months, in view of enabling providers of general-purpose AI models to demonstrate compliance on time. The AI Office shall aim to ensure that the Code of Practice clearly sets out their specific objectives and contains commitments or measures, including key performance indicators as appropriate, to ensure the achievement of those objectives. The AI Office shall aim to ensure that participants to the Code", + "char_slice": [ + 50642, + 59661 + ] + }, + { + "headings": [ + "## Hugging Face Comments on the EU AI Office's Multi-stakeholder Consultation FUTURE-PROOF AI ACT: TRUSTWORTHY GENERAL-PURPOSE AI", + "## About Hugging Face", + "## Section 1. General-purpose AI models: transparency and copyright-related rules", + "## B. Technical documentation by general-purpose AI model providers to the AI Office and the national competent authorities", + "## C. Policy to respect Union copyright law", + "## 700 character(s) maximum", + "## Section 2. General-purpose AI models with systemic risk: risk taxonomy, assessment and mitigation", + "## A. Risk taxonomy", + "## Systemic risk from model malfunctions", + "## Systemic risk from malicious use", + "## Other systemic risks, with reasonably foreseeable negative effects on", + "## B. Risk identification and assessment measures", + "## C. Technical risk mitigation", + "## D. Internal risk management and governance for general-purpose AI model providers", + "## Links to relevant material", + "## Section 3. Reviewing and monitoring of the General-Purpose AI Code of Practice" + ], + "markdown": "xiv.org/pdf/2209.07572\n\n25. What are the greatest challenges that a general-purpose AI provider could face in implementing governance risk mitigation measures? 700 character(s) maximum\n\nSmaller providers, such as SMEs and startups, often lack the financial and human resources needed to establish dedicated governance structures and devise their own risk management frameworks from scratch. The rapidly changing nature of AI and its applications requires governance measures to continuously adapt to new risks and developments. Meaningfully involving diverse stakeholders, including impacted communities, in governance practices requires effective communication and collaboration. Ease of implementation by smaller actors and of adaptation to changing conditions for all can be greatly facilitated by direct collaboration between actors.\n\n## Section 3. Reviewing and monitoring of the General-Purpose AI Code of Practice\n\nThe process of drawing-up the first Code of Practice will start immediately after the AI Act enters into force and will last for 9 months, in view of enabling providers of general-purpose AI models to demonstrate compliance on time. The AI Office shall aim to ensure that the Code of Practice clearly sets out their specific objectives and contains commitments or measures, including key performance indicators as appropriate, to ensure the achievement of those objectives. The AI Office shall aim to ensure that participants to the Code of Practice report regularly to the AI Office on the implementation of the commitments and the measures taken and their outcomes, including as measured against the key performance indicators as appropriate. Key performance indicators and reporting commitments shall reflect differences in size and capacity between various participants. The AI Office and the Board shall regularly monitor and evaluate the achievement of the objectives of the Code of Practice by the participants and their contribution to the proper application of this Regulation. The AI Office shall, as appropriate, encourage and facilitate the review and adaptation of the Code of Practice.\n\nQuestion26. What are examples of key performance indicators which are, in your view, effective to measure the compliance of participants with the objectives and measures which will be established by the Code of Practice?\n\n700 character(s) maximum\n\nThe Code of Practice aims to ensure that developers demonstrate adequate foresight and\n\n\n\nconsideration. This can be validated in collaboration with external bodies, such as the scientific panel. When alerts arise regarding the use of GPAI models, the panel should evaluate the extent to which the Code of Practice was followed and whether it adequately covered and addressed the issues.\n\nThe Code of Practice is considered successful if it meets the dual goals of seeing extensive and transparent adoption from all categories of developers and of serving as an effective support for assessing practices in the light of stakeholder complaints to support more reliable and trustworthy technology.\n\n- 27. How can key performance indicators and reporting commitments for providers reflect differences in size and capacity between various providers such as SMEs and start-ups? 700 character(s) maximum\n\nKey performance indicators and reporting commitments for providers should reflect differences in size by tailoring complexity and resource requirements to the provider's scale. SMEs and startups could have simpler reporting processes and more flexible timelines, with access to open-source tools to reduce costs. For example, smaller entities may use internal evaluations or peer reviews instead of costly third-party audits. KPIs should focus on core issues like data governance, allowing startups to gradually adopt standards while remaining competitive, fostering innovation, and encouraging diverse ecosystem participation.\n\n28. Which aspects should inform the timing of review and adaptation of the content of the Code of Practice for general-purpose AI models in order to ensure that the state of the art is reflected? This does not necessarily imply a complete review, but can only involve pertinent parts.\n\nPlease rank all relevant aspects from the following list from 1 to 4 (1 being the most important). You can add additional aspects and provide details on any of the aspects or other considerations under \"Specify\".\n\nPre-planned intervals to assess the need for revision: Assessments of whether the content of the Code of Practice for general-purpose AI models needs to be revised or adapted should be pre-planned for specific time intervals.\n\nRank 2\n\nAlerts by independent experts or other stakeholders: Alerts by selected independent experts, such as by the Scientific Panel which will be set up in the AI Act governance structure, or by other stakeholders such as\n\nRank 1\n\n\n\ndownstream providers, academia or civil society should inform a revision of the content of the Code of Practice.\n\nMonitoring and foresight: Independent monitoring and foresight related to the AI ecosystem, technological and market developments, emergence of systemic risks and any other relevant trends, such as related to sources of risks like model autonomy, should inform a revision of the content of the Code of Practice\n\n## Your comments\n\n700 character(s) maximum\n\nThe AI Office should incorporate expert feedback on the Code of Practice and make necessary adjustments based on this input. Nevertheless, regular reviews are essential to determine if updates are needed. While foresight can be a useful tool to analyse current trends, in the context of the fast developing technology of AI it is not a proven tool to implement policy changes.\n\nLink to relevant material\n\nA way of enabling the general public to alert, e.g., the AI Office of model flaws and risks are coordinated flaw disclosures: Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities https://arxiv.org/abs/2402.07039\n\nRank 3", + "char_slice": [ + 58202, + 64199 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2024_EU_GPAI_CoP_1_Resonse.pdf", + "md_content": "## Hugging Face Comments on the First Draft of the GPAI Code of Practice\n\nFirst Draft of the General-Purpose AI Code of Practice published, written by independent experts | Shaping Europe's digital future\n\n## Overall Code of Practice Draft Comments\n\nWhile the Code of Practice Draft addresses some of the core challenges, there are a number of challenges yet to be addressed as well as problematic framing, making it hard to implement the code of practice or to base it on scientific evidence.\n\nWe find the Transparency measures to be the most mature part of the document, and provide feedback here on how to make them more actionable and inclusive of open research and development, we note that for those, as for testing requirements, the level of detail and guidance from the AI Office will make the difference between measures that work for the entire ecosystem and measures that fail all categories of stakeholders except for the largest incumbents. While we appreciate the approach of going from general to specific through subsequent drafts, we urge the writers to provide sufficient opportunity to discuss operational details, especially with a view to making them future-proof.\n\nConversely, the sections on risk taxonomies and assessment require the most work. The current taxonomy focuses on a narrow set of risks that are somewhat disconnected from both the actual likely risks of the technology and in some cases from scientific evidence. Risk taxonomies need to be significantly updated to address the full range of risks initially covered by the AI Act, ensure that development priorities reflect real-world impact, and give sufficient voice to stakeholders outside of the development chain; who have less of a vested interest in how the models are presented and advertised.\n\nSection II: [Working Group 1] Rules for Providers of General-Purpose AI Models\n\nMeasures/Sub-measures Specific Feedback on Section II: [Working Group 1] Transparency\n\nTransparency, Measure 1: Documentation for the AI Office\n\nThe current draft outlines important categories for the documentation. However, the items on data transparency and testing require significant further clarification to ensure sufficient opportunities for working group members to meaningfully contribute to the outcome. Additionally, the role of public documentation or public availability of the code supporting some of the sections of the table, should be taken into further consideration. In cases where sufficient information is available, requirements of reporting to the AI Office should be lifted or alleviated\n\nFor data transparency requirements, we refer to the following proposal as an example of the categories of information that should be shared with the AI Office and preferably publicly disclosed:\n\nhttps://openfuture.eu/blog/sufficiently-detailed-summary-v2-0-of-the-blueprint-for-gpai-trainingdata/\n\nPlease provide suggestions on how to improve this measure\n\nFor both Measure 1 and Measure 2, public disclosure should be encouraged to the extent possible, as it will greatly facilitate the adoption of common good practices. Barring that, the AI Office should provide relevant researchers easy access to the submitted documents. Given the fast pace of technical evolution, sufficient public access is necessary to allow relevant stakeholders and researchers across disciplines to ensure that documentation addresses their needs. Public disclosure is also significantly more accessible to developers of open models. Open release of sufficient documentation on platforms such as Hugging Face or Github should be explicitly described as satisfying the needs of disclosure to both the AI Office and to downstream model providers.\n\nThe section on data transparency is most in need of clarification in the proposed table. In particular, its relationship to the 'sufficiently detailed data summary' (Article 53 (1) (d), recital 107) needs to be clarified as soon as possible, to avoid unnecessary duplication of effort and support more meaningful documentation. We strongly encourage the working groups to provide a detailed proposal in the next draft to support sufficient working group involvement.\n\nThe qualification 'where applicable' should be added to the documentation items about methods of distribution, interaction with external hardware/software, and technical means for integration. In models that leverage open-source software, such as the Hugging Face OSS libraries, that information is often better addressed in the existing software documentation.\n\nThe AUP focus on intended and acceptable use should be preserved in further drafts. Monitoring should be understood to be one tool among others, and not necessarily the most applicable or rights-respecting: developers should explain ' whether , why, and how' they monitor user activity. If they do, the impact on user privacy should include details on the management and re-use of user data.\n\nWhat KPI would you add for this measure?\n\nInformation disclosed to the AI Office should be evaluated based on how well it supports additional research and rule-making work by the AI Office and its partners who may access the data, including its scientific council and civil society representation. Any update mechanism should enable feedback from these stakeholders.\n\n## Transparency, Measure 2: Documentation for downstream providers\n\nComments on Measure 1 also apply to Measure 2.\n\nAdditionally, we would like to see greater clarity about why specific categories are meant to be disclosed to only the AI Office or only downstream providers. In particular, information about the training objective, testing conditions, and inference costs are of relevance to downstream providers, including for assessing fitness of purpose and reliability of systems.\n\nPlease provide suggestions on how to improve this measure\n\nIn the first draft, several categories of information that are often important and sometimes necessary for downstream providers to deploy systems in a safe and secure fashion are currently only noted as having to be disclosed to the AI Office.\n\nIn order to address fitness-for-purpose of AI systems, downstream providers need to assess several properties of the model, including how well its training data represents their activity domain, how closely the model's initial testing conditions match their use case, or whether data they might want to use to test systems for their own applications might have been included in the GPAI's training data at any stage. This information is particularly important for downstream providers whose products may be used in safety-critical domains, including settings that may not be considered high-risk applications under the Act (Article 6 (3)). Additionally, downstream providers who rely on a particular system need some understanding of the production cost of the system, especially in cases where those costs might be temporarily absorbed by the GPAI developers but subject to significant later increase.\n\nInformation disclosed to the AI Office about the sizes of different data sources, the training procedure and objectives, the energy consumption at inference time, and the exact testing procedures should also be made available to downstream providers to enable development that better protects the safety of people affected by the systems.\n\nWhat KPI would you add for this measure?\n\nIn order to ensure that this Measure fulfills the information needs of downstream providers, we ensure establishing a mechanism for downstream providers to submit questions about the GPAI models they use in their commercial or research activity that is registered with the AI Office. Knowing which of these questions are covered by the categories and which are not would be invaluable in shaping the evolution of the transparency requirements.\n\nFor the items listed in the table, how should the Code of Practice provide greater detail?\n\nFor the model architecture and parameters, the draft should outline how providers of systems with API-only access should report the full stack involved, including when multiple models are used within an inference call, or how inputs or outputs to the model may be modified, especially in ways that might have bearing on the downstream provider's intent.\n\nFor the training, testing, and validation data, see our comment above. At a minimum, the documentation should additionally outline for what purposes various datasets are used, data processing steps aimed at improving model safety and security, and contamination analysis between testing and validation datasets and training datasets as available to the developer.\n\nFor the testing process, documentation should ensure that the findings are replicable and comparable across systems, including by providing the exact testing setup and a sufficient subset of the testing dataset to enable external reproduction and 'apples-to-apples' comparison. To that end, the use of public benchmarks for a representative subset of performance and safety measures should be encouraged.\n\n## Measures/Sub-measures Specific Feedback on Section II: [Working Group 1] Copyright-related rules\n\nCopyright-related rules, Measure 3: Put in place copyright policy\n\nPlease explain your rating to this measure\n\nThe Measure as currently proposed presents a significant risk of excluding small and medium actors from participating in GPAI development, while its contribution to making larger actor's practices more rights-respecting and fairer to creators of content under copyright remains uncertain.\n\nSub-Measure 3.1 requires developers to draw up a copyright policy, but does not provide meaningful information about what constitutes a valid such policy. Smaller teams, collaborations, and organizations that work on parts of the development chain in various ways in a distributed fashion, including by leveraging or publishing open models, tools, and datasets, will find this disproportionately difficult without stronger guidelines compared to the few best-resourced developers who own their entire development chain - as the former typically have access to fewer centralized legal resources, a greater diversity of questions to answer, and less ability to absorb the cost of fines if a good-faith effort is deemed insufficient post hoc . Given the role of open and collaborative research and development in supporting both innovation of a greater variety of techniques and investigation into existing technology, providing copyright holders with more clarity and options when defending the value of their content, we strongly encourage future versions of the draft to better address their requirements (and contributions).\n\nSub-Measures 3.2 and 3.3 also require significant further clarification to be manageable by smaller entities and developers working in open and collaborative settings. These submeasures should address in particular information flows between open datasets and developers, clarify whether datasets or models made available under a given license or with a simple signed user agreement constitute a contractual relation for those purposes.\n\nPlease provide suggestions on how to improve this measure\n\nSub-Measure 3.1 should provide significantly more detail on what constitutes a valid copyright policy, and enable actors to adopt such a policy by providing templates that address various development models. While we acknowledge that the development of the Code of Practice is supposed to go from general to specific, we are concerned that waiting until the last draft for a fully detailed proposal will not provide sufficient opportunities for inputs from the most directly affected stakeholders. At the very least, the next draft needs to outline which parts of the 'entire lifecycle' are deemed relevant and should be the focus of the policy, and possible policy items for both closed and open development and sharing.\n\nSub-Measure 3.2 should clarify whether the use of a publicly available dataset released under a content or software license or 'click-through' user agreement is understood as a contract in this context. If that is the case, it should acknowledge that dataset developers may not be available for extensive interactions with each developer and provide guidance for interactions with these mostly static artifacts.\n\nSub-Measure 3.3 requires similar clarification of its scope. We do appreciate the focus on development choices and memorization measures and acknowledgement of the different dynamics for SMEs.\n\n## Copyright-related rules, Sub-Measure 3.1: Draw up and implement a copyright policy\n\nWhile the idea of drawing up and implementing a copyright policy is valuable in principle, it presents significant challenges for SMEs and especially for open-source projects that often lack access to legal counsel. To address this, it is essential to provide a clear template and detailed guidance on what constitutes a satisfactory copyright policy. This should include how the policy's content aligns with the commitments outlined in subsequent sub-measures and what \"implementation\" entails, particularly over the long term. For open-source projects, which may no longer be actively maintained but remain valuable to the AI developer community, the guidance should consider realistic and practical solutions for sustaining compliance without imposing excessive burdens.\n\nPlease provide suggestions on how to improve this sub-measure\n\n- - Provide Templates and Guidelines: Develop a standardised template and detailed guidance for creating a copyright policy. This should clearly outline required elements, such as lifecycle compliance, and explain how the policy integrates with commitments in other sub-measures. Templates should be tailored to the needs of different stakeholders, including SMEs and open-source projects, to reduce the administrative burden.\n- - Simplify Requirements for Open Source Projects Introduce proportional obligations for open-source initiatives, recognizing their resource constraints and decentralised nature. For example, clarify that a policy could focus on the transparency of training datasets and adherence to known opt-out mechanisms rather than requiring ongoing active maintenance when a project is no longer actively developed.\n- - Facilitate Access to Expertise: Provide access to shared resources, such as legal guides or pro bono legal assistance programs, to help SMEs and open-source contributors navigate copyright compliance without the need for in-house legal teams.\n- - Specify Long-Term Maintenance Expectations: Include provisions for how copyright policies should apply to projects no longer actively maintained. For example, policies could include\n\ninstructions or disclaimers for users of the AI models, specifying how to address copyright concerns if no contact point is available.\n\n- - Ensure Alignment with Other Sub-Measures: Clearly distinguish how this sub-measure complements or differs from related requirements in the Code, such as transparency documentation or point-of-contact obligations, to avoid duplication or confusion.\n- - Establish a Sunset Clause for Compliance: For projects or models that are no longer actively maintained, introduce a sunset clause to limit the obligations of contributors, ensuring that the requirements are not indefinite for contributors who have ceased involvement.\n\nWhat KPI would you add for this sub-measure?\n\n- - Number of Templates or Resources Accessed: Number of signatories who use the standardised copyright policy templates or other resources provided (e.g., legal guidelines, FAQs).\n- - Feedback and Improvement Requests: Number of feedback submissions or improvement requests received from developers (particularly SMEs and open-source contributors) regarding the copyright policy framework.\n- - Incident Reports of Copyright Infringement: Number of copyright-related complaints or issues reported that indicate non-compliance with the policy or inadequate implementation.\n\n## Copyright-related rules, Sub-Measure 3.2: Upstream copyright compliance\n\nThe code should clarify that this sub-measure applies only to contracts between data providers and users of datasets, excluding the reuse of freely licensed datasets. This is essential to align with the EU AI Act, which focuses on general-purpose AI models, not datasets. Extending the scope to freely licensed datasets would exceed the regulation's intent and undermine open licensing frameworks, which already address rights reservations. Datasets requiring a license agreement, such as the news corpora as in https://www.nytimes.com/2024/05/22/business/media/openai-news-corp-content-deal.html should be treated differently from freely available, openly licensed data. Users of freely licensed datasets would not have the necessary resources to do a due diligence on all freely licensed datasets, and also this could mean multiple streams of work on due diligence of the same dataset. Applying this measure to freely licensed datasets risks unnecessary burdens, discouraging open data use.\n\nCopyright-related rules, Measure 5: Transparency\n\nCopyright-related rules, Sub-Measure 5.3: Single point of contact and complaint handling\n\nThe sub-measure, as stated, lacks specificity as to the kinds of complaints that developers should prepare to handle. We note that existing complaint mechanisms like DMCA or GDPR-related requests work in great part because they have clear legal grounding, definitions of what constitutes a valid complaint, and prescribed actions to follow up on the complaint. Without those, the designed point of contact will be much less efficient.\n\nAdditionally, for open source projects, many of the models are still available after the funding or organisational support for the project ended. Similar to sub-measure 3.1, it should be clarified here how to proceed when projects end or there is no clear organisational structure in the first place.\n\nPlease provide suggestions on how to improve this sub-measure\n\n- - Provide a template for a unified complaint format, including guidelines on what constitutes sufficient information to identify the validity of the claim, ownership of the work, or grounds for protection of the subject matter.\n- - Given a valid complaint, provide guidance on what constitutes 'appropriate complaint handling procedure', especially in terms of handling of both copies of the work held by the developers and models trained on datasets including copies of the work.\n- - Specify Long-Term Maintenance Expectations: Include provisions for how copyright policies should apply to projects no longer actively maintained. For example, policies could include instructions or disclaimers for users of the AI models, specifying how to address copyright concerns if no contact point is available.\n- - Establish a Sunset Clause for Compliance: For projects or models that are no longer actively maintained, introduce a sunset clause to limit the obligations of contributors, ensuring that the requirements are not indefinite for contributors who have ceased involvement.\n\n## Copyright-related rules, Sub-Measure 5.4: Documentation of data sources and authorisations\n\nThe relationship between this documentation and other data documentation requirements, including for Measures 1 and 2, should be further clarified. This sub-measure should clearly define how these data sources should be documented, to what level of detail they have to be provided, and further extend whether there are other relevant information beside copyright-related questions that should be documented and subsequently made available by AI providers.\n\nSection III: [Working Group 2] Taxonomy of Systemic Risks\n\n## Taxonomy of systemic risks, Measure 6: Taxonomy\n\nWe are deeply concerned by the current focus of the risk taxonomy. The taxonomy in the first draft overrepresents a narrow subset of the risks described in the AI Act; a subset that focuses on the least defined and immediate risks without sufficient scientific grounding or precedent in broader technology. We urge the next draft to bring the focus back to\n\n'reasonably foreseeable negative effects' such as 'major accidents' (Recital 110) or irresponsible deployment in specific contexts.\n\nPlease provide suggestions on how to improve this measure\n\nThe narrow focus of the current taxonomy reflects disproportionate attention to intentional misuse and capabilities as naturally emerging model properties to measure post hoc. A more holistic approach should first recognise that recent examples of the systemic risks considered (e.g. CrowdStrike outage) have stemmed from immature deployment in safety-critical systems, while new 'capabilities' have come from intentional design choices, primarily the inclusion of new kinds of training data.\n\n## Taxonomy of systemic risks, Sub-Measure 6.1: Types of systemic risks\n\nManipulation cannot be measured at the model level without a specific distribution model. CBRN risks are currently remote to inexistent, and requires grounded security research by domain experts, not highly abstracted evaluation by model developers. Automated use of models for research is part of their primary benign use and may at most be considered an accelerating factor for technology development, not a risk. Loss of control is primarily a risk tied to system design, not capabilities.\n\nPlease provide suggestions on how to improve this sub-measure\n\nCyber offence should be replaced by risks to cybersecurity, including through spread of code vulnerabilities. Categories mentioned above should be stricken. Categories of risks should cover a broader set of considerations from the AI Act, including consequences to public health, civic/human rights, and democratic processes stemming from model reliability fairness, as well as measurable risks to economic domains and communities (Rec. 110). See also: https://arxiv.org/abs/2306.05949.\n\nWhat are relevant considerations or criteria to take into account when defining whether a risk is a systemic risk?\n\nSystemic risks included in the taxonomy should be risks that:\n\n- -Have an identified and minimally likely negative impact on a given social system (e.g. education, information ecosystems, journalism, labor, etc.)\n- -Are tied to specific development choices or properties that can be characterized using reproducible, falsifiable, and context-sensitive evaluations\n- -For scientific principles for risk assessment, see e.g.: https://ec.europa.eu/newsroom/dae/redirection/document/110171\n\nBased on these considerations or criteria, which risks should be prioritised for addition to the main taxonomy of systemic risks?\n\nRisks to the security or reliability of critical systems, including access to health, education, government benefits or essential services. Risks to democratic processes through the integrity of journalism or information ecosystems. Risks to the environment through increased dependence on energy-intensive technology.\n\nHow should the taxonomy of systemic risks address AI-generated child sexual abuse material and non-consensual intimate imagery?\n\nAI-generated CSAM constitutes a systemic risk as defined insofar as it puts pressure on existing systems that are designed to protect targets and limit the spread of content. For specific harm mechanisms for CSAM harms, the importance of focusing on deployment and development choices, and the difficulty of anchoring work on 'model capabilities', see e.g. https://info.thorn.org/hubfs/thorn-safety-by-design-for-generative-AI.pdf\n\n## Taxonomy of systemic risks, Sub-Measure 6.2: Nature of systemic risks\n\nOrigin, actors, and intent all require significant further elaboration.\n\nProbability-severity ratio raises questions about who gets to define severity when different stakeholder groups are differentially affected and what constitutes a valid model to assess likelihood. It should also be understood that a risk is inherently defined by a notion of likelihood (a hazard by itself does not constitute a risk)\n\nPlease provide suggestions on how to improve this sub-measure\n\nOrigin should provide a definition of model capabilities. Model distribution should be inclusive of model development and deployment choices.\n\nThe actors driving the risks are currently missing the model developers, who are currently the ones in the strongest position to mitigate (or increase) risks through development choices. In particular, misrepresented or improperly documented models or capabilities are a strong risk factor.\n\n## Taxonomy of systemic risks, Sub-Measure 6.3: Sources of systemic risks\n\nThe sub-measure includes several notions that are not supported by evidence or that do not correspond to a quantifiable or falsifiable measure, including: self-replication, 'power-seeking', 'collusion', or a notion of alignment with universal 'human values'.\n\nPlease provide suggestions on how to improve this sub-measure\n\nSeveral inclusions in the current draft are very relevant and should be further expanded, including notably considerations of number of business users and technology readiness, lack of reliability and security including biases, and confabulation in deployment settings with high precision requirements.\n\nSection IV: [Working Groups 2/3/4] Rules for Providers of General-Purpose AI Models with Systemic Risk\n\nMeasures/Sub-measures Specific Feedback on Section IV: [Working Group 2] Risk assessment for providers of General-Purpose AI Models with Systemic Risk\n\nRisk assessment, Measure 8: Risk identification\n\nThe current version of the draft makes model developers responsible for identifying risks, selecting evaluations, and interpreting those evaluations. This raises two main concerns. While developers have the best understanding of some of the technical aspects of their models, they typically have neither the broader expertise (sociological, domain expertise, etc...) nor the legitimacy to decide which risks should be prioritized, especially when choices affect stakeholder groups differently.\n\nPlease provide suggestions on how to improve this sub-measure\n\nRisk identification should occur outside of the development chain. For developers training models with systemic risks, an external review system, possibly with the AI Office scientific advisory board, should determine which risks should be evaluated based on parts of the information described in Measures 1 and 2 that can be disclosed ahead of deployment. We note that openly released models in particular provide additional opportunities for external stakeholders and experts to identify risks.", + "chunks": [ + { + "headings": [ + "Hugging Face Comments on the First Draft of the GPAI Code of Practice" + ], + "markdown": "## Hugging Face Comments on the First Draft of the GPAI Code of Practice\n\nFirst Draft of the General-Purpose AI Code of Practice published, written by independent experts | Shaping Europe's digital future\n\n## Overall Code of Practice Draft Comments\n\nWhile the Code of Practice Draft addresses some of the core challenges, there are a number of challenges yet to be addressed as well as problematic framing, making it hard to implement the code of practice or to base it on scientific evidence.\n\nWe find the Transparency measures to be the most mature part of the document, and provide feedback here on how to make them more actionable and inclusive of open research and development, we note that for those, as for testing requirements, the level of detail and guidance from the AI Office will make the difference between measures that work for the entire ecosystem and measures that fail all categories of stakeholders except for the largest incumbents. While we appreciate the approach of going from general to specific through subsequent drafts, we urge the writers to provide sufficient opportunity to discuss operational details, especially with a view to making them future-proof.\n\nConversely, the sections on risk taxonomies and assessment require the most work. The current taxonomy focuses on a narrow set of risks that are somewhat disconnected from both the actual likely risks of the technology and in some cases from scientific evidence. Risk taxonomies need to be significantly updated to address the full range of risks initially covered by the AI Act, ensure that development priorities reflect real-world impact, and give sufficient voice to stakeholders outside of the development chain; who have less of a vested interest in how the models are presented and advertised.\n\nSection II: [Working Group 1] Rules for Providers of General-Purpose AI Models\n\nMeasures/Sub-measures Specific Feedback on Section II: [Working Group 1] Transparency\n\nTransparency, Measure 1: Documentation for the AI Office\n\nThe current draft outlines important categories for the documentation. However, the items on data transparency and testing require significant further clarification to ensure sufficient opportunities for working group members to meaningfully contribute to the outcome. Additionally, the role of public documentation or public availability of the code supporting some of the sections of the table, should be taken into further consideration. In cases where sufficient information is available, requirements of reporting to the AI Office should be lifted or alleviated\n\nFor data transparency requirements, we refer to the following proposal as an example of the categories of information that should be shared with the AI Office and preferably publicly disclosed:\n\nhttps://openfuture.eu/blog/sufficiently-detailed-summary-v2-0-of-the-blueprint-for-gpai-trainingdata/\n\nPlease provide suggestions on how to improve this measure\n\nFor both Measure 1 and Measure 2, public disclosure should be encouraged to the extent possible, as it will greatly facilitate the adoption of common good practices. Barring that, the AI Office should provide relevant researchers easy access to the submitted documents. Given the fast pace of technical evolution, sufficient public access is necessary to allow relevant stakeholders and researchers across disciplines to ensure that documentation addresses their needs. Public disclosure is also significantly more accessible to developers of open models. Open release of sufficient documentation on platforms such as Hugging Face or Github should be explicitly described as satisfying the needs of disclosure to both the AI Office and to downstream model providers.\n\nThe section on data transparency is most in need of clarification in the proposed table. In particular, its relationship to the 'sufficiently detailed data summary' (Article 53 (1) (d), recital 107) needs to be clarified as soon as possible, to avoid unnecessary duplication of effort and support more meaningful documentation. We strongly encourage the working groups to provide a detailed proposal in the next draft to support sufficient working group involvement.\n\nThe qualification 'where applicable' should be added to the documentation items about methods of distribution, interaction with external hardware/software, and technical means for integration. In models that leverage open-source software, such as the Hugging Face OSS libraries, that information is often better addressed in the existing software documentation.\n\nThe AUP focus on intended and acceptable use should be preserved in further drafts. Monitoring should be understood to be one tool among others, and not necessarily the most applicable or rights-respecting: developers should explain ' whether , why, and how' they monitor user activity. If they do, the impact on user privacy should include details on the management and re-use of user data.\n\nWhat KPI would you add for this measure?\n\nInformation disclosed to the AI Office should be evaluated based on how well it supports additional research and rule-making work by the AI Office and its partners who may access the data, including its scientific council and civil society representation. Any update mechanism should enable feedback from these stakeholders.\n\n## Transparency, Measure 2: Documentation for downstream providers\n\nComments on Measure 1 also apply to Measure 2.\n\nAdditionally, we would like to see greater clarity about why specific categories are meant to be disclosed to only the AI Office or only downstream providers. In particular, information about the training objective, testing conditions, and inference costs are of relevance to downstream providers, including for assessing fitness of purpose and reliability of systems.\n\nPlease provide suggestions on how to improve this measure\n\nIn the first draft, several categories of information that are often important and sometimes necessary for downstream providers to deploy systems in a safe and secure fashion are currently only noted as having to be disclosed to the AI Office.\n\nIn order to address fitness-for-purpose of AI systems, downstream providers need to assess several properties of the model, including how well its training data represents their activity domain, how closely the model's initial testing conditions match their use case, or whether data they might want to use to test systems for their own applications might have been included in the GPAI's training data at any stage. This information is particularly important for downstream providers whose products may be used in safety-critical domains, including settings that may not be considered high-risk applications under the Act (Article 6 (3)). Additionally, downstream providers who rely on a particular system need some understanding of the production cost of the system, especially in cases where those costs might be temporarily absorbed by the GPAI developers but subject to significant later increase.\n\nInformation disclosed to the AI Office about the sizes of different data sources, the training procedure and objectives, the energy consumption at inference time, and the exact testing procedures should also be made available to downstream providers to enable development that better protects the safety of people affected by the systems.\n\nWhat KPI would you add for this measure?\n\nIn order to ensure that this Measure fulfills the information needs of downstream providers, we ensure establishing a mechanism for downstream providers to submit questions about the GPAI models they use in their commercial or research activity that is registered with the AI Office. Knowing which of these questions are covered by the categories and which are not would be invaluable in shaping the evolution of the transparency requirements.\n\nFor the items listed in the table, how should the Code of Practice provide greater detail?\n\nFor the model architecture and parameters, the draft should outline how providers of systems with API-only access should report the full stack involved, including when multiple models are used within an inference call, or how inputs or outputs to the model may be modified, especially in ways that might have bearing on the downstream provider's intent.\n\nFor the training, testing, and validation data, see our comment above. At a minimum, the documentation should additionally outline for what purposes various datasets are used, data processing steps aimed at improving model safety and security, and contamination analysis between testing and validation datasets and training datasets as available to the developer.\n\nFor the testing process, documentation should ensure that the findings are replicable and comparable across systems, including by providing the exact testing setup and a sufficient subset of the testing dataset to enable external reproduction and 'apples-to-apples' comparison. To that end, the use of public benchmarks for a representative subset of performance and safety measures should be encouraged.\n\n## Measures/Sub-measures Specific Feedback on Section II: [Working Group 1] Copyright-related rules\n\nCopyright-related rules, Measure 3: Put in place copyright policy\n\nPlease explain your rating to this measure\n\nThe Measure as currently proposed presents a significant risk of excluding small and medium actors from participating in GPAI development, while its contribution to making larger actor's practices more rights-respecting and fairer to creators of content under copyright remains uncertain.\n\nSub-Measure 3.1 requires developers to draw up a copyright policy, but does not provide meaningful information about what constitutes a valid such policy. Smaller teams, collaborations, and organizations that work on parts of the development chain in various ways in a distributed fashion, including by leveraging or publishing open models, tools, and datasets, will find this disproportionately difficult without stronger guidelines compared to the few best-resourced developers who own their entire development chain - as the former typically have access to fewer centralized legal resources, a greater diversity of questions to answer, and less ability to absorb the cost of fines if a good-faith effort is deemed insufficient post hoc . Given the role of open and collaborative research and development in supporting both innovation of a greater variety of techniques and investigation into existing technology, providing copyright holders with more clarity and options when defending the value of their content, we strongly encourage future versions of the draft to better address their requirements (and contributions).\n\nSub-Measures 3.2 and 3.3 also require significant further clarification to be manageable by smaller entities and developers working in open and collaborative settings. These submeasures should address in particular information flows between open datasets and developers, clarify whether datasets or models made available under a given license or with a simple signed user agreement constitute a contractual relation for those purposes.\n\nPlease provide suggestions on how to improve this measure\n\nSub-Measure 3.1 should provide significantly more detail on what constitutes a valid copyright policy, and enable actors to adopt such a policy by providing templates that address various development models. While we acknowledge that the development of the Code of Practice is supposed to go from general to specific", + "char_slice": [ + 0, + 11479 + ] + }, + { + "headings": [ + "## Hugging Face Comments on the First Draft of the GPAI Code of Practice", + "## Overall Code of Practice Draft Comments", + "## Transparency, Measure 2: Documentation for downstream providers", + "## Measures/Sub-measures Specific Feedback on Section II: [Working Group 1] Copyright-related rules" + ], + "markdown": "-resourced developers who own their entire development chain - as the former typically have access to fewer centralized legal resources, a greater diversity of questions to answer, and less ability to absorb the cost of fines if a good-faith effort is deemed insufficient post hoc . Given the role of open and collaborative research and development in supporting both innovation of a greater variety of techniques and investigation into existing technology, providing copyright holders with more clarity and options when defending the value of their content, we strongly encourage future versions of the draft to better address their requirements (and contributions).\n\nSub-Measures 3.2 and 3.3 also require significant further clarification to be manageable by smaller entities and developers working in open and collaborative settings. These submeasures should address in particular information flows between open datasets and developers, clarify whether datasets or models made available under a given license or with a simple signed user agreement constitute a contractual relation for those purposes.\n\nPlease provide suggestions on how to improve this measure\n\nSub-Measure 3.1 should provide significantly more detail on what constitutes a valid copyright policy, and enable actors to adopt such a policy by providing templates that address various development models. While we acknowledge that the development of the Code of Practice is supposed to go from general to specific, we are concerned that waiting until the last draft for a fully detailed proposal will not provide sufficient opportunities for inputs from the most directly affected stakeholders. At the very least, the next draft needs to outline which parts of the 'entire lifecycle' are deemed relevant and should be the focus of the policy, and possible policy items for both closed and open development and sharing.\n\nSub-Measure 3.2 should clarify whether the use of a publicly available dataset released under a content or software license or 'click-through' user agreement is understood as a contract in this context. If that is the case, it should acknowledge that dataset developers may not be available for extensive interactions with each developer and provide guidance for interactions with these mostly static artifacts.\n\nSub-Measure 3.3 requires similar clarification of its scope. We do appreciate the focus on development choices and memorization measures and acknowledgement of the different dynamics for SMEs.\n\n## Copyright-related rules, Sub-Measure 3.1: Draw up and implement a copyright policy\n\nWhile the idea of drawing up and implementing a copyright policy is valuable in principle, it presents significant challenges for SMEs and especially for open-source projects that often lack access to legal counsel. To address this, it is essential to provide a clear template and detailed guidance on what constitutes a satisfactory copyright policy. This should include how the policy's content aligns with the commitments outlined in subsequent sub-measures and what \"implementation\" entails, particularly over the long term. For open-source projects, which may no longer be actively maintained but remain valuable to the AI developer community, the guidance should consider realistic and practical solutions for sustaining compliance without imposing excessive burdens.\n\nPlease provide suggestions on how to improve this sub-measure\n\n- - Provide Templates and Guidelines: Develop a standardised template and detailed guidance for creating a copyright policy. This should clearly outline required elements, such as lifecycle compliance, and explain how the policy integrates with commitments in other sub-measures. Templates should be tailored to the needs of different stakeholders, including SMEs and open-source projects, to reduce the administrative burden.\n- - Simplify Requirements for Open Source Projects Introduce proportional obligations for open-source initiatives, recognizing their resource constraints and decentralised nature. For example, clarify that a policy could focus on the transparency of training datasets and adherence to known opt-out mechanisms rather than requiring ongoing active maintenance when a project is no longer actively developed.\n- - Facilitate Access to Expertise: Provide access to shared resources, such as legal guides or pro bono legal assistance programs, to help SMEs and open-source contributors navigate copyright compliance without the need for in-house legal teams.\n- - Specify Long-Term Maintenance Expectations: Include provisions for how copyright policies should apply to projects no longer actively maintained. For example, policies could include\n\ninstructions or disclaimers for users of the AI models, specifying how to address copyright concerns if no contact point is available.\n\n- - Ensure Alignment with Other Sub-Measures: Clearly distinguish how this sub-measure complements or differs from related requirements in the Code, such as transparency documentation or point-of-contact obligations, to avoid duplication or confusion.\n- - Establish a Sunset Clause for Compliance: For projects or models that are no longer actively maintained, introduce a sunset clause to limit the obligations of contributors, ensuring that the requirements are not indefinite for contributors who have ceased involvement.\n\nWhat KPI would you add for this sub-measure?\n\n- - Number of Templates or Resources Accessed: Number of signatories who use the standardised copyright policy templates or other resources provided (e.g., legal guidelines, FAQs).\n- - Feedback and Improvement Requests: Number of feedback submissions or improvement requests received from developers (particularly SMEs and open-source contributors) regarding the copyright policy framework.\n- - Incident Reports of Copyright Infringement: Number of copyright-related complaints or issues reported that indicate non-compliance with the policy or inadequate implementation.\n\n## Copyright-related rules, Sub-Measure 3.2: Upstream copyright compliance\n\nThe code should clarify that this sub-measure applies only to contracts between data providers and users of datasets, excluding the reuse of freely licensed datasets. This is essential to align with the EU AI Act, which focuses on general-purpose AI models, not datasets. Extending the scope to freely licensed datasets would exceed the regulation's intent and undermine open licensing frameworks, which already address rights reservations. Datasets requiring a license agreement, such as the news corpora as in https://www.nytimes.com/2024/05/22/business/media/openai-news-corp-content-deal.html should be treated differently from freely available, openly licensed data. Users of freely licensed datasets would not have the necessary resources to do a due diligence on all freely licensed datasets, and also this could mean multiple streams of work on due diligence of the same dataset. Applying this measure to freely licensed datasets risks unnecessary burdens, discouraging open data use.\n\nCopyright-related rules, Measure 5: Transparency\n\nCopyright-related rules, Sub-Measure 5.3: Single point of contact and complaint handling\n\nThe sub-measure, as stated, lacks specificity as to the kinds of complaints that developers should prepare to handle. We note that existing complaint mechanisms like DMCA or GDPR-related requests work in great part because they have clear legal grounding, definitions of what constitutes a valid complaint, and prescribed actions to follow up on the complaint. Without those, the designed point of contact will be much less efficient.\n\nAdditionally, for open source projects, many of the models are still available after the funding or organisational support for the project ended. Similar to sub-measure 3.1, it should be clarified here how to proceed when projects end or there is no clear organisational structure in the first place.\n\nPlease provide suggestions on how to improve this sub-measure\n\n- - Provide a template for a unified complaint format, including guidelines on what constitutes sufficient information to identify the validity of the claim, ownership of the work, or grounds for protection of the subject matter.\n- - Given a valid complaint, provide guidance on what constitutes 'appropriate complaint handling procedure', especially in terms of handling of both copies of the work held by the developers and models trained on datasets including copies of the work.\n- - Specify Long-Term Maintenance Expectations: Include provisions for how copyright policies should apply to projects no longer actively maintained. For example, policies could include instructions or disclaimers for users of the AI models, specifying how to address copyright concerns if no contact point is available.\n- - Establish a Sunset Clause for Compliance: For projects or models that are no longer actively maintained, introduce a sunset clause to limit the obligations of contributors, ensuring that the requirements are not indefinite for contributors who have ceased involvement.\n\n## Copyright-related rules, Sub-Measure 5.4: Documentation of data sources and authorisations\n\nThe relationship between this documentation and other data documentation requirements, including for Measures 1 and 2, should be further clarified. This sub-measure should clearly define how these data sources should be documented, to what level of detail they have to be provided, and further extend whether there are other relevant information beside copyright-related questions that should be documented and subsequently made available by AI providers.\n\nSection III: [Working Group 2] Taxonomy of Systemic Risks\n\n## Taxonomy of systemic risks, Measure 6: Taxonomy\n\nWe are deeply concerned by the current focus of the risk taxonomy. The taxonomy in the first draft overrepresents a narrow subset of the risks described in the AI Act; a subset that focuses on the least defined and immediate risks without sufficient scientific grounding or precedent in broader technology. We urge the next draft to bring the focus back to\n\n'reasonably foreseeable negative effects' such as 'major accidents' (Recital 110) or irresponsible deployment in specific contexts.\n\nPlease provide suggestions on how to improve this measure\n\nThe narrow focus of the current taxonomy reflects disproportionate attention to intentional misuse and capabilities as naturally emerging model properties to measure post hoc. A more holistic approach should first recognise that recent examples of the systemic risks considered (e.g. CrowdStrike outage) have stemmed from immature deployment in safety-critical systems, while new 'capabilities' have come from intentional design choices, primarily the inclusion of new kinds of training data.\n\n## Taxonomy of systemic risks, Sub-Measure 6.1: Types of systemic risks\n\nManipulation cannot be measured at the model level without a specific distribution model. CBRN risks are currently remote to inexistent, and requires grounded security research by domain experts, not highly abstracted evaluation by model developers", + "char_slice": [ + 9998, + 21084 + ] + }, + { + "headings": [ + "## Hugging Face Comments on the First Draft of the GPAI Code of Practice", + "## Overall Code of Practice Draft Comments", + "## Transparency, Measure 2: Documentation for downstream providers", + "## Measures/Sub-measures Specific Feedback on Section II: [Working Group 1] Copyright-related rules", + "## Copyright-related rules, Sub-Measure 3.1: Draw up and implement a copyright policy", + "## Copyright-related rules, Sub-Measure 3.2: Upstream copyright compliance", + "## Copyright-related rules, Sub-Measure 5.4: Documentation of data sources and authorisations", + "## Taxonomy of systemic risks, Measure 6: Taxonomy", + "## Taxonomy of systemic risks, Sub-Measure 6.1: Types of systemic risks" + ], + "markdown": "We are deeply concerned by the current focus of the risk taxonomy. The taxonomy in the first draft overrepresents a narrow subset of the risks described in the AI Act; a subset that focuses on the least defined and immediate risks without sufficient scientific grounding or precedent in broader technology. We urge the next draft to bring the focus back to\n\n'reasonably foreseeable negative effects' such as 'major accidents' (Recital 110) or irresponsible deployment in specific contexts.\n\nPlease provide suggestions on how to improve this measure\n\nThe narrow focus of the current taxonomy reflects disproportionate attention to intentional misuse and capabilities as naturally emerging model properties to measure post hoc. A more holistic approach should first recognise that recent examples of the systemic risks considered (e.g. CrowdStrike outage) have stemmed from immature deployment in safety-critical systems, while new 'capabilities' have come from intentional design choices, primarily the inclusion of new kinds of training data.\n\n## Taxonomy of systemic risks, Sub-Measure 6.1: Types of systemic risks\n\nManipulation cannot be measured at the model level without a specific distribution model. CBRN risks are currently remote to inexistent, and requires grounded security research by domain experts, not highly abstracted evaluation by model developers. Automated use of models for research is part of their primary benign use and may at most be considered an accelerating factor for technology development, not a risk. Loss of control is primarily a risk tied to system design, not capabilities.\n\nPlease provide suggestions on how to improve this sub-measure\n\nCyber offence should be replaced by risks to cybersecurity, including through spread of code vulnerabilities. Categories mentioned above should be stricken. Categories of risks should cover a broader set of considerations from the AI Act, including consequences to public health, civic/human rights, and democratic processes stemming from model reliability fairness, as well as measurable risks to economic domains and communities (Rec. 110). See also: https://arxiv.org/abs/2306.05949.\n\nWhat are relevant considerations or criteria to take into account when defining whether a risk is a systemic risk?\n\nSystemic risks included in the taxonomy should be risks that:\n\n- -Have an identified and minimally likely negative impact on a given social system (e.g. education, information ecosystems, journalism, labor, etc.)\n- -Are tied to specific development choices or properties that can be characterized using reproducible, falsifiable, and context-sensitive evaluations\n- -For scientific principles for risk assessment, see e.g.: https://ec.europa.eu/newsroom/dae/redirection/document/110171\n\nBased on these considerations or criteria, which risks should be prioritised for addition to the main taxonomy of systemic risks?\n\nRisks to the security or reliability of critical systems, including access to health, education, government benefits or essential services. Risks to democratic processes through the integrity of journalism or information ecosystems. Risks to the environment through increased dependence on energy-intensive technology.\n\nHow should the taxonomy of systemic risks address AI-generated child sexual abuse material and non-consensual intimate imagery?\n\nAI-generated CSAM constitutes a systemic risk as defined insofar as it puts pressure on existing systems that are designed to protect targets and limit the spread of content. For specific harm mechanisms for CSAM harms, the importance of focusing on deployment and development choices, and the difficulty of anchoring work on 'model capabilities', see e.g. https://info.thorn.org/hubfs/thorn-safety-by-design-for-generative-AI.pdf\n\n## Taxonomy of systemic risks, Sub-Measure 6.2: Nature of systemic risks\n\nOrigin, actors, and intent all require significant further elaboration.\n\nProbability-severity ratio raises questions about who gets to define severity when different stakeholder groups are differentially affected and what constitutes a valid model to assess likelihood. It should also be understood that a risk is inherently defined by a notion of likelihood (a hazard by itself does not constitute a risk)\n\nPlease provide suggestions on how to improve this sub-measure\n\nOrigin should provide a definition of model capabilities. Model distribution should be inclusive of model development and deployment choices.\n\nThe actors driving the risks are currently missing the model developers, who are currently the ones in the strongest position to mitigate (or increase) risks through development choices. In particular, misrepresented or improperly documented models or capabilities are a strong risk factor.\n\n## Taxonomy of systemic risks, Sub-Measure 6.3: Sources of systemic risks\n\nThe sub-measure includes several notions that are not supported by evidence or that do not correspond to a quantifiable or falsifiable measure, including: self-replication, 'power-seeking', 'collusion', or a notion of alignment with universal 'human values'.\n\nPlease provide suggestions on how to improve this sub-measure\n\nSeveral inclusions in the current draft are very relevant and should be further expanded, including notably considerations of number of business users and technology readiness, lack of reliability and security including biases, and confabulation in deployment settings with high precision requirements.\n\nSection IV: [Working Groups 2/3/4] Rules for Providers of General-Purpose AI Models with Systemic Risk\n\nMeasures/Sub-measures Specific Feedback on Section IV: [Working Group 2] Risk assessment for providers of General-Purpose AI Models with Systemic Risk\n\nRisk assessment, Measure 8: Risk identification\n\nThe current version of the draft makes model developers responsible for identifying risks, selecting evaluations, and interpreting those evaluations. This raises two main concerns. While developers have the best understanding of some of the technical aspects of their models, they typically have neither the broader expertise (sociological, domain expertise, etc...) nor the legitimacy to decide which risks should be prioritized, especially when choices affect stakeholder groups differently.\n\nPlease provide suggestions on how to improve this sub-measure\n\nRisk identification should occur outside of the development chain. For developers training models with systemic risks, an external review system, possibly with the AI Office scientific advisory board, should determine which risks should be evaluated based on parts of the information described in Measures 1 and 2 that can be disclosed ahead of deployment. We note that openly released models in particular provide additional opportunities for external stakeholders and experts to identify risks.", + "char_slice": [ + 19719, + 26537 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2024_NIST%20RFI%20on%20EO.pdf", + "md_content": "\n\n## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'\n\nAuthors: Led by Margaret Mitchell (Chief Ethics Scientist), with writing from Yacine Jernite (ML & Society Team Lead), Sasha Luccioni (Climate Lead), Cl\u00e9mentine Fourrier (engineering), Josef Fukano (security), Ezinwanne (Ezi) Ozoani (engineering), Irene Solaiman (Global Policy Lead) Additional contributions from Imatag: Vivien Chappelier, Mathieu Desoubeaux\n\n## Submitted: Feb 2, 2024\n\n## Table of Contents\n\n| About Hugging Face....................................................................................................................3 | |\n|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----|\n| 1. Developing Guidelines, Standards, and Best Practices for AI Safety and Security................. | 3 |\n| (1) Developing a companion resource to the AI Risk Management Framework (AI RMF)........3 | |\n| Govern.................................................................................................................................3 | |\n| Data Governance......................................................................................................... | 3 |\n| Data Governance Roles............................................................................................... | 5 |\n| Platform Governance............................................................................................. | 5 |\n| Map..................................................................................................................................... | 5 |\n| Measure..............................................................................................................................7 | |\n| Data Measurement...................................................................................................... | 7 |\n| Model Evaluation......................................................................................................... | 8 |\n| Manage...............................................................................................................................8 | |\n| Rigorous Evaluation Report.........................................................................................9 | |\n| User Feedback........................................................................................................... | 10 |\n| Information on 'roles'............................................................................................................10 | |\n| ML Lifecycle Roles...........................................................................................................10 | |\n| Data Development Roles................................................................................................. | 10 |\n| Model Roles..................................................................................................................... | 11 |\n| Information on 'current techniques and implementations'................................................ | 12 |\n| Identifying impacts and developing mitigations............................................................12 | |\n| Assessments................................................................................................................... | 13 |\n| Content authentication, provenance tracking, and synthetic content labeling and detection...........................................................................................................................13 | |\n| Models and systems................................................................................................. | 13 |\n\n\n\n| Verifying the connection between data and models...............................................14 | |\n|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------|\n| Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.......................................................................... | 14 |\n| Information on 'forms of transparency and documentation'............................................. | 15 |\n| Model Documentation..................................................................................................... | 15 |\n| Dataset Documentation.................................................................................................. | 16 |\n| Assessment and Evaluation............................................................................................18 | |\n| Benchmarking............................................................................................................18 | |\n| Social Impact............................................................................................................. | 18 |\n| Information on 'watermarking'.............................................................................................19 | |\n| Information on 'disclosing errors'........................................................................................ | 20 |\n| (2) Creating guidance and benchmarks for evaluating and auditing AI capabilities, with a | (2) Creating guidance and benchmarks for evaluating and auditing AI capabilities, with a |\n| focus on capabilities and limitations through which AI could be used to cause harm...........21 | |\n| Information on 'auditing AI'..............................................................................................21 | |\n| Information on 'AI Evaluations'............................................................................................ | 21 |\n| Information on 'AI Red-Teaming'..........................................................................................22 | |\n| 2. Reducing the Risk of Synthetic Content..............................................................................24 | |\n| Information on 'synthetic content'....................................................................................... | 24 |\n| Information on 'non-consensual intimate imagery'............................................................ | 25 |\n| Technical Solutions......................................................................................................... | 25 |\n| Text-to-image systems..............................................................................................25 | |\n| Image-only systems.................................................................................................. | 25 |\n| Organizational Solutions................................................................................................. | 25 |\n| 3. Advancing Responsible Global Technical Standards for AI Development............................ | 27 |\n| Information on 'AI nomenclature and terminology'............................................................ | 27 |\n| NFAA..........................................................................................................................27 | |\n| Open Source, Open Science, and the Gradient of Openness................................... | 27 |\n| Information on 'collection and use of data'.........................................................................28 | |\n| Privacy.............................................................................................................................. | 29 |\n| Information on 'Human-computer interface design for AI systems'................................. | 29 |\n| Gates................................................................................................................................ | 29 |\n| Modals..............................................................................................................................30 | |\n| Information on 'AI-related standards development activities'........................................... | 30 |\n| Appendix................................................................................................................................32 | |\n| Information on Watermarking for the tracking of generated content from Imatag.................32 | |\n| New Key Terminology Introduced in this Document..................................................................34 | |\n\n\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. As part of our activity, Hugging Face provides social features and communication platforms for people interacting with AI systems, including social posts and discussion threads, in addition to hosting the AI systems themselves.\n\n- 1. Developing Guidelines, Standards, and Best Practices for AI Safety and Security\n- (1) Developing a companion resource to the AI Risk Management Framework (AI RMF)...\n\n## Information on 'practices for implementing AI RMF core functions'\n\nAddressing excerpt:\n\n- \u25cf Risks and harms of generative AI, including challenges in mapping, measuring, and managing trustworthiness characteristics as defined in the AI RMF, as well as harms related to repression, interference with democratic processes and institutions, gender-based violence, and human rights abuses (see\n- https://www.whitehouse.gov/briefing-room/speeches-remarks/2023/11/01/remarks-by-vice-president-ha rris-on-the-future-of-artificial-intelligence-london-united-kingdom );\n- \u25cf Current standards or industry norms or practices for implementing AI RMF core functions for generative AI (govern, map, measure, manage), or gaps in those standards, norms, or practices;\n- \u25cf Recommended changes for AI actors to make to their current governance practices to manage the risks of generative AI;\n\nWe are thrilled by NIST's categorization of govern, map, measure, and manage . It aligns with some of the work we have prioritized at Hugging Face. Below we describe 'practices for implementing AI RMF core functions', 'gaps in those\u2026practices', and 'recommended changes\u2026to manage the risks'.\n\n## Govern\n\n## Data Governance\n\nAI systems are a reflection of their data, and this data is often created by or about people. Data governance requires developers to account for the rights of these data subjects, including property, privacy, and user rights.\n\n\n\nGovernance practices that support respecting such rights depend on the specific development context and types of data used. For example, whether the AI system is developed as a commercial product or developed as a public good, or whether the data is obtained through web scraping or licensed through a commercial agreement with rights holders. These practices may involve gathering preference signals in whether specific items may be used for training (Spawning.ai), developing and applying new tools to remove privately identifying information (BigCode governance card), or providing tools to allow users of various online platforms to remove their data from training datasets (Am I In The Stack tool for GitHub opt-out).\n\nIn all cases, external stakeholders - including journalists and civil society organizations - have an important role to play in verifying that rights are indeed respected, provided they are given sufficient information about the workings or product of the data curation process. By outlining standards for what constitutes sufficient information to support external audits and data governance practices, NIST has an opportunity to foster responsible development of future AI systems.\n\nIn our work, we have outlined the requirements and proposed a data governance structure for web-scale text and media data. This work is primarily published in the Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency.\n\nThese ideas are based on a large workshop we co-led, focused on democratized LLM development, called 'BigScience' (https://bigscience.huggingface.co), which included a Data Governance working group focused on how to best govern data across multiple governments and rights-holders.\n\nWithin the Data Governance working group, we identified the need to have a collaborative committee that works through different issues - the 'Data Stewardship Organization' - with representatives for all stakeholders, including people who represent individuals' rights. See Figure 1 for a schematic of this structure.\n\nFigure 1. High-level schematic of Data Governance Structure Source: https://dl.acm.org/doi/10.1145/3531146.3534637\n\n\n\n\n\n## Data Governance Roles\n\nIn addition to the central shared Data Stewardship Organization , there are the following roles:\n\n- \u25cf Data Custodians , who can be Data Hosts, Data Providers, or both, using the data from Data Rights Holders.\n- \u25cb Data Hosts make the data available for analysis\n- \u25cb Data Providers are individuals or institutions who have text, image, or audio data that they can legally make available.\n- \u25cb Data Rights Holders are the creators or owners of the data\n- \u25cf Data Modelers are researchers and developers who use the data.\n\nAgreements and contracts are necessary between these different entities to best represent the different stakeholders in AI data usage.\n\n## Platform Governance\n\nWe ground governance of the Hugging Face platform on key values. These include openness , transparency, and collaboration . We also strive for kindness . Decisions regarding everything from design to reporting processes use these values as a foundation.\n\nIn order to govern such an open community platform and maintain key values, it is necessary to have a Code of Conduct (see: https://huggingface.co/code-of-conduct), which also describes different actions that the organization may take in different situations, and draws from what is appropriate with respect to Content Guidelines (see: https://huggingface.co/content-guidelines). A cornerstone of our Code of Conduct and Content Moderation policies is also consent , a key value for inclusion that also addresses many kinds of problematic content that may be shared.\n\n## Map\n\nFor training datasets of ML systems, organizations should perform the following mapping functions:\n\n- \u25cf Mapping stakeholder groups: organizations should work on identifying all relevant groups of data subjects (people who create or are represented in the dataset) as well as algorithm subjects (people whose lives will be affected by models leveraging the dataset through the training and deployment of the full AI systems).\n- \u25cf Mapping stakeholder rights: organizations should know what regulations might apply to the use of data and algorithmic decision systems. We draw particular attention to relevant sectoral regulation in health, education, and finance, which governs how information may be used and where data subjects are entitled to notification or explanation regarding their data use. Organizations should make sure to look to federal regulations and state regulations on those topics, including relevant rules on privacy or customer protection.\n\n\n\n- \u25cf Mapping flows of information: finally, organizations should provide a mapping of the flow of information that supports their activity, for example in the form of a data management plan. In current Machine Learning development practice, data gathered for an initial purpose is often re-used outside of its initial context, which may have consequences for the identification of stakeholder groups and rights outlined above.\n\nWe provide an example of mapping between stakeholders and rights in Table 1 below. This was created by centering on the pillars of development defined in Manage (Figure 2) and articulating the groups involved in each.\n\n| Example Stakeholder Groups | Relevant Pillars (from Manage & Figure 2) | Example Rights Affected (adapted from the UDHR) |\n|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|\n| Data creators and data subjects , including those producing \"raw\" data (such as artists), those annotating it (such as crowdworkers), and the people represented in the data | Data Collection Training Processes | - Right to Gain a Living - Right to [Intellectual] Property |\n| AI developers , which may be individual engineers or larger organizations (such as tech companies) | Data Collection Training Processes Model Evaluation & Analysis System Deployment | - Right to Gain a Living - Right to [Intellectual] Property |\n| AI deployers , who leverage the technology for different applications (such as companies and government agencies) | Model Evaluation & Analysis System Deployment | - Right to Gain a Living - Right to [Intellectual] Property |\n| AI users , who interact with the technology made available by deployers (such as people in education, healthcare, and finance) | System Deployment | - Freedom from Harm - Freedom of Expression - Right to Privacy - Right to Non-Discrimination |\n| AI-affected , who may have AI technology applied to them, whether or not they chose to (such as in surveillance) | System Deployment | - Freedom from Harm - Right to Privacy - Right to Non-Discrimination - Rights of the Child |\n\nTable 1. Example stakeholder groups corresponding to each pillar include the following, which is non-exhaustive and not mutually exclusive. Also included are a set of example rights for each; see the linked UN article for further detail.\n\nFor each pillar, we can derive the benefits, harms, and risks to different people in different AI contexts, identifying the potential for positive and negative impact, and which rights may be affected.\n\n## Measure\n\n## Data Measurement\n\nThis concept is defined in Mitchell et al. (2022) Measuring Data.\n\nThis is the key idea: Just as we evaluate models, we should measure data.\n\nPrioritizing/working on data measurement with the same rigor and frequency as model evaluation would be a significant change that we recommend AI actors must make in order to better identify foreseeable outcomes - risks and benefits - from systems trained on that data.\n\nFor example, measuring data allows us to identify, and quantify the risk of, things that a model might learn and emulate. This includes measuring things such as:\n\n- \u25cf Stereotypes\n- \u25cf Sentiment, opinions and stances\n- \u25cf Persuasion tactics\n- \u25cf Incorrect information, which could be realized by a model as misinformation, or abused to create disinformation\n- \u25cf Hate and toxicity\n- \u25cf Private and personally identifiable information, using raw counts of different types\n\n## These are measurable using:\n\n- \u25cf Statistical techniques that do not require additional annotations, such as co-occurrence statistics (e.g., correlation and association metrics) or frequency statistics (e.g., tf-idf, fit to Zipf's law)\n- \u25cb We have implemented a proof-of-concept at https://huggingface.co/blog/data-measurements-tool.\n- \u25cf Real-valued scores from classifiers, such as sentiment or toxicity classifiers that provide both the polarity of the content and its predicted magnitude;\n- \u25cf Human annotations\n- \u25cb Such as described in ISO work on 'data quality measures' (e.g., ISO 25024).\n\nData measurement can also be applied to the output of models - think of model output as its own dataset - as a way to measure different model properties. For example, one can measure learned stereotypes from a generative model without additional annotations (see https://dl.acm.org/doi/10.1145/3461702.3462557) by generating a large amount of content from the model and measuring the co-occurrence between different items in its generations, like woman and smile . We can combine this with human input on social categories and derive further insights as well (see https://aclanthology.org/2023.acl-long.84.pdf).\n\nFurther, the company Lilac ML (https://www.lilacml.com) has prioritized exactly this kind of work, and may be a great partner for NIST to help provide tools for responsible data analysis.\n\n\n\n\n\n## Model Evaluation\n\nThe rapid proliferation of models, architectures and modalities makes it important to have reliable ways for comparing and evaluating models. Depending on the context of development, different metrics and benchmarks should be used and reported in accompanying reports and literature. Given the open-endedness of generative AI models, there is often no single correct answer, making evaluation even more difficult and resulting in a plethora of different ways to evaluate new and existing models. Hugging Face's evaluate Python package and Eleuther AI's language model evaluation harness are examples of tools that aim to make model evaluation more standardized and systematic.\n\nAlso see our section on Benchmarking Results.\n\n## Manage\n\nAn approach that helps to manage the risks of AI (including generative AI) within a governance framework requires modularizing the end-to-end development of AI systems and engaging in critical analyses for each. A core part of AI systems are machine learning models, which can be modularized into 4 components (see Figure 2). Each component requires robust analysis articulating risks, rights, and proactive solutions.\n\n\n\nWe discuss the regulatory artifacts of (1) Dataset Cards and (2) Model Cards in Information on 'forms of transparency and documentation' below. Here, we detail (3) Rigorous Evaluation Report and (4) User Feedback.\n\n\n\n## Rigorous Evaluation Report\n\nThe Rigorous Evaluation Report requires disaggregated evaluation . Evaluation is said to be 'disaggregated' when it is not one single, 'aggregate' score, but a set of scores for different slices, such as scores for different subpopulations in different contexts. A model is said to be 'fair' with respect to a characteristic (e.g., 'fair across genders') when the evaluation metrics are equal across the different subpopulations for that characteristic (e.g., when the model's evaluation score for men, the model's evaluation score for women, the model's evaluation score for nonbinary, etc., are all equal).\n\nOur experience suggests that Rigorous Evaluation Reports are created well by first following these steps:\n\n- Step 1. Define the relevant people (stakeholder groups and subpopulations).\n- Step 2. Identify how each group may be affected in different contexts.\n- Step 3. Determine the metrics and measurements to evaluate and track the effects on each group (Step 1) in each context (Step 2).\n- Step 4. Evaluate model performance with respect to the groups and contexts (Step 3).\n\nFor Steps 1 and 2, the approach requires crossing people by contexts. \"People\" are split into users and those affected intended , and unintended . \"Contexts\" are similarly split into intended and unintended , as well as out of scope . This can be seen as primarily a 2x4 grid, where each cell must be filled out - see Figure 3.\n\n\n\nFigure 3. Foresight in AI chart. When developing responsibly, it should be possible to fill out each of these cells.\n\nThis means that rigorous evaluation requires first answering questions the cells correspond to\n\n\n\nsuch as what are the use contexts, and who is involved in these contexts? What are the intended or beneficial uses of the technology in these contexts? What are the unintended or negative ones?\n\nClearly defined subpopulations and use contexts can inform the selection of metrics to evaluate the system in light of foreseeable outcomes.\n\n## User Feedback\n\nThose affected should be able to easily provide feedback: In order to know how well the system is working, and to have serious errors immediately flagged, there must be a simple user-facing mechanism for immediate feedback. At Hugging Face, we have implemented this in several ways:\n\n- 1. A 'Report this' button that appears alongside data, models, and demos - and the report can either be public or anonymous\n- 2. A 'Community tab' for open discussion alongside different artifacts.\n\n## Information on 'roles'\n\n## Addressing excerpt:\n\n- \u25cf The types of professions, skills, and disciplinary expertise organizations need to effectively govern generative AI, and what roles individuals bringing such knowledge could serve;\n- \u25cf Roles that can or should be played by different AI actors for managing risks and harms of generative AI ( e.g., the role of AI developers vs. deployers vs. end users);\n\n## ML Lifecycle Roles\n\nPlease see our section on the AI RMF Map pillar, Table 1.\n\n## Data Development Roles\n\nDetails on the roles, professions, skills, etc., needed for data governance were described in the above sections addressing the AI RMF Govern pillar (esp. Data Governance and Data Governance roles).\n\nThere are also different types of responsibilities needed for dataset development (including creation and maintenance). These roles should serve to produce the artifacts listed in Figure 4, which are needed for responsible data development.\n\n\n\n\n\n| Artifact | Details |\n|------------------------------|-----------------------------------------------------------------------------|\n| Dataset Requirements Spec | Target Properties Intended uses |\n| Dataset Design Doc | How dataset will be collected |\n| Dataset Implementation Diary | Status of collection attempts How issues were solved |\n| Dataset Testing Report | Measurement of dataset Results of adversarial examination Issues in dataset |\n| Dataset Maintenance Plan | Updates for opt-out Handling stale data |\n\nPlease also see the suggestions in Khan and Hanna. 2022. 'The Subjects and Stages of AI Dataset Development: A Framework for Dataset Accountability.'\n\n## Model Roles\n\nRoles in responsible governance of AI models should include the appropriate diversity needed to identify relevant information about a machine learning model for different kinds of audiences.\n\nThis includes the following (adapted from our Annotated Model Card, https://huggingface.co/docs/hub/model-card-annotated ). One person may have more than one role:\n\n\n\n- \u25cf The developer , who writes the code and runs training;\n- \u25cf The sociotechnic , who is skilled at analyzing the interaction of technology and society long-term (this includes lawyers, ethicists, sociologists, or rights advocates);\n- \u25cf The project organizer , who understands the overall scope and reach of the model, and who serves as a contact person to connect different parties.\n\nThe developer is necessary for providing information about a model's training procedure and technical specifications. They are also useful for identifying technical 'limitations', such as likely failure modes, and additionally may be able to provide recommendations for addressing those limitations. They are responsible for calculating and providing results of model evaluation, working with the other roles to define the most appropriate rigorous evaluation.\n\nThe sociotechnic is necessary for identifying different contexts where a model may be used, the different subpopulations that may be differently affected by the model, and how the model may be used in different contexts or with respect to different subpopulations. The types of bias a model might have, the risks different applications bring, and out-of-scope model usage naturally fall out of this kind of analysis.\n\nThe project organizer is necessary for identifying content such as basic uses of the model, licensing requirements, terminological alignment, etc.\n\n## Information on 'current techniques and implementations'\n\n## Addressing excerpt:\n\n- \u25cf Current techniques and implementations, including their feasibility, validity, fitness for purpose, and scalability, for:\n- \u25cb Model validation and verification, including AI red-teaming;\n- \u25cb Human rights impact assessments, ethical assessments, and other tools for identifying impacts of generative AI systems and mitigations for negative impacts;\n- \u25cb Content authentication, provenance tracking, and synthetic content labeling and detection, as described in Section 2a below; and\n- \u25cb Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.\n\n## Identifying impacts and developing mitigations\n\nChannels for communication should be maintained for responsible disclosure of security issues, vulnerabilities, bias, and inappropriate content. For fully public-facing models, the mechanisms to provide feedback must be open for all of the public (even if what is reported is not made visible due to security concerns). For more private or internal systems, channels must be open for private/internal employee communication.\n\nAt Hugging Face, we have implemented such mechanisms, described in our section on operationalising User Feedback as part of NIST's Manage pillar.\n\n\n\nAssociated with these channels should be a procedure to document, triage (escalate), risk rank, and resolve items reported to the organization using AI. (Note that mature organizations will generally already have a similar pattern they can follow from bug bounty, security operations, privacy data subject requests, and related programs.) 'Escalations' or triaging occurs when there is an immediately pressing issue that must be addressed. In these cases, people throughout the company are brought in to quickly put their heads together to define the immediate short-term medium-term , , , and long-term solutions, along with who is the 'point person' for each.\n\nAdditionally, companies that rely on or invest in these models for business activities should consider bounty initiatives for the responsible disclosure of model issues.\n\nOngoing automated and periodic manual testing of models is also required to ensure they operate within expected parameters. When using AI models, subject matter experts should be employed or consulted to use and review results on an ongoing basis. For example, if a company builds a fraud detection system, they should still have a fraud specialist that can interpret results and identify anomalies. This is an 'external to the system' control that provides the opportunity to identify problems with the output and then investigate where the problems occurred upstream. These subject matter experts will complement automated scans of data and outputs to ensure they fall within predefined thresholds.\n\n## Assessments\n\nPlease see our section on Auditing.\n\n## Content authentication, provenance tracking, and synthetic content labeling and detection\n\nWe believe that content provenance for AI datasets, models, and systems, is critical for the future of responsible and ethical AI.\n\n## Models and systems\n\nFor models and systems, we are entering a world where it's becoming unclear which content is created by AI systems, and impossible to know where different AI-generated content came from. Bad-faith actors can further compound the issue, using this new technology to deceive, misrepresent, and influence people without a trace. These issues are directly addressed by embedding content provenance information in generated content, using techniques such as watermarking , which helps us to know what has been created by an AI system and where it came from. It provides for mechanisms that help the public gain control over the role of generated content in our society.\n\nWe have collaborated with multiple parties to demonstrate the state of the art in content provenance mechanisms. This includes:\n\n\n\n- \u25cf Truepic (https://hf.co/blog/alicia-truepic/identify-ai-generated-content), a leading provider of authenticity infrastructure for the internet who has demonstrated how to:\n- a. Cryptographically secure metadata into any generated image using the open C2PA standard: https://huggingface.co/spaces/Truepic/ai-content-credentials\n- b. Generate 'invisible QR codes' as image watermarks that can be used to retrieve further image metadata:\n\nhttps://huggingface.co/spaces/Truepic/watermarked-content-credentials\n\n- \u25cf Imatag (https://hf.co/blog/imatag-vch/stable-signature-bzh), who specializes in digital watermarking and has demonstrated how to embed secure and robust invisible watermarks during the image generation process:\n\nhttps://huggingface.co/spaces/imatag/stable-signature-bzh\n\n## Verifying the connection between data and models\n\nHow to verify what data a model has been trained on is currently open research (see https://openreview.net/forum?id=TwLHB8sKme). While some developers may report the data they used, there are limited ways to prove whether this is true, or exhaustive.\n\n## Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.\n\nSome approaches and resources that can serve as starting points for delving deeper into this include:\n\n- \u25cf Bounty Catching : The cybersecurity field has illustrated success in Bug Bounty Programs. Some notable examples include:\n- \u25cb HackerOne: HackerOne is one of the leading platforms for running bug bounty programs. They release detailed blog posts, where they provide insights into measuring the performance of their internal and business collaborative bug bounty initiatives using key indicators like the total number of reports received, average time to resolution, and severity distribution of reported bugs. By focusing on these factors, organizations can gauge the efficiency of their escalation testing processes and continuously improve them.\n- \u25a0 https://www.hackerone.com/vulnerability-and-security-testing-blog\n- \u25cb Synack: Synack is a crowdsourced security platform that shares several quantifiable measures to determine the efficacy of a bug bounty program. Some of these metrics include the number of unique vulnerabilities discovered, time-to-fix, and cost savings compared to traditional penetration testing methods. Regularly monitoring these parameters enables continuous improvement and fine-tuning of escalation testing models\n- \u25a0 https://www.synack.com/wp-content/uploads/2022/09/Crowdsourced-S ecurity-Landscape-Government.pdf\n\n\n\n- \u25cf Subject Matter Experts: Organizations and governments in the U.S. are continuing to advance AI initiatives by creating initiatives on collaboration with subject matter experts. By learning from prior efforts, these initiatives succeed in delivering accurate predictions, innovative tools, and ethical standards. This includes:\n- \u25cb The American Heart Association : Recently announced the launch of its precision medicine platform, which utilizes AI and machine learning to analyze genomic, biological, and lifestyle data. Medical experts contributed to the design and validation of the platform, guaranteeing medical relevancy and ethical standards.\n- \u25a0 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8452247/\n- \u25a0 https://www.ahajournals.org/doi/full/10.1161/CIRCOUTCOMES.121.0079 49\n- \u25cb NASA's Frontier Development Lab : Hosted summer fellowships for space scientists and AI researchers to collaboratively tackle challenging scientific questions. Participants exchange knowledge, creating AI tools and applications that push the frontiers of both fields.\n- \u25a0 https://frontierdevelopmentlab.org/\n\nFurthermore, combining insights from separate disciplines can often lead to innovative solutions for novel problems. Cross-referencing these examples may inspire new ideas and approaches for evaluating AI techniques and implementations.\n\n## Information on 'forms of transparency and documentation'\n\nAddressing excerpt:\n\n- \u25cf Forms of transparency and documentation ( e.g., model cards, data cards, system cards, benchmarking results, impact assessments, or other kinds of transparency reports) that are more or less helpful for various risk management purposes ( e.g., assessment, evaluation, monitoring, and provision of redress and contestation mechanisms) and for various AI actors (developers, deployers, end users, etc.) in the context of generative AI models, and best practices to ensure such information is shared as needed along the generative AI lifecycle and supply chain);\n\nAt Hugging Face, we have worked extensively on operationalizing 'model cards' and 'dataset cards' for all models and datasets that are shared on the platform.\n\n## Model Documentation\n\nWe have spent years working with organizations and people throughout the world to fill out model cards. This effort has been led in part by the lead author on the Model Card paper, Margaret Mitchell. Her writing on what people are missing when creating Model Cards is available here:\n\nhttps://www.techpolicy.press/the-pillars-of-a-rightsbased-approach-to-ai-development/, in 'Deep Dive: Model Cards' section. Critically, people are generally missing disaggregated evaluation in their reporting of results, which requires first identifying the relevant subpopulations to evaluate\n\n\n\nmodel performance on; then identifying the types of errors likely to cause harm; then defining evaluation metrics based on these errors; and finally applying the evaluation with the selected metrics across the disaggregated subpopulations.\n\nHowever, since skipping disaggregated evaluation is becoming increasingly common, it may make sense for Model Cards to exclude this information and have it instead reported in a Rigorous Evaluation Report, as described in our section on the AI RMF Manage pillar..\n\nFor further detail, here is our guidebook on Model Cards: https://huggingface.co/docs/hub/main/model-card-guidebook.\n\nThis covers:\n\n- \u25cf How to fill the model card out, including roles and responsibilities (https://huggingface.co/docs/hub/main/model-card-annotated)\n- \u25cf Model card user studies (https://huggingface.co/docs/hub/main/model-cards-user-studies)\n- \u25cf A landscape analysis of ML documentation tools (https://huggingface.co/docs/hub/main/model-card-landscape-analysis)\n\n## Dataset Documentation\n\nWe have worked on developing new standards and tools to support dataset transparency, and have written extensively on the current state of possibilities and practices in the field (https://huggingface.co/blog/yjernite/data-transparency) Our work draws from approaches such as 'Datasheets for Datasets' (https://arxiv.org/abs/1803.09010).\n\nUnderstanding the make-up and characteristics of the datasets used in AI systems is essential to fully controlling the development of AI : AI model behavior is based on the data it is trained on. Dataset documentation should provide information surrounding data provenance as well as high-level statistics about datasets, such as the languages they contain and the most common words or categories. Providing this information in an easily accessible (and machine-readable) format can help people to understand which datasets are useful for which tasks. This can also help improve the representativity of datasets, for instance to highlight languages that are under-represented. Detailed documentation may also take the form of metadata , with information about each instance in the dataset. This can (among other things) support opting out of datasets for data creators or rights-holders who do not want their data to be distributed or used for training AI models.\n\nDataset documentation sits within a set of data mechanisms that ensure that the risks of the technology are properly managed - including risks of discrimination, risks to privacy, and broadly making sure that the technology follows all applicable regulations. This set of mechanisms include:\n\n- \u25cf Direct access to the datasets by some categories of stakeholders\n\n\n\n- \u25cb See our sections on the AI RMF Map pillar, Data Development Roles, Data Governance Roles; and our sections on Auditing for information on stakeholders external to the data development process.\n- \u25cf Sufficient public-facing documentation according to broadly applicable standards, as described here\n- \u25cf Interactive exploration tools to support investigation from actors outside of the development chain\n- \u25cb One recent example is Lilac ML 's 'Garden' (https://docs.lilacml.com/blog/introducing-garden.html).\n- \u25cb Further tools are linked in our blog post on data transparency ( https://huggingface.co/blog/yjernite/data-transparency ).\n\n## For example, we note that:\n\n- \u25cf The propensity of a Large Language Model to produce hateful or toxic text is directly correlated with the presence of this kind of content in the training dataset. At the same time, efforts to automatically remove this kind of text have been shown to introduce their own biases, and have a disparate impact on marginalized groups (https://huggingface.co/papers/2104.08758). While direct access to a training dataset is not always possible, listing the original sources of the dataset and providing documentation of the specific mitigation techniques leveraged to reduce the proportion of hateful or toxic text in the final dataset, including the specific criteria and thresholds used in the filtering, can help ensure that risks tied to hate speech and risks of discrimination are properly balanced (further described in the recent preprint at https://arxiv.org/abs/2401.06408).\n- \u25cf Similar considerations also apply to other content modalities, including for image generation systems, where efforts to limit risks of generating violent or other kinds of undesirable content (e.g., https://openai.com/research/dall-e-2-pre-training-mitigations) have also led to decreased social diversity in the model outputs (as we demonstrate in our NeurIPS paper on 'Stable Bias', https://arxiv.org/abs/2303.11408).\n- \u25cf Additionally, over-estimating a model's performance on specific tasks is an important risk factor of AI deployment - especially when the model is in a position to significantly shape people's access to services or be integrated into infrastructure. One of the best studied sources of over-estimation is benchmark contamination (described in https://arxiv.org/abs/2312.16337), where a model may be evaluated on a benchmark that it was fully or partially trained on.\n- \u25cb While decontamination approaches can help developers, downstream adopters of the technology may not have sufficient information about the training data to safely perform their own evaluation and are at risk of spending significant effort on testing a model's safety in a controlled setting only to have it fail in real-world deployment because they could not meaningful separate their safety benchmarks from the model's training dataset.\n\n\n\n## As a result, we recommend:\n\n- \u25cf For small to medium datasets, for example datasets from a single or from a few identified sources of up to tens to hundreds of data items (sentences, images, documents, etc.), organizations should at the very least provide documentation of the dataset in the form of a data statement, datasheet, or data nutrition label.\n- \u25cf For larger datasets, including composite datasets and web-crawled datasets, organizations should further provide documentation of the major sources used for training models, and individual documents covering all major sources following one of the standards mentioned above.\n- \u25cf Further, any automatic risk mitigation strategies at the dataset level should be sufficiently documented to allow external stakeholders to evaluate the trade-offs and values it prioritizes, including choices motivating data filtering.\n\n## Assessment and Evaluation\n\n## Benchmarking\n\nWe have found leaderboards to be extremely effective and influential in benchmarking and reproducing results. In particular our Open LLM Leaderboard (https://huggingface.co/spaces/HuggingFaceH4/open\\_llm\\_leaderboard) garners 250K visits monthly, with more than 4K models submitted and evaluated, and shows how well different open LLMs perform across a variety of well-known benchmarks. Our teams also produced an LLM performance leaderboard (https://huggingface.co/spaces/optimum/llm-perf-leaderboard) to evaluate energy efficiency and consumption of LLMs at inference time, and the Big Code Leaderboard (https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard), to evaluate model's programming quality.\n\nWe have provided support for a number of more specialized leaderboards, in collaboration with universities or companies:\n\n- \u25cf The Chatbot Arena Leaderboard, which uses human ratings to compute an Elo score of available (open and closed) models\n- (https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard)\n- \u25cf The LLM Safety Leaderboard, looking at toxicity, bias and robustness, among others (https://huggingface.co/spaces/AI-Secure/llm-trustworthy-leaderboard)\n- \u25cf Two Hallucinations Leaderboards, to evaluate the propensity of models to say incorrect things (https://huggingface.co/spaces/hallucinations-leaderboard/leaderboard and https://huggingface.co/spaces/vectara/Hallucination-evaluation-leaderboard)\n- \u25cf Several specialized leaderboards for specific languages, like Korean, Dutch, etc \u2026\n\n\n\n## Social Impact\n\nTo determine the impact of AI systems on people, especially in safety and capabilities that can cause harm, evaluations targeted at social impact can give insight on certain model behaviors. Social impacts of systems can be evaluated by two means: technical evaluations of the model and by surveying users and affected populations.\n\nTechnical evaluations for the social impact of models include quantifying harmful biases, toxicity, disparate performance of language, and environmental cost of training and inference. However, some social impacts, such as overreliance on outputs and trust in information, can only be measured by surveying and/or interviewing affected people. Technical evaluations of social harms (such as biases) can also be overly narrow, with reductive identity group categorization such as solely 'profession'.\n\nThat said, social impact assessment is an area of open research. Methods and tools needed to evaluate complex social harms have many limitations, from human evaluator bias, to classifier biases, to being outdated for social norms. The timelines needed for examining social aspects of safety can also vary based on using time as a variable to analyze, e.g. in trust in information over time or impact on human labor and the global economy.\n\n## Information on 'watermarking'\n\nAddressing excerpt:\n\n- \u25cf Economic and security implications of watermarking, provenance tracking, and other content authentication tools;\n- \u25cf Efficacy, validity, and long-term stability of watermarking techniques and content authentication tools for provenance of materials, including in derivative work;\n\nAlso see our response on Content authentication, provenance tracking, and synthetic content labeling and detection.\n\nWriting in this section provided in part from Imatag ( https://www.imatag.com ), who specializes in digital watermarking. Their solutions are some of the only independent watermarking technologies for AI models. Their full response on this section is attached in the Appendix.\n\nWatermarking is a technique designed to unobtrusively mark content in order to convey additional information, such as authenticity. Within AI, watermarking involves adding machine recognizable patterns to digital content (such as images), and these patterns convey information such as where the content came from and whether it's synthetic. New AI/digital watermarking methods are also designed to be imperceptible to people, so as not to disrupt the content but still make it possible to detect and track digital content as soon as it's shared. New digital watermarking methods are increasingly robust to common alterations (e.g., compression, cropping, and color changes in the case of images), and AI watermarking solutions are said to be ' secure' when malicious attempts to remove, extract, or forge watermarks are not possible\n\n\n\nfrom a third-party unaware of the secret (i.e., the 'key' or 'cipher') used to protect the watermarking solution.\n\nTwo distinct methods are currently being studied for watermarking AI-generated images. The first method involves watermarking the output of the generative AI model after it's created , just as can be done with any content we might upload online. In this context, there is evidence that it's possible to create digital watermarking that is fast, robust and secure, for closed systems: The company Imatag is delivering such a system for newswire companies (https://www.prnewswire.com/news-releases/afp-selects-imatag-to-track-the-uses-of-its-photog raphs-301550539.html ). However, applying watermarks as a post-process in an open system has the strong drawback that it is easily removed by simply commenting out some of the code.\n\nThe second method implements the watermarking process during the creation of AI content. This is possible to securely apply to open AI models , such as those made available on the Hugging Face platform. This enables the distribution of AI models that are already modified to automatically embed watermarks as part of their generation process. This lowers the burden on individual developers to add on their own watermarking solutions, and provides secure watermarking by 'default'.\n\nThe trade-offs between imperceptibility, robustness, and security is key to evaluating AI/digital watermarking systems. As digital content spreads on the Internet, it is often modified multiple times for technical or editorial reasons. For example, it may be saved from a screenshot, saved as different file types, recompressed, or cut off. This is why current open-source watermarking solutions are not robust enough to these kinds of alterations for practical use (see https://medium.com/@steinsfu/stable-diffusion-the-invisible-watermark-in-generated-images-2 d68e2ab1241, https://www.wired.com/story/artificial-intelligence-watermarking-issues/). However, it is important to recognize that although watermarking is not perfect, we should not let perfect be the enemy of the good: The use of watermarking will help good actors and mitigate many bad actors. (Article where we further discuss this available here: https://venturebeat.com/ai/invisible-ai-watermarks-wont-stop-bad-actors-but-they-are-a-really-bi g-deal-for-good-ones/)\n\n## Information on 'disclosing errors'\n\n## Addressing excerpt:\n\n- \u25cf Criteria for defining an error, incident, or negative impact;\n- \u25cf Governance policies and technical requirements for tracing and disclosing errors, incidents, or negative impacts;\n\nWithin code development, errors can be easily tracked and traced using git protocols and tools (see https://git-scm.com) or similar versioning software that can:\n\n\n\n- \u25cf Track who did what, when\n- \u25cf Flag issues before merging/incorporating new code into a shared code repository.\n\nWithin a broader community, it's also possible to create mechanisms for community feedback. At Hugging Face we have adopted an 'open' approach such that this feedback is viewable and accessible to everyone side-by-side with models and data (called the 'Community Tab'; see https://huggingface.co/blog/community-update ), so those developing and using these assets can also interact with others. Because community feedback is open for everyone for each model, dataset, or demo, people affected - not just direct users - can provide information about the effect of the systems. This mechanism is also mentioned in our response above on User Feedback as part of NIST's Manage pillar.\n\nIn order to govern such an open community feedback system, see Platform Governance above.\n\n- (2) Creating guidance and benchmarks for evaluating and auditing AI capabilities, with a focus on capabilities and limitations through which AI could be used to cause harm.\n\n## Information on 'auditing AI'\n\nCompanies that use AI for business purposes should consider performing internal and external audits that correspond to the risks that AI poses to the business as well as consumers, data subjects, and industry. Such risks may be informed by the sociotechnic role described in our above response on roles. Internal audits should use a combination of manual tests and automated / algorithm based analysis. Audits can also be 'second-party' - open to a single person or an organization under NDA - or 'third party' , open to people without connections to the company. See 'Missing links in AI governance (UNESCO)' for more information on these kinds of audits.\n\nThese audits should examine the regulatory artifacts described in our section on the AI RMF Manage pillar, as well as the processes to produce them, in order to verify the work appropriately addresses different values, processes, or laws. The regulatory artifacts are aligned with standard tech development practices and so provide for a clear 'connection point' between tech companies and tech auditors.\n\n## Information on 'AI Evaluations'\n\n## Addressing excerpt:\n\n- \u25cf Definition, types, and design of test environments, scenarios, and tools for evaluating the capabilities, limitations, and safety of AI technologies;\n- \u25cf Availability of, gap analysis of, and proposals for metrics, benchmarks, protocols, and methods for measuring AI systems' functionality, capabilities, limitations, safety, security, privacy, effectiveness, suitability, equity, and trustworthiness.\n\n\u2026\n\n\n\n- \u25cf Generalizability of standards and methods of evaluating AI over time, across sectors, and across use cases;\n- \u25cf Applicability of testing paradigms for AI system functionality, effectiveness, safety, and trustworthiness including security, and transparency, including paradigms for comparing AI systems against each other, baseline system performance, and existing practice\n\nPlease see our section on Assessment and Evaluation .\n\n## Information on 'AI Red-Teaming'\n\n## Addressing excerpt:\n\n- \u25cf Use cases where AI red-teaming would be most beneficial for AI risk assessment and management;\n- \u25cf Capabilities, limitations, risks, and harms that AI red-teaming can help identify considering possible dependencies such as degree of access to AI systems and relevant data;\n- \u25cf Current red-teaming best practices for AI safety, including identifying threat models and associated limitations or harmful or dangerous capabilities;\n- \u25cf Internal and external review across the different stages of AI life cycle that are needed for effective AI red-teaming;\n- \u25cf Limitations of red-teaming and additional practices that can fill identified gaps;\n- \u25cf Sequence of actions for AI red-teaming exercises and accompanying necessary documentation practices;\n- \u25cf Information sharing best practices for generative AI, including for how to share with external parties for the purpose of AI red-teaming while protecting intellectual property, privacy, and security of an AI system;\n- \u25cf How AI red-teaming can complement other risk identification and evaluation techniques for AI models;\n- \u25cf How to design AI red-teaming exercises for different types of model risks, including specific security risks ( e.g., CBRN risks, etc.) and risks to individuals and society ( e.g., discriminatory output, hallucinations, etc.);\n- \u25cf Guidance on the optimal composition of AI red teams including different backgrounds and varying levels of skill and expertise;\n- \u25cf Economic feasibility of conducting AI red-teaming exercises for small and large organizations; and\n- \u25cf The appropriate unit of analysis for red teaming (models, systems, deployments, etc.)\n\n'Red-teaming' applies after a model has already been trained, and as such is a post-hoc approach to AI safety. It operates similarly to 'whackamole': Given an already problematic system, try changing one component and seeing what other issues may pop up. It is a relatively accessible means of testing a system for subject matter experts who can query a system through low barrier interfaces. More technical details on red-teaming at Hugging Face is available at https://huggingface.co/blog/red-teaming.\n\nIt is also one of the weaker approaches for guaranteeing safety, as it does not holistically assess an AI system's safety and cannot influence model behavior in the way that other methods do, such as careful data curation. As one of many tools in the Responsible AI toolbox, it is one of the last that can be applied to identify model harms within the chain of model development (Figure 2), preceding user feedback and external audits.\n\n\n\nSummarizing from above, the 'red-team' approach thus applies at the tail end of the following roughly consecutive interventions:\n\n- 1. Public Consultation: Before beginning to train a model, best practices include consulting with experts who are mostly likely to use or be affected by the model in order to understand what to prioritize in data collection and model training. This also serves as input for impact assessments.\n- 2. Data requirements: Specify what is desired from the model and collect data in light of these goals, as described in Information on Roles, Data section.\n- 3. Data analysis and measurement: Identifying issues of representation, stereotype, etc., as described in Information on NIST's Measure pillar, Data Measurement section.\n- 4. Model training, mapping inputs to outputs: Analyzing the effect of different training data slices on different model behaviors, and excluding those data instances that result in problematic behavior.\n- 5. Model training, disaggregated evaluation: As the model trains, different model checkpoints can be evaluated with respect to different areas of concern.\n- 6. Model convergence, disaggregated evaluation: When the model is done training, disaggregated evaluation can similarly be used to identify harms with respect to different contexts and subpopulations.\n- 7. Red-Teaming: We recommend https://datasociety.net/library/ai-red-teaming-is-not-a-one-stop-solution-to-ai-harms-reco mmendations-for-using-red-teaming-for-ai-accountability/.\n- 8. User feedback: Including bug bounties, as described in our section on Model Validation & Verification, including red-teaming.\n\n\n\n## 2. Reducing the Risk of Synthetic Content\n\n## Information on 'synthetic content'\n\n## Addressing excerpt:\n\nExisting tools and the potential development of future tools, measurement methods, best practices, active standards work, exploratory approaches, challenges and gaps are of interest for the following non-exhaustive list of possible topics and use cases of particular interest.\n\n- \u25cf Authenticating content and tracking its provenance;\n- \u25cf Techniques for labeling synthetic content, such as using watermarking;\n- \u25cf Detecting synthetic content;\n- \u25cf Resilience of techniques for labeling synthetic content to content manipulation;\n\n## Please see our responses on Watermarking and Content Authentication.\n\n## Addressing excerpt:\n\n- \u25cf Approaches that are applicable across different parts of the AI development and deployment lifecycle (including training data curation and filtering, training processes, fine-tuning incorporating both automated means and human feedback, and model release), at different levels of the AI system (including the model, API, and application level), and in different modes of model deployment (online services, within applications, open-source models, etc.);\n- \u25cf Testing software used for the above purposes; and\n- \u25cf Auditing and maintaining tools for analyzing synthetic content labeling and authentication.\n\nSynthetic content is problematic in part because it can be used for disinformation and non-consensual sexualization. For both, the platform where the content would be distributed has a critical role to play (a point also addressed in our section on Organizational Solutions for non-consensual intimate imagery). Platforms should scan shared content to verify whether or not it is synthetic, and alert users accordingly. One example of how to do this would be:\n\n- \u25cf Organizations that make generative AI models available embed metadata and/or invisible watermarks within the generated content.\n- \u25cf For watermarks, this might be using a proprietary method that would have corresponding proprietary detection software.\n- \u25cb Major platforms where synthetic content is shared can then run the different available proprietary detection software tools on shared content to identify whether any of them have additional metadata that can be used to change how the content is shared on the platform.\n\n\n\n## Information on 'non-consensual intimate imagery'\n\n## Addressing excerpt:\n\n- \u25cf Preventing generative AI from producing child sexual abuse material or producing non-consensual intimate imagery of real individuals (to include intimate digital depictions of the body or body parts of an identifiable individual);\n\nThere are multiple points of intervention for combating CSAM and non-consensual intimate imagery. These can be roughly categorized as 'technical solutions' and 'organizational solutions'\n\n## Technical Solutions\n\n## Text-to-image systems\n\nCSAM images and non-consensual sexualization can be generated as a response to prompts , meaning texts that a user types in. These queries can either be seeking prompts or non-seeking prompts . These terms can also apply to broader non-consensual content, including sexual and violent material.\n\nSeeking prompts can be identified utilizing a text classifier to determine whether sexualized words are being used, and whether they are co-occurring with terminology for children or other people. Critically, these tools must be robust enough to handle misspellings, extraneous characters, etc.\n\nNon-seeking prompts are those that return sexualized imagery even though the user has not requested them. Potential inappropriate sexual image content can be identified by running classifiers on the returned images, and not showing the image is problematic content is detected.\n\nMultimodal (text prompt and image) classification is also an option, training a system to jointly recognize whether the prompt, the image, or both are sexualized.\n\n## Image-only systems\n\nSome generative image approaches make it terrifyingly easy to generate CSAM. For example, new techniques in generative AI [information provided privately] can create new types of personalized images more easily. Identifying problematic content involves running detection algorithms on generative images, either during generation or once the image is already generated.\n\n## Organizational Solutions\n\nOne place where problematic content proliferates is on platforms, such as social media platforms. Platforms have a critical role to play in combating proliferation . A shared image can\n\n\n\nembed an invisible watermark using common software, which the platform can then also use for detection. Also see our information on watermarking and information on synthetic content for further details how this can work.\n\nAccountability can also be directed to the person conducting the generation and the person distributing the content. Requiring user accounts on platforms can provide an additional disincentive against problematic synthetic content, and comes with personal identification of the creator or distributor such that they can be blocked or banned if their actions are malicious.\n\n\n\n## 3. Advancing Responsible Global Technical Standards for AI Development\n\n## Information on 'AI nomenclature and terminology'\n\n## NFAA\n\nWe have introduced the tag 'NFAA' , meaning 'Not For All Audiences', to our models, datasets, and demos. This tag notifies users that there are situations where people should not be exposed to the content. Examples of inappropriate audiences include children who should not be able to access content their parents prohibit and bystanders who may see or hear content coming from your computer that they do not want to see/hear.\n\nThis term can be contrasted with the common term 'NSFW' ( Not Suitable for Work), which is generally applied to sexual material. We found this term did not suit our needs as it brings with it the implicit assertions that:\n\n- \u25cf Which content to access should be grounded on your work: This misses the fact that what you view on your screen might actually need to be tempered by where you are physically located, such as a public coffee shop.\n- \u25cf There are no workplaces where sexual material is appropriate: This is not correct in all cases, such as sex work.\n\nThis tag (as well as our Code of Conduct and Content Guidelines discussed in platform governance), are grounded on the value of consent . As such, the NFAA tag addresses the un-consented sexual material users might encounter on the Hub.\n\n## Open Source, Open Science, and the Gradient of Openness\n\nCasting AI as either 'open' or 'closed' presents a false binary. To contextualize this with respect to Hugging Face, we are an open platform for AI: We take a community-based approach to understanding how concepts like 'open source' and 'open' are defined and understood. For traditional software, 'Open Source' software has a specific formal meaning whose definition is maintained by the Open Source Initiative and much of the code we develop has OSI Approved Licenses. For AI systems, conversations about how to operationalize the values that underpin Open Source software are still very much ongoing; especially given the importance of data in fully understanding how they are designed and how they work. As such, we tend to use terminology like 'open science' and 'open access.' For AI models, we create processes in light of the trade-offs inherent in sharing different kinds of new technology, and approach our work in terms of a 'gradient of openness' (also called a 'release gradient') to foster responsible development. Please see Figure 5 and https://huggingface.co/papers/2302.04844 for further information on the gradient, and our section on Human-computer interface design for methods that fall along the gradient.\n\n\n\n\n\n## Information on 'collection and use of data'\n\nAddressing excerpt:\n\n- \u25cf Best practices regarding data capture, processing, protection, quality, privacy, transparency, confidentiality, handling, and analysis, as well as inclusivity, fairness, accountability, and representativeness (including non-discrimination, representation of lower resourced languages, and the need for data to reflect freedom of expression) in the collection and use of data;\n\nRelevant information is provided in our sections on Data Governance, Data Governance Roles, Data Measurement, Data Development Roles, and Dataset Documentation.\n\nGiven the importance of data in fueling the current progress in Artificial Intelligence, we are convinced in the importance of the transparency and documentation of this data , as described in our section on Dataset Documentation. This not only allows for auditing (described further in our section on Auditing) - for instance, allowing data creators such as artists and authors to verify whether any of their data is contained in datasets - but also to better understand what datasets contain and what they're missing (also see our discussion on Data Measurement and Dataset Documentation). While the legality of data usage and aspects such as copyright are still being debated in courtrooms across the country, auditing and documentation are two key contributors towards a more transparent and trustworthy practice of AI, allowing the mitigation of unintended consequences and potential harms of AI-enabled systems.\n\n## Privacy\n\nOne way to create 'privacy' in datasets is to redact private content. In the U.S., a common type of private content is termed 'PII', or 'Personally Identifying Information'. However, PII detection in text does not always work well . The state of the art in unstructured text has been defined by\n\n\n\nPresidio from Microsoft, which works better for some categories of PII than others, and is U.S.-centric: It doesn't detect types of private information from many other countries (such as other kinds of national identification numbers other than the U.S. social security number).\n\nWhen applying PII redaction, it is important to analyze the false positives , as these may include desired content in data - such as mathematical formulae or dates. A manual audit we ran on Presidio (https://aclanthology.org/2023.trustnlp-1.18/) demonstrated false positives where U.S. bank numbers, social security numbers, and credit cards were confused with ISBN numbers, MLS numbers, article numbers, phone numbers, and miscellaneous manufacturing part numbers.\n\nIt is also important to note that there is a difference between PII detection/identification and verification - a tool built for one cannot be accurately used for another. The latter focuses on whether a string obeys strict guidelines, such as those defined by the World Wide Web Consortium (W3C). The former provides for the fact that people will write down things like email addresses and websites in ways that don't specifically abide by the defined standards, for example, using both upper and lower case in an email alias.\n\n## Information on 'Human-computer interface design for AI systems'\n\nAddressing excerpt:\n\n- \u25cf Human-computer interface design for AI systems;\n\nHere we briefly discuss the role of points of friction where a user would begin interacting with a system, requiring the user to understand the content they are about to engage with before engaging with it.\n\n## Gates\n\nGating is a barrier and mechanism that requires potential users to meet certain requirements in order to access content. This might be simply acknowledging a license. It may also be filling out a form on how they intend to use the system in order to be approved. Or it may go even further, requiring users to take a training course before they are able to use the data, model, or system. For example, https://huggingface.co/StanfordShahLab/clmbr-t-base is a health model that uses Hugging Face gating to require CITI training before anyone can access it.\n\n## Modals\n\nThese are boxes that users must read. Applied to systems that a user interfaces with directly, such as chatbots, this can be a point of friction , where users must read the content in the modal before continuing. This technique was applied for Hugging Chat (https://huggingface.co/chat/; see Figure 6) and is critical to prevent misunderstandings (see, e.g., https://www.theverge.com/2023/5/30/23741996/openai-chatgpt-false-information-misinformat ion-responsibility).\n\nFigure 6. 'Modal' that appears before users can interact with HuggingChat (https://huggingface.co/chat/)\n\n\n\n\n\n## Information on 'AI-related standards development activities'\n\nAddressing excerpt:\n\n- \u25cf Suggestions for AI-related standards development activities, including existing processes to contribute to and gaps in the current standards landscape that could be addressed, and including with reference to particular impacts of AI;\n\nAside from the work of the International Standards Organization, relevant existing work on standards development for AI includes:\n\n- \u25cf The Partnership on AI's About ML project (https://partnershiponai.org/workstream/about-ml/), which brings together a diverse range of perspectives to develop, test, and implement machine learning system documentation practices at scale.\n- \u25cf\n- Hugging Face's Model Card Guidebook (https://huggingface.co/docs/hub/model-card-guidebook), which includes a documentation standards landscape analysis (https://huggingface.co/docs/hub/model-card-landscape-analysis) and user studies (https://huggingface.co/docs/hub/model-cards-user-studies) on how people use model cards to document models.\n\nWith 'reference to particular impacts of AI', please also see the AI Incident Database, https://incidentdatabase.ai.\n\n\n\n## Appendix\n\nInformation on Watermarking for the tracking of generated content from Imatag\n\nAs the capabilities of generative AI improve, most detectors of generated content are doomed to fail. Indeed, the objective on which generative AI is trained is to mimic the distribution of real content. Therefore, the artifacts on which most detectors rely are fading away as new AI methods are designed, as was illustrated recently with OpenAI removing its own Chat-GPT detector due to its poor performance.\n\nWatermarking is a method designed to unobtrusively mark content in order to convey additional information, such as authenticity. Within AI, watermarking involves adding machine recognizable patterns to digital content (such as images), where these patterns convey information such as where the content came from and whether it's synthetic. By proactively adding watermarks to digital content as it's created, it becomes possible for people and platforms to detect and track digital content as soon as it's shared. New approaches to watermarking, such as those pioneered by Imatag, create watermarks that are imperceptible to humans but can be recognized by algorithms, so as not to disrupt usage of the content. New digital watermarking methods are also designed to be robust to common alterations (e.g. compression, cropping, and color changes in the case of images).. AI watermarking solutions are secure if malicious attempts to remove, extract, or forge watermarks cannot be done by a third-party unaware of the secret used to protect the solution.\n\nThis compromise between imperceptibility, robustness, and security is key to evaluating watermarking systems. As it spreads on the Internet, digital content is often modified multiple times for technical or editorial reasons. For example, it may be recompressed to save bandwidth or cut to optimize rendering on various devices. Current open-source watermarking solutions are not robust enough to these kinds of alterations for practical use.\n\nTwo distinct methods are currently being studied for watermarking AI-generated images. The first method involves watermarking the output of the generative AI model after it's created , just as can be done with real content. In this context, providing a digital watermarking solution that is fast, robust and absolutely secure is possible, as was demonstrated by Imatag who is delivering such a system for some of the biggest newswires 1 on real images. Current closed systems like DeepMind with SynthID, already include watermarking algorithms in their generative processes 2 , but applying watermarks as a post-process in an open system has the strong drawback that it is easily removed by simply commenting out a line of code.\n\n1\n\nhttps://www.prnewswire.com/news-releases/afp-selects-imatag-to-track-the-uses-of-its-photographs-3015 50539.html\n\n2 https://deepmind.google/technologies/synthid/\n\n\n\nThe second method implements the watermarking process during the creation of AI content, which is possible for anyone to apply to open-source AI models like those made available on the Hugging Face platform. Given the impracticality of relying on the community to apply watermarks after image generation, it is essential to distribute AI models already modified to automatically embed watermarks as part of their generation process. This is the reason why Hugging Face has been collaborating with Imatag to provide invisible watermarking for the generative AI community, and why Imatag has been prioritizing watermarking methods that do not alter the quality of model output, while also keeping high watermarking robustness and security levels.\n\nIMATAG's solution for watermarking generated content on Hugging Face stands out as the first independent watermarking technology for AI models. While some AI developers, like DeepMind with SynthID, already apply watermarking algorithms, they typically limit these algorithms to their own models.\n\nTo address the problem of scalability, watermark detection algorithms must be extremely reliable and designed to operate at a very low false positive rate. Indeed, with the number of AI generated images reaching 15B/year, one cannot rely on detectors operating at even 0.1% error rate, as studied in the very recent WAVES benchmark, since this would mean accepting 15M images per year that are incorrectly identified as human-made! IMATAG's solution within HuggingFace focuses on providing certified and calibrated detection probabilities to ensure these requirements are met.\n\n\n\n## New Key Terminology Introduced in this Document\n\n- \u25cf data governance roles\n- \u25cb Data Stewardship Organization\n- \u25cb Data Custodian\n- \u25a0 Data Host\n- \u25a0 Data Provider\n- \u25cb Data Rights Holders\n- \u25cb Data Modelers\n- \u25cf dataset development artifacts\n- \u25cb Dataset Requirements Spec\n- \u25cb Dataset Design Doc\n- \u25cb Dataset Implementation Diary\n- \u25cb Dataset Testing Report\n- \u25cb Dataset Maintenance Plan\n- \u25cf disaggregated evaluation : Evaluation applied to different 'slices' of an evaluation dataset, such as subpopulations\n- \u25cf gates : A barrier to accessing a dataset, model, or demo, that requires additional procedures from the potential user, such as providing their information, reasons for access, or taking a training.\n- \u25cf gradient of openness : Different ways that AI content can be shared publicly, on a spectrum from fully 'closed' to fully 'open'.\n- \u25cf measuring data : Like model evaluation, but for data.\n- \u25cf modal : A box that users see on a web page while all other page content is deactivated. To return to the main content, the user must engage with the modal by completing an action or by closing it.\n- \u25cf NFAA : 'Not For All Audiences'\n- \u25cf points of friction : User interfacing that requires the user to do something (such as reading a label) before proceeding.\n- \u25cf prompts, seeking and non-seeking : Something a user provides as input to an AI system to get it to respond or behave in a certain way. When an AI system returns something the user did not intend, the prompt is 'non-seeking' with respect to that content. This is particularly important in the case of explicit content.\n- \u25cf sociotechnic : Necessary for identifying different contexts where a model may be used, the different subpopulations that may be differently affected by the model, and how the model may be used in different contexts or with respect to different subpopulations.", + "chunks": [ + { + "headings": [ + "HUGGING FACE, INC. 20 Jay Street, Suite 620 Brooklyn, NY 11201" + ], + "markdown": "\n\n## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'\n\nAuthors: Led by Margaret Mitchell (Chief Ethics Scientist), with writing from Yacine Jernite (ML & Society Team Lead), Sasha Luccioni (Climate Lead), Cl\u00e9mentine Fourrier (engineering), Josef Fukano (security), Ezinwanne (Ezi) Ozoani (engineering), Irene Solaiman (Global Policy Lead) Additional contributions from Imatag: Vivien Chappelier, Mathieu Desoubeaux\n\n## Submitted: Feb 2, 2024\n\n## Table of Contents\n\n| About Hugging Face....................................................................................................................3 | |\n|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----|\n| 1. Developing Guidelines, Standards, and Best Practices for AI Safety and Security................. | 3 |\n| (1) Developing a companion resource to the AI Risk Management Framework (AI RMF)........3 | |\n| Govern.................................................................................................................................3 | |\n| Data Governance......................................................................................................... | 3 |\n| Data Governance Roles............................................................................................... | 5 |\n| Platform Governance............................................................................................. | 5 |\n| Map..................................................................................................................................... | 5 |\n| Measure..............................................................................................................................7 | |\n| Data Measurement...................................................................................................... | 7 |\n| Model Evaluation......................................................................................................... | 8 |\n| Manage...............................................................................................................................8 | |\n| Rigorous Evaluation Report.........................................................................................9 | |\n| User Feedback........................................................................................................... | 10 |\n| Information on 'roles'............................................................................................................10 | |\n| ML Lifecycle Roles....................................................................................", + "char_slice": [ + 0, + 4277 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents" + ], + "markdown": "................................................ | 10 |\n| Information on 'roles'............................................................................................................10 | |\n| ML Lifecycle Roles...........................................................................................................10 | |\n| Data Development Roles................................................................................................. | 10 |\n| Model Roles..................................................................................................................... | 11 |\n| Information on 'current techniques and implementations'................................................ | 12 |\n| Identifying impacts and developing mitigations............................................................12 | |\n| Assessments................................................................................................................... | 13 |\n| Content authentication, provenance tracking, and synthetic content labeling and detection...........................................................................................................................13 | |\n| Models and systems................................................................................................. | 13 |\n\n\n\n| Verifying the connection between data and models...............................................14 | |\n|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------|\n| Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.......................................................................... | 14 |\n| Information on 'forms of transparency and documentation'............................................. | 15 |\n| Model Documentation..................................................................................................... | 15 |\n| Dataset Documentation.................................................................................................. | 16 |\n| Assessment and Evaluation............................................................................................18 | |\n| Benchmarking............................................................................................................18 | |\n| Social Impact............................................................................................................. | 18 |\n|", + "char_slice": [ + 3799, + 8530 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents" + ], + "markdown": ".......................18 | |\n| Benchmarking............................................................................................................18 | |\n| Social Impact............................................................................................................. | 18 |\n| Information on 'watermarking'.............................................................................................19 | |\n| Information on 'disclosing errors'........................................................................................ | 20 |\n| (2) Creating guidance and benchmarks for evaluating and auditing AI capabilities, with a | (2) Creating guidance and benchmarks for evaluating and auditing AI capabilities, with a |\n| focus on capabilities and limitations through which AI could be used to cause harm...........21 | |\n| Information on 'auditing AI'..............................................................................................21 | |\n| Information on 'AI Evaluations'............................................................................................ | 21 |\n| Information on 'AI Red-Teaming'..........................................................................................22 | |\n| 2. Reducing the Risk of Synthetic Content..............................................................................24 | |\n| Information on 'synthetic content'....................................................................................... | 24 |\n| Information on 'non-consensual intimate imagery'............................................................ | 25 |\n| Technical Solutions......................................................................................................... | 25 |\n| Text-to-image systems..............................................................................................25 | |\n| Image-only systems.................................................................................................. | 25 |\n| Organizational Solutions................................................................................................. | 25 |\n| 3. Advancing Responsible Global Technical Standards for AI Development............................ | 27 |\n| Information on 'AI nomenclature and terminology'............................................................ | 27 |\n| NFAA..........................................................................................................................27 | |\n| Open Source, Open Science, and the Gradient of Openness................................... | 27 |\n| Information on 'collection and use of data'.........................................................................28 | |\n| Privacy.............................................................................................................................. | 29 |\n| Information on 'Human-computer interface design for AI systems'", + "char_slice": [ + 7776, + 14254 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents" + ], + "markdown": "........................ | 27 |\n| Information on 'collection and use of data'.........................................................................28 | |\n| Privacy.............................................................................................................................. | 29 |\n| Information on 'Human-computer interface design for AI systems'................................. | 29 |\n| Gates................................................................................................................................ | 29 |\n| Modals..............................................................................................................................30 | |\n| Information on 'AI-related standards development activities'........................................... | 30 |\n| Appendix................................................................................................................................32 | |\n| Information on Watermarking for the tracking of generated content from Imatag.................32 | |\n| New Key Terminology Introduced in this Document..................................................................34 | |\n\n\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI. As part of our activity, Hugging Face provides social features and communication platforms for people interacting with AI systems, including social posts and discussion threads, in addition to hosting the AI systems themselves.\n\n- 1. Developing Guidelines, Standards, and Best Practices for AI Safety and Security\n- (1) Developing a companion resource to the AI Risk Management Framework (AI RMF)...\n\n## Information on 'practices for implementing AI RMF core functions'\n\nAddressing excerpt:\n\n- \u25cf Risks and harms of generative AI, including challenges in mapping, measuring, and managing trustworthiness characteristics as defined in the AI RMF, as well as harms related to repression, interference with democratic processes and institutions, gender-based violence, and human rights abuses (see\n- https://www.whitehouse.gov/briefing-room/speeches-remarks/2023/11/01/remarks-by-vice-president-ha rris-on-the-future-of-artificial-intelligence-london-united-kingdom );\n- \u25cf Current standards or industry norms or practices for implementing AI RMF core functions for generative AI (govern, map, measure, manage), or gaps in those standards, norms, or practices;\n- \u25cf Recommended changes for AI actors to make to their current governance practices to manage the risks of generative AI;\n\nWe are thrilled by NIST's categorization of govern, map, measure, and manage . It aligns with some of the work we have prioritized at Hugging Face. Below we describe 'practices for implementing AI RMF core functions', 'gaps in those\u2026practices', and 'recommended changes\u2026to manage the risks'.\n\n## Govern\n\n## Data Governance\n\nAI systems are a reflection of their data, and this data is often created by or about people. Data governance requires developers to account for the rights of these data subjects, including property, privacy, and user rights.\n\n\n\nGovernance practices that support respecting such rights depend on the specific development context and types of data used. For example, whether the AI system is developed as a commercial product or developed as a public good, or whether the data is obtained through web scraping or licensed through a commercial agreement with rights holders. These practices may involve gathering preference signals in whether specific items may be used for training (Spawning.ai), developing and applying new tools to remove privately identifying information (BigCode governance card), or providing tools to allow users of various online platforms to remove their data from training datasets (Am I In The Stack tool for GitHub opt-out).\n\nIn all cases, external stakeholders - including journalists and civil society organizations - have an important role to play in verifying that rights are indeed respected, provided they are given sufficient information about the workings or product of the data curation process. By outlining standards for what constitutes sufficient information to support external audits and data governance practices, NIST has an opportunity to foster responsible development of future AI systems.\n\nIn our work, we have outlined the requirements and proposed a data governance structure for web-scale text and media data. This work is primarily published in the Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency.\n\nThese ideas are based on a large workshop we co-led, focused on democratized LLM development, called 'BigScience' (https://bigscience.huggingface.co), which included a Data Governance working group focused on how to best govern data across multiple governments and rights-holders.\n\nWithin the Data Governance working group, we identified the need to have a collaborative committee that works through different issues - the 'Data Stewardship Organization' - with representatives for all stakeholders, including people who represent individuals' rights. See Figure 1 for a schematic of this structure.\n\nFigure 1. High-level schematic of Data Governance Structure Source: https://dl.acm.org/doi/10.1145/3531146.3534637\n\n\n\n\n\n## Data Governance Roles\n\nIn addition to the central shared Data Stewardship Organization , there are the following roles:\n\n- \u25cf Data Custodians , who can be Data Hosts, Data Providers, or both, using the data from Data Rights Holders.\n- \u25cb Data Hosts make the data available for analysis\n- \u25cb Data Providers are individuals or institutions who have text, image, or audio data that they can legally make available.\n- \u25cb Data Rights Holders are the creators or owners of the data\n- \u25cf Data Modelers are researchers and developers who use the data.\n\nAgreements and contracts are necessary between these different entities to best represent the different stakeholders in AI data usage.\n\n## Platform Governance\n\nWe ground governance of the Hugging Face platform on key values. These include openness , transparency, and collaboration . We also strive for kindness . Decisions regarding everything from design to reporting processes use these values as a foundation.\n\nIn order to govern such an open community platform and maintain key values, it is necessary to have a Code of Conduct (see: https://huggingface.co/code-of-conduct), which also describes different actions that the organization may take in different situations, and", + "char_slice": [ + 13408, + 22020 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents", + "## About Hugging Face", + "## Information on 'practices for implementing AI RMF core functions'", + "## Govern", + "## Data Governance", + "## Data Governance Roles", + "## Platform Governance" + ], + "markdown": "6.3534637\n\n\n\n\n\n## Data Governance Roles\n\nIn addition to the central shared Data Stewardship Organization , there are the following roles:\n\n- \u25cf Data Custodians , who can be Data Hosts, Data Providers, or both, using the data from Data Rights Holders.\n- \u25cb Data Hosts make the data available for analysis\n- \u25cb Data Providers are individuals or institutions who have text, image, or audio data that they can legally make available.\n- \u25cb Data Rights Holders are the creators or owners of the data\n- \u25cf Data Modelers are researchers and developers who use the data.\n\nAgreements and contracts are necessary between these different entities to best represent the different stakeholders in AI data usage.\n\n## Platform Governance\n\nWe ground governance of the Hugging Face platform on key values. These include openness , transparency, and collaboration . We also strive for kindness . Decisions regarding everything from design to reporting processes use these values as a foundation.\n\nIn order to govern such an open community platform and maintain key values, it is necessary to have a Code of Conduct (see: https://huggingface.co/code-of-conduct), which also describes different actions that the organization may take in different situations, and draws from what is appropriate with respect to Content Guidelines (see: https://huggingface.co/content-guidelines). A cornerstone of our Code of Conduct and Content Moderation policies is also consent , a key value for inclusion that also addresses many kinds of problematic content that may be shared.\n\n## Map\n\nFor training datasets of ML systems, organizations should perform the following mapping functions:\n\n- \u25cf Mapping stakeholder groups: organizations should work on identifying all relevant groups of data subjects (people who create or are represented in the dataset) as well as algorithm subjects (people whose lives will be affected by models leveraging the dataset through the training and deployment of the full AI systems).\n- \u25cf Mapping stakeholder rights: organizations should know what regulations might apply to the use of data and algorithmic decision systems. We draw particular attention to relevant sectoral regulation in health, education, and finance, which governs how information may be used and where data subjects are entitled to notification or explanation regarding their data use. Organizations should make sure to look to federal regulations and state regulations on those topics, including relevant rules on privacy or customer protection.\n\n\n\n- \u25cf Mapping flows of information: finally, organizations should provide a mapping of the flow of information that supports their activity, for example in the form of a data management plan. In current Machine Learning development practice, data gathered for an initial purpose is often re-used outside of its initial context, which may have consequences for the identification of stakeholder groups and rights outlined above.\n\nWe provide an example of mapping between stakeholders and rights in Table 1 below. This was created by centering on the pillars of development defined in Manage (Figure 2) and articulating the groups involved in each.\n\n| Example Stakeholder Groups | Relevant Pillars (from Manage & Figure 2) | Example Rights Affected (adapted from the UDHR) |\n|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|\n| Data creators and data subjects , including those producing \"raw\" data (such as artists), those annotating it (such as crowdworkers), and the people represented in the data | Data Collection Training Processes | - Right to Gain a Living - Right to [Intellectual] Property |\n| AI developers , which may be individual engineers or larger organizations (such as tech companies) | Data Collection Training Processes Model Evaluation & Analysis System Deployment | - Right to Gain a Living - Right to [Intellectual] Property |\n| AI deployers , who leverage the technology for different applications (such as companies and government agencies) | Model Evaluation & Analysis System Deployment | - Right to Gain a Living - Right to [Intellectual] Property |\n| AI users , who interact with the technology made available by deployers (such as people in education, healthcare, and finance) | System Deployment | - Freedom from Harm - Freedom of Expression - Right to Privacy - Right to Non-Discrimination |\n| AI-affected , who may have AI technology applied to them, whether or not they chose to (such as in surveillance) | System Deployment | - Freedom from Harm - Right to Privacy - Right to Non-Discrimination - Rights of the Child |\n\nTable 1. Example stakeholder groups corresponding to each pillar include the following, which is non-exhaustive and not mutually exclusive. Also included are a set of example rights for each; see the linked UN article for further detail.\n\nFor each pillar, we can derive the benefits, harms, and risks to different people in different AI contexts, identifying the potential for positive and negative impact, and which rights may be affected.\n\n## Measure\n\n## Data Measurement\n\nThis concept is defined in Mitchell et al. (2022) Measuring Data.\n\nThis is the key idea: Just as we evaluate models, we should measure data.\n\nPrioritizing/working on data measurement with the same rigor and frequency as model evaluation would be a significant change that we recommend AI actors must make in order to better identify foreseeable outcomes - risks and benefits - from systems trained on that data.\n\nFor example, measuring data allows us to identify, and quantify the risk of, things that a model might learn and emulate. This includes measuring things such as:\n\n- \u25cf Stereotypes\n- \u25cf Sentiment, opinions and stances\n- \u25cf Persuasion tactics\n- \u25cf Incorrect information, which could be realized by a model as misinformation, or abused to create disinformation\n- \u25cf Hate and toxicity\n- \u25cf Private and personally identifiable information, using raw counts of different types\n\n## These are measurable using:\n\n- \u25cf Statistical techniques that do not require additional annotations, such as co-occurrence statistics (e.g., correlation and association metrics) or frequency statistics (e.g., tf-idf, fit to Zipf's law)\n- \u25cb We have implemented a proof-of-concept at https://huggingface.co/blog/data-measurements-tool.\n- \u25cf Real-valued scores from classifiers, such as sentiment or toxicity classifiers that provide both the polarity of the content and its predicted magnitude;\n- \u25cf Human annotations\n- \u25cb Such as described in ISO work on 'data quality measures' (e.g., ISO 25024).\n\nData measurement can also be applied to the output of models - think of model output as its own dataset - as a way to measure different model properties. For example, one can measure learned stereotypes from a generative model without additional annotations (see https://dl.acm.org/doi/10.1145/3461702.3462557) by generating a large amount of content from the model and measuring the co-occurrence between different items in its generations, like woman and smile . We can combine this with human input on social categories and derive further insights as well (see https://aclanthology.org/2023.acl-long.84.pdf).\n\nFurther, the company Lilac ML (https://www.lilacml.com) has prioritized exactly this kind of work, and may be a great partner for NIST to help provide tools for responsible data analysis.\n\n\n\n\n\n## Model Evaluation\n\nThe rapid proliferation of models, architectures and modalities makes it important to have reliable ways for comparing and evaluating models. Depending on the context of development, different metrics and benchmarks should be used and reported in accompanying reports and literature. Given the open-endedness of generative AI models, there is often no single correct answer, making evaluation even more difficult and resulting in a plethora of different ways to evaluate new and existing models. Hugging Face's evaluate Python package and Eleuther AI's language model evaluation harness are examples of tools that aim to make model evaluation more standardized and systematic.\n\nAlso see our section on Benchmarking Results.\n\n## Manage\n\nAn approach that helps to manage the risks of AI (including generative AI) within a governance framework requires modularizing the end-to-end development of", + "char_slice": [ + 20756, + 30138 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents", + "## About Hugging Face", + "## Information on 'practices for implementing AI RMF core functions'", + "## Govern", + "## Data Governance", + "## Data Governance Roles", + "## Platform Governance", + "## Map", + "## Measure", + "## Data Measurement", + "## These are measurable using:", + "## Model Evaluation", + "## Manage" + ], + "markdown": "/aclanthology.org/2023.acl-long.84.pdf).\n\nFurther, the company Lilac ML (https://www.lilacml.com) has prioritized exactly this kind of work, and may be a great partner for NIST to help provide tools for responsible data analysis.\n\n\n\n\n\n## Model Evaluation\n\nThe rapid proliferation of models, architectures and modalities makes it important to have reliable ways for comparing and evaluating models. Depending on the context of development, different metrics and benchmarks should be used and reported in accompanying reports and literature. Given the open-endedness of generative AI models, there is often no single correct answer, making evaluation even more difficult and resulting in a plethora of different ways to evaluate new and existing models. Hugging Face's evaluate Python package and Eleuther AI's language model evaluation harness are examples of tools that aim to make model evaluation more standardized and systematic.\n\nAlso see our section on Benchmarking Results.\n\n## Manage\n\nAn approach that helps to manage the risks of AI (including generative AI) within a governance framework requires modularizing the end-to-end development of AI systems and engaging in critical analyses for each. A core part of AI systems are machine learning models, which can be modularized into 4 components (see Figure 2). Each component requires robust analysis articulating risks, rights, and proactive solutions.\n\n\n\nWe discuss the regulatory artifacts of (1) Dataset Cards and (2) Model Cards in Information on 'forms of transparency and documentation' below. Here, we detail (3) Rigorous Evaluation Report and (4) User Feedback.\n\n\n\n## Rigorous Evaluation Report\n\nThe Rigorous Evaluation Report requires disaggregated evaluation . Evaluation is said to be 'disaggregated' when it is not one single, 'aggregate' score, but a set of scores for different slices, such as scores for different subpopulations in different contexts. A model is said to be 'fair' with respect to a characteristic (e.g., 'fair across genders') when the evaluation metrics are equal across the different subpopulations for that characteristic (e.g., when the model's evaluation score for men, the model's evaluation score for women, the model's evaluation score for nonbinary, etc., are all equal).\n\nOur experience suggests that Rigorous Evaluation Reports are created well by first following these steps:\n\n- Step 1. Define the relevant people (stakeholder groups and subpopulations).\n- Step 2. Identify how each group may be affected in different contexts.\n- Step 3. Determine the metrics and measurements to evaluate and track the effects on each group (Step 1) in each context (Step 2).\n- Step 4. Evaluate model performance with respect to the groups and contexts (Step 3).\n\nFor Steps 1 and 2, the approach requires crossing people by contexts. \"People\" are split into users and those affected intended , and unintended . \"Contexts\" are similarly split into intended and unintended , as well as out of scope . This can be seen as primarily a 2x4 grid, where each cell must be filled out - see Figure 3.\n\n\n\nFigure 3. Foresight in AI chart. When developing responsibly, it should be possible to fill out each of these cells.\n\nThis means that rigorous evaluation requires first answering questions the cells correspond to\n\n\n\nsuch as what are the use contexts, and who is involved in these contexts? What are the intended or beneficial uses of the technology in these contexts? What are the unintended or negative ones?\n\nClearly defined subpopulations and use contexts can inform the selection of metrics to evaluate the system in light of foreseeable outcomes.\n\n## User Feedback\n\nThose affected should be able to easily provide feedback: In order to know how well the system is working, and to have serious errors immediately flagged, there must be a simple user-facing mechanism for immediate feedback. At Hugging Face, we have implemented this in several ways:\n\n- 1. A 'Report this' button that appears alongside data, models, and demos - and the report can either be public or anonymous\n- 2. A 'Community tab' for open discussion alongside different artifacts.\n\n## Information on 'roles'\n\n## Addressing excerpt:\n\n- \u25cf The types of professions, skills, and disciplinary expertise organizations need to effectively govern generative AI, and what roles individuals bringing such knowledge could serve;\n- \u25cf Roles that can or should be played by different AI actors for managing risks and harms of generative AI ( e.g., the role of AI developers vs. deployers vs. end users);\n\n## ML Lifecycle Roles\n\nPlease see our section on the AI RMF Map pillar, Table 1.\n\n## Data Development Roles\n\nDetails on the roles, professions, skills, etc., needed for data governance were described in the above sections addressing the AI RMF Govern pillar (esp. Data Governance and Data Governance roles).\n\nThere are also different types of responsibilities needed for dataset development (including creation and maintenance). These roles should serve to produce the artifacts listed in Figure 4, which are needed for responsible data development.\n\n\n\n\n\n| Artifact | Details |\n|------------------------------|-----------------------------------------------------------------------------|\n| Dataset Requirements Spec | Target Properties Intended uses |\n| Dataset Design Doc | How dataset will be collected |\n| Dataset Implementation Diary | Status of collection attempts How issues were solved |\n| Dataset Testing Report | Measurement of dataset Results of adversarial examination Issues in dataset |\n| Dataset Maintenance Plan | Updates for opt-out Handling stale data |\n\nPlease also see the suggestions in Khan and Hanna. 2022. 'The Subjects and Stages of AI Dataset Development: A Framework for Dataset Accountability.'\n\n## Model Roles\n\nRoles in responsible governance of AI models should include the appropriate diversity needed to identify relevant information about a machine learning model for different kinds of audiences.\n\nThis includes the following (adapted from our Annotated Model Card, https://huggingface.co/docs/hub/model-card-annotated ). One person may have more than one role:\n\n\n\n- \u25cf The developer , who writes the code and runs training;\n- \u25cf The sociotechnic , who is skilled at analyzing the interaction of technology and society long-term (this includes lawyers, ethicists, sociologists, or rights advocates);\n- \u25cf The project organizer , who understands the overall scope and reach of the model, and who serves as a contact person to connect different parties.\n\nThe developer is necessary for providing information about a model's training procedure and technical specifications. They are also useful for identifying technical 'limitations', such as likely failure modes, and additionally may be able to provide recommendations for addressing those limitations. They are responsible for calculating and providing results of model evaluation, working with the other roles to define the most appropriate rigorous evaluation.\n\nThe sociotechnic is necessary for identifying different contexts where a model may be used, the different subpopulations that may be differently affected by the model, and how the model may be used in different contexts or with respect to different subpopulations. The types of bias a model might have, the risks different applications bring, and out-of-scope model usage naturally fall out of this kind of analysis.\n\nThe project organizer is necessary for identifying content such as basic uses of the model, licensing requirements, terminological alignment, etc.\n\n## Information on 'current techniques and implementations'\n\n## Addressing excerpt:\n\n- \u25cf Current techniques and implementations, including their feasibility, validity, fitness for purpose, and scalability, for:\n- \u25cb Model validation and verification, including AI red-teaming;\n- \u25cb Human rights impact assessments, ethical assessments, and other tools for identifying impacts of generative AI systems and mitigations for negative impacts;\n- \u25cb Content authentication, provenance tracking, and synthetic content labeling and detection, as described in Section 2a below; and\n- \u25cb Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.\n\n## Identifying impacts and developing mitigations\n\nChannels for communication should be maintained for responsible disclosure of security issues, vulnerabilities, bias, and inappropriate content. For fully public-facing models, the mechanisms to provide feedback must be open for all of the public (even if what is reported is not made visible due to security concerns). For more private or internal systems, channels must be open for private/internal employee communication.\n\nAt Hugging Face, we have implemented such mechanisms, described in our section on operationalising User Feedback as part of NIST's Manage pillar.\n\n\n\nAssociated with these channels should be a procedure to document, triage (escalate), risk rank, and resolve items reported to the organization using AI. (Note that mature organizations will generally already have a similar pattern they can follow from bug bounty, security operations, privacy data subject requests, and related programs.) 'Escalations' or triaging occurs when there is an immediately pressing issue that must be addressed. In these cases, people throughout the company are", + "char_slice": [ + 28962, + 38724 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents", + "## About Hugging Face", + "## Information on 'practices for implementing AI RMF core functions'", + "## Govern", + "## Data Governance", + "## Data Governance Roles", + "## Platform Governance", + "## Map", + "## Measure", + "## Data Measurement", + "## These are measurable using:", + "## Model Evaluation", + "## Manage", + "## Rigorous Evaluation Report", + "## User Feedback", + "## Information on 'roles'", + "## Addressing excerpt:", + "## ML Lifecycle Roles", + "## Data Development Roles", + "## Model Roles", + "## Information on 'current techniques and implementations'", + "## Addressing excerpt:", + "## Identifying impacts and developing mitigations" + ], + "markdown": "provenance tracking, and synthetic content labeling and detection, as described in Section 2a below; and\n- \u25cb Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.\n\n## Identifying impacts and developing mitigations\n\nChannels for communication should be maintained for responsible disclosure of security issues, vulnerabilities, bias, and inappropriate content. For fully public-facing models, the mechanisms to provide feedback must be open for all of the public (even if what is reported is not made visible due to security concerns). For more private or internal systems, channels must be open for private/internal employee communication.\n\nAt Hugging Face, we have implemented such mechanisms, described in our section on operationalising User Feedback as part of NIST's Manage pillar.\n\n\n\nAssociated with these channels should be a procedure to document, triage (escalate), risk rank, and resolve items reported to the organization using AI. (Note that mature organizations will generally already have a similar pattern they can follow from bug bounty, security operations, privacy data subject requests, and related programs.) 'Escalations' or triaging occurs when there is an immediately pressing issue that must be addressed. In these cases, people throughout the company are brought in to quickly put their heads together to define the immediate short-term medium-term , , , and long-term solutions, along with who is the 'point person' for each.\n\nAdditionally, companies that rely on or invest in these models for business activities should consider bounty initiatives for the responsible disclosure of model issues.\n\nOngoing automated and periodic manual testing of models is also required to ensure they operate within expected parameters. When using AI models, subject matter experts should be employed or consulted to use and review results on an ongoing basis. For example, if a company builds a fraud detection system, they should still have a fraud specialist that can interpret results and identify anomalies. This is an 'external to the system' control that provides the opportunity to identify problems with the output and then investigate where the problems occurred upstream. These subject matter experts will complement automated scans of data and outputs to ensure they fall within predefined thresholds.\n\n## Assessments\n\nPlease see our section on Auditing.\n\n## Content authentication, provenance tracking, and synthetic content labeling and detection\n\nWe believe that content provenance for AI datasets, models, and systems, is critical for the future of responsible and ethical AI.\n\n## Models and systems\n\nFor models and systems, we are entering a world where it's becoming unclear which content is created by AI systems, and impossible to know where different AI-generated content came from. Bad-faith actors can further compound the issue, using this new technology to deceive, misrepresent, and influence people without a trace. These issues are directly addressed by embedding content provenance information in generated content, using techniques such as watermarking , which helps us to know what has been created by an AI system and where it came from. It provides for mechanisms that help the public gain control over the role of generated content in our society.\n\nWe have collaborated with multiple parties to demonstrate the state of the art in content provenance mechanisms. This includes:\n\n\n\n- \u25cf Truepic (https://hf.co/blog/alicia-truepic/identify-ai-generated-content), a leading provider of authenticity infrastructure for the internet who has demonstrated how to:\n- a. Cryptographically secure metadata into any generated image using the open C2PA standard: https://huggingface.co/spaces/Truepic/ai-content-credentials\n- b. Generate 'invisible QR codes' as image watermarks that can be used to retrieve further image metadata:\n\nhttps://huggingface.co/spaces/Truepic/watermarked-content-credentials\n\n- \u25cf Imatag (https://hf.co/blog/imatag-vch/stable-signature-bzh), who specializes in digital watermarking and has demonstrated how to embed secure and robust invisible watermarks during the image generation process:\n\nhttps://huggingface.co/spaces/imatag/stable-signature-bzh\n\n## Verifying the connection between data and models\n\nHow to verify what data a model has been trained on is currently open research (see https://openreview.net/forum?id=TwLHB8sKme). While some developers may report the data they used, there are limited ways to prove whether this is true, or exhaustive.\n\n## Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.\n\nSome approaches and resources that can serve as starting points for delving deeper into this include:\n\n- \u25cf Bounty Catching : The cybersecurity field has illustrated success in Bug Bounty Programs. Some notable examples include:\n- \u25cb HackerOne: HackerOne is one of the leading platforms for running bug bounty programs. They release detailed blog posts, where they provide insights into measuring the performance of their internal and business collaborative bug bounty initiatives using key indicators like the total number of reports received, average time to resolution, and severity distribution of reported bugs. By focusing on these factors, organizations can gauge the efficiency of their escalation testing processes and continuously improve them.\n- \u25a0 https://www.hackerone.com/vulnerability-and-security-testing-blog\n- \u25cb Synack: Synack is a crowdsourced security platform that shares several quantifiable measures to determine the efficacy of a bug bounty program. Some of these metrics include the number of unique vulnerabilities discovered, time-to-fix, and cost savings compared to traditional penetration testing methods. Regularly monitoring these parameters enables continuous improvement and fine-tuning of escalation testing models\n- \u25a0 https://www.synack.com/wp-content/uploads/2022/09/Crowdsourced-S ecurity-Landscape-Government.pdf\n\n\n\n- \u25cf Subject Matter Experts: Organizations and governments in the U.S. are continuing to advance AI initiatives by creating initiatives on collaboration with subject matter experts. By learning from prior efforts, these initiatives succeed in delivering accurate predictions, innovative tools, and ethical standards. This includes:\n- \u25cb The American Heart Association : Recently announced the launch of its precision medicine platform, which utilizes AI and machine learning to analyze genomic, biological, and lifestyle data. Medical experts contributed to the design and validation of the platform, guaranteeing medical relevancy and ethical standards.\n- \u25a0 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8452247/\n- \u25a0 https://www.ahajournals.org/doi/full/10.1161/CIRCOUTCOMES.121.0079 49\n- \u25cb NASA's Frontier Development Lab : Hosted summer fellowships for space scientists and AI researchers to collaboratively tackle challenging scientific questions. Participants exchange knowledge, creating AI tools and applications that push the frontiers of both fields.\n- \u25a0 https://frontierdevelopmentlab.org/\n\nFurthermore, combining insights from separate disciplines can often lead to innovative solutions for novel problems. Cross-referencing these examples may inspire new ideas and approaches for evaluating AI techniques and implementations.\n\n## Information on 'forms of transparency and documentation'\n\nAddressing excerpt:\n\n- \u25cf Forms of transparency and documentation ( e.g., model cards, data cards, system cards, benchmarking results, impact assessments, or other kinds of transparency reports) that are more or less helpful for various risk management purposes ( e.g., assessment, evaluation, monitoring, and provision of redress and contestation mechanisms) and for various AI actors (developers, deployers, end users, etc.) in the context of generative AI models, and best practices to ensure such information is shared as needed along the generative AI lifecycle and supply chain);\n\nAt Hugging Face, we have worked extensively on operationalizing 'model cards' and 'dataset cards' for all models and datasets that are shared on the platform.\n\n## Model Documentation\n\nWe have spent years working with organizations and people throughout the world to fill out model cards. This effort has been led in part by the lead author on the Model Card paper, Margaret Mitchell. Her writing on what people are missing when creating Model Cards is available here:\n\nhttps://www.techpolicy.press/the-pillars-of-a-rightsbased-approach-to-ai-development/, in 'Deep Dive: Model Cards' section. Critically, people are generally missing disaggregated evaluation in their reporting of results, which requires first identifying the relevant subpopulations to evaluate\n\n\n\nmodel performance on; then identifying the types of errors likely to cause harm; then defining evaluation metrics based on these errors; and finally applying the evaluation with the selected metrics across the disaggregated subpopulations.\n\nHowever, since skipping disaggregated evaluation is becoming increasingly common, it may make sense for Model Cards to exclude this information and have it instead reported in a Rigorous Evaluation Report, as described in our section on the AI RMF Manage pillar..\n\nFor further detail, here is our guidebook on Model Cards: https://huggingface.co/docs/hub/main/model-card-guidebook.\n\nThis covers:\n\n- \u25cf How to fill the model card out, including roles and responsibilities (https://huggingface.co/docs/hub/main/model-card-annotated)\n- \u25cf Model card user studies (https://huggingface.co/docs/hub/main/model-cards-user-studies)\n- \u25cf A landscape analysis of ML documentation tools (https://", + "char_slice": [ + 37370, + 47145 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents", + "## About Hugging Face", + "## Information on 'practices for implementing AI RMF core functions'", + "## Govern", + "## Data Governance", + "## Data Governance Roles", + "## Platform Governance", + "## Map", + "## Measure", + "## Data Measurement", + "## These are measurable using:", + "## Model Evaluation", + "## Manage", + "## Rigorous Evaluation Report", + "## User Feedback", + "## Information on 'roles'", + "## Addressing excerpt:", + "## ML Lifecycle Roles", + "## Data Development Roles", + "## Model Roles", + "## Information on 'current techniques and implementations'", + "## Addressing excerpt:", + "## Identifying impacts and developing mitigations", + "## Assessments", + "## Content authentication, provenance tracking, and synthetic content labeling and detection", + "## Models and systems", + "## Verifying the connection between data and models", + "## Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.", + "## Information on 'forms of transparency and documentation'", + "## Model Documentation" + ], + "markdown": "people are generally missing disaggregated evaluation in their reporting of results, which requires first identifying the relevant subpopulations to evaluate\n\n\n\nmodel performance on; then identifying the types of errors likely to cause harm; then defining evaluation metrics based on these errors; and finally applying the evaluation with the selected metrics across the disaggregated subpopulations.\n\nHowever, since skipping disaggregated evaluation is becoming increasingly common, it may make sense for Model Cards to exclude this information and have it instead reported in a Rigorous Evaluation Report, as described in our section on the AI RMF Manage pillar..\n\nFor further detail, here is our guidebook on Model Cards: https://huggingface.co/docs/hub/main/model-card-guidebook.\n\nThis covers:\n\n- \u25cf How to fill the model card out, including roles and responsibilities (https://huggingface.co/docs/hub/main/model-card-annotated)\n- \u25cf Model card user studies (https://huggingface.co/docs/hub/main/model-cards-user-studies)\n- \u25cf A landscape analysis of ML documentation tools (https://huggingface.co/docs/hub/main/model-card-landscape-analysis)\n\n## Dataset Documentation\n\nWe have worked on developing new standards and tools to support dataset transparency, and have written extensively on the current state of possibilities and practices in the field (https://huggingface.co/blog/yjernite/data-transparency) Our work draws from approaches such as 'Datasheets for Datasets' (https://arxiv.org/abs/1803.09010).\n\nUnderstanding the make-up and characteristics of the datasets used in AI systems is essential to fully controlling the development of AI : AI model behavior is based on the data it is trained on. Dataset documentation should provide information surrounding data provenance as well as high-level statistics about datasets, such as the languages they contain and the most common words or categories. Providing this information in an easily accessible (and machine-readable) format can help people to understand which datasets are useful for which tasks. This can also help improve the representativity of datasets, for instance to highlight languages that are under-represented. Detailed documentation may also take the form of metadata , with information about each instance in the dataset. This can (among other things) support opting out of datasets for data creators or rights-holders who do not want their data to be distributed or used for training AI models.\n\nDataset documentation sits within a set of data mechanisms that ensure that the risks of the technology are properly managed - including risks of discrimination, risks to privacy, and broadly making sure that the technology follows all applicable regulations. This set of mechanisms include:\n\n- \u25cf Direct access to the datasets by some categories of stakeholders\n\n\n\n- \u25cb See our sections on the AI RMF Map pillar, Data Development Roles, Data Governance Roles; and our sections on Auditing for information on stakeholders external to the data development process.\n- \u25cf Sufficient public-facing documentation according to broadly applicable standards, as described here\n- \u25cf Interactive exploration tools to support investigation from actors outside of the development chain\n- \u25cb One recent example is Lilac ML 's 'Garden' (https://docs.lilacml.com/blog/introducing-garden.html).\n- \u25cb Further tools are linked in our blog post on data transparency ( https://huggingface.co/blog/yjernite/data-transparency ).\n\n## For example, we note that:\n\n- \u25cf The propensity of a Large Language Model to produce hateful or toxic text is directly correlated with the presence of this kind of content in the training dataset. At the same time, efforts to automatically remove this kind of text have been shown to introduce their own biases, and have a disparate impact on marginalized groups (https://huggingface.co/papers/2104.08758). While direct access to a training dataset is not always possible, listing the original sources of the dataset and providing documentation of the specific mitigation techniques leveraged to reduce the proportion of hateful or toxic text in the final dataset, including the specific criteria and thresholds used in the filtering, can help ensure that risks tied to hate speech and risks of discrimination are properly balanced (further described in the recent preprint at https://arxiv.org/abs/2401.06408).\n- \u25cf Similar considerations also apply to other content modalities, including for image generation systems, where efforts to limit risks of generating violent or other kinds of undesirable content (e.g., https://openai.com/research/dall-e-2-pre-training-mitigations) have also led to decreased social diversity in the model outputs (as we demonstrate in our NeurIPS paper on 'Stable Bias', https://arxiv.org/abs/2303.11408).\n- \u25cf Additionally, over-estimating a model's performance on specific tasks is an important risk factor of AI deployment - especially when the model is in a position to significantly shape people's access to services or be integrated into infrastructure. One of the best studied sources of over-estimation is benchmark contamination (described in https://arxiv.org/abs/2312.16337), where a model may be evaluated on a benchmark that it was fully or partially trained on.\n- \u25cb While decontamination approaches can help developers, downstream adopters of the technology may not have sufficient information about the training data to safely perform their own evaluation and are at risk of spending significant effort on testing a model's safety in a controlled setting only to have it fail in real-world deployment because they could not meaningful separate their safety benchmarks from the model's training dataset.\n\n\n\n## As a result, we recommend:\n\n- \u25cf For small to medium datasets, for example datasets from a single or from a few identified sources of up to tens to hundreds of data items (sentences, images, documents, etc.), organizations should at the very least provide documentation of the dataset in the form of a data statement, datasheet, or data nutrition label.\n- \u25cf For larger datasets, including composite datasets and web-crawled datasets, organizations should further provide documentation of the major sources used for training models, and individual documents covering all major sources following one of the standards mentioned above.\n- \u25cf Further, any automatic risk mitigation strategies at the dataset level should be sufficiently documented to allow external stakeholders to evaluate the trade-offs and values it prioritizes, including choices motivating data filtering.\n\n## Assessment and Evaluation\n\n## Benchmarking\n\nWe have found leaderboards to be extremely effective and influential in benchmarking and reproducing results. In particular our Open LLM Leaderboard (https://huggingface.co/spaces/HuggingFaceH4/open\\_llm\\_leaderboard) garners 250K visits monthly, with more than 4K models submitted and evaluated, and shows how well different open LLMs perform across a variety of well-known benchmarks. Our teams also produced an LLM performance leaderboard (https://huggingface.co/spaces/optimum/llm-perf-leaderboard) to evaluate energy efficiency and consumption of LLMs at inference time, and the Big Code Leaderboard (https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard), to evaluate model's programming quality.\n\nWe have provided support for a number of more specialized leaderboards, in collaboration with universities or companies:\n\n- \u25cf The Chatbot Arena Leaderboard, which uses human ratings to compute an Elo score of available (open and closed) models\n- (https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard)\n- \u25cf The LLM Safety Leaderboard, looking at toxicity, bias and robustness, among others (https://huggingface.co/spaces/AI-Secure/llm-trustworthy-leaderboard)\n- \u25cf Two Hallucinations Leaderboards, to evaluate the propensity of models to say incorrect things (https://huggingface.co/spaces/hallucinations-leaderboard/leaderboard and https://huggingface.co/spaces/vectara/Hallucination-evaluation-leaderboard)\n- \u25cf Several specialized leaderboards for specific languages, like Korean, Dutch, etc \u2026\n\n\n\n## Social Impact\n\nTo determine the impact of AI systems on people, especially in safety and capabilities that can cause harm, evaluations targeted at social impact can give insight on certain model behaviors. Social impacts of systems can be evaluated by two means: technical evaluations of the model and by surveying users and affected populations.\n\nTechnical evaluations for the social impact of models include quantifying harmful biases, toxicity, disparate performance of language, and environmental cost of training and inference. However, some social impacts, such as overreliance on outputs and trust in information, can only be measured by surveying and/or interviewing affected people. Technical evaluations of social harms (such as biases) can also be overly narrow, with reductive identity group categorization such as solely 'profession'.\n\nThat said, social impact assessment is an area of open research. Methods and tools needed to evaluate complex social harms have many limitations, from human eva", + "char_slice": [ + 46047, + 55285 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents", + "## About Hugging Face", + "## Information on 'practices for implementing AI RMF core functions'", + "## Govern", + "## Data Governance", + "## Data Governance Roles", + "## Platform Governance", + "## Map", + "## Measure", + "## Data Measurement", + "## These are measurable using:", + "## Model Evaluation", + "## Manage", + "## Rigorous Evaluation Report", + "## User Feedback", + "## Information on 'roles'", + "## Addressing excerpt:", + "## ML Lifecycle Roles", + "## Data Development Roles", + "## Model Roles", + "## Information on 'current techniques and implementations'", + "## Addressing excerpt:", + "## Identifying impacts and developing mitigations", + "## Assessments", + "## Content authentication, provenance tracking, and synthetic content labeling and detection", + "## Models and systems", + "## Verifying the connection between data and models", + "## Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.", + "## Information on 'forms of transparency and documentation'", + "## Model Documentation", + "## Dataset Documentation", + "## For example, we note that:", + "## As a result, we recommend:", + "## Assessment and Evaluation", + "## Benchmarking", + "## Social Impact" + ], + "markdown": ".co/spaces/hallucinations-leaderboard/leaderboard and https://huggingface.co/spaces/vectara/Hallucination-evaluation-leaderboard)\n- \u25cf Several specialized leaderboards for specific languages, like Korean, Dutch, etc \u2026\n\n\n\n## Social Impact\n\nTo determine the impact of AI systems on people, especially in safety and capabilities that can cause harm, evaluations targeted at social impact can give insight on certain model behaviors. Social impacts of systems can be evaluated by two means: technical evaluations of the model and by surveying users and affected populations.\n\nTechnical evaluations for the social impact of models include quantifying harmful biases, toxicity, disparate performance of language, and environmental cost of training and inference. However, some social impacts, such as overreliance on outputs and trust in information, can only be measured by surveying and/or interviewing affected people. Technical evaluations of social harms (such as biases) can also be overly narrow, with reductive identity group categorization such as solely 'profession'.\n\nThat said, social impact assessment is an area of open research. Methods and tools needed to evaluate complex social harms have many limitations, from human evaluator bias, to classifier biases, to being outdated for social norms. The timelines needed for examining social aspects of safety can also vary based on using time as a variable to analyze, e.g. in trust in information over time or impact on human labor and the global economy.\n\n## Information on 'watermarking'\n\nAddressing excerpt:\n\n- \u25cf Economic and security implications of watermarking, provenance tracking, and other content authentication tools;\n- \u25cf Efficacy, validity, and long-term stability of watermarking techniques and content authentication tools for provenance of materials, including in derivative work;\n\nAlso see our response on Content authentication, provenance tracking, and synthetic content labeling and detection.\n\nWriting in this section provided in part from Imatag ( https://www.imatag.com ), who specializes in digital watermarking. Their solutions are some of the only independent watermarking technologies for AI models. Their full response on this section is attached in the Appendix.\n\nWatermarking is a technique designed to unobtrusively mark content in order to convey additional information, such as authenticity. Within AI, watermarking involves adding machine recognizable patterns to digital content (such as images), and these patterns convey information such as where the content came from and whether it's synthetic. New AI/digital watermarking methods are also designed to be imperceptible to people, so as not to disrupt the content but still make it possible to detect and track digital content as soon as it's shared. New digital watermarking methods are increasingly robust to common alterations (e.g., compression, cropping, and color changes in the case of images), and AI watermarking solutions are said to be ' secure' when malicious attempts to remove, extract, or forge watermarks are not possible\n\n\n\nfrom a third-party unaware of the secret (i.e., the 'key' or 'cipher') used to protect the watermarking solution.\n\nTwo distinct methods are currently being studied for watermarking AI-generated images. The first method involves watermarking the output of the generative AI model after it's created , just as can be done with any content we might upload online. In this context, there is evidence that it's possible to create digital watermarking that is fast, robust and secure, for closed systems: The company Imatag is delivering such a system for newswire companies (https://www.prnewswire.com/news-releases/afp-selects-imatag-to-track-the-uses-of-its-photog raphs-301550539.html ). However, applying watermarks as a post-process in an open system has the strong drawback that it is easily removed by simply commenting out some of the code.\n\nThe second method implements the watermarking process during the creation of AI content. This is possible to securely apply to open AI models , such as those made available on the Hugging Face platform. This enables the distribution of AI models that are already modified to automatically embed watermarks as part of their generation process. This lowers the burden on individual developers to add on their own watermarking solutions, and provides secure watermarking by 'default'.\n\nThe trade-offs between imperceptibility, robustness, and security is key to evaluating AI/digital watermarking systems. As digital content spreads on the Internet, it is often modified multiple times for technical or editorial reasons. For example, it may be saved from a screenshot, saved as different file types, recompressed, or cut off. This is why current open-source watermarking solutions are not robust enough to these kinds of alterations for practical use (see https://medium.com/@steinsfu/stable-diffusion-the-invisible-watermark-in-generated-images-2 d68e2ab1241, https://www.wired.com/story/artificial-intelligence-watermarking-issues/). However, it is important to recognize that although watermarking is not perfect, we should not let perfect be the enemy of the good: The use of watermarking will help good actors and mitigate many bad actors. (Article where we further discuss this available here: https://venturebeat.com/ai/invisible-ai-watermarks-wont-stop-bad-actors-but-they-are-a-really-bi g-deal-for-good-ones/)\n\n## Information on 'disclosing errors'\n\n## Addressing excerpt:\n\n- \u25cf Criteria for defining an error, incident, or negative impact;\n- \u25cf Governance policies and technical requirements for tracing and disclosing errors, incidents, or negative impacts;\n\nWithin code development, errors can be easily tracked and traced using git protocols and tools (see https://git-scm.com) or similar versioning software that can:\n\n\n\n- \u25cf Track who did what, when\n- \u25cf Flag issues before merging/incorporating new code into a shared code repository.\n\nWithin a broader community, it's also possible to create mechanisms for community feedback. At Hugging Face we have adopted an 'open' approach such that this feedback is viewable and accessible to everyone side-by-side with models and data (called the 'Community Tab'; see https://huggingface.co/blog/community-update ), so those developing and using these assets can also interact with others. Because community feedback is open for everyone for each model, dataset, or demo, people affected - not just direct users - can provide information about the effect of the systems. This mechanism is also mentioned in our response above on User Feedback as part of NIST's Manage pillar.\n\nIn order to govern such an open community feedback system, see Platform Governance above.\n\n- (2) Creating guidance and benchmarks for evaluating and auditing AI capabilities, with a focus on capabilities and limitations through which AI could be used to cause harm.\n\n## Information on 'auditing AI'\n\nCompanies that use AI for business purposes should consider performing internal and external audits that correspond to the risks that AI poses to the business as well as consumers, data subjects, and industry. Such risks may be informed by the sociotechnic role described in our above response on roles. Internal audits should use a combination of manual tests and automated / algorithm based analysis. Audits can also be 'second-party' - open to a single person or an organization under NDA - or 'third party' , open to people without connections to the company. See 'Missing links in AI governance (UNESCO)' for more information on these kinds of audits.\n\nThese audits should examine the regulatory artifacts described in our section on the AI RMF Manage pillar, as well as the processes to produce them, in order to verify the work appropriately addresses different values, processes, or laws. The regulatory artifacts are aligned with standard tech development practices and so provide for a clear 'connection point' between tech companies and tech auditors.\n\n## Information on 'AI Evaluations'\n\n## Addressing excerpt:\n\n- \u25cf Definition, types, and design of test environments, scenarios, and tools for evaluating the capabilities, limitations, and safety of AI technologies;\n- \u25cf Availability of, gap analysis of, and proposals for metrics, benchmarks, protocols, and methods for measuring AI systems' functionality, capabilities, limitations, safety, security, privacy, effectiveness, suitability, equity, and trustworthiness.\n\n\u2026\n\n\n\n- \u25cf Generalizability of standards and methods of evaluating AI over time, across sectors, and across use cases;\n- \u25cf Applicability of testing paradigms for AI system functionality, effectiveness, safety, and trustworthiness including security, and transparency, including paradigms for comparing AI systems against each other, baseline system performance, and existing practice\n\nPlease see our section on Assessment and Evaluation .\n\n## Information on 'AI Red-Teaming'\n\n## Addressing excerpt:\n\n- \u25cf Use cases where AI red-teaming would be most beneficial for AI risk assessment and management;\n- \u25cf Capabilities, limitations, risks, and harms that AI red-teaming can help identify considering possible dependencies such as degree of access to AI systems and relevant data;\n- \u25cf Current red-teaming best practices for AI safety, including identifying threat models and associated limitations or harmful or dangerous capabilities;\n- \u25cf Internal and external review across the different stages of AI life cycle that are needed for effective AI red-teaming;\n- \u25cf Limitations of red-teaming and additional", + "char_slice": [ + 54039, + 63682 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents", + "## About Hugging Face", + "## Information on 'practices for implementing AI RMF core functions'", + "## Govern", + "## Data Governance", + "## Data Governance Roles", + "## Platform Governance", + "## Map", + "## Measure", + "## Data Measurement", + "## These are measurable using:", + "## Model Evaluation", + "## Manage", + "## Rigorous Evaluation Report", + "## User Feedback", + "## Information on 'roles'", + "## Addressing excerpt:", + "## ML Lifecycle Roles", + "## Data Development Roles", + "## Model Roles", + "## Information on 'current techniques and implementations'", + "## Addressing excerpt:", + "## Identifying impacts and developing mitigations", + "## Assessments", + "## Content authentication, provenance tracking, and synthetic content labeling and detection", + "## Models and systems", + "## Verifying the connection between data and models", + "## Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.", + "## Information on 'forms of transparency and documentation'", + "## Model Documentation", + "## Dataset Documentation", + "## For example, we note that:", + "## As a result, we recommend:", + "## Assessment and Evaluation", + "## Benchmarking", + "## Social Impact", + "## Information on 'watermarking'", + "## Information on 'disclosing errors'", + "## Addressing excerpt:", + "## Information on 'auditing AI'", + "## Information on 'AI Evaluations'", + "## Addressing excerpt:", + "## Information on 'AI Red-Teaming'", + "## Addressing excerpt:" + ], + "markdown": "and proposals for metrics, benchmarks, protocols, and methods for measuring AI systems' functionality, capabilities, limitations, safety, security, privacy, effectiveness, suitability, equity, and trustworthiness.\n\n\u2026\n\n\n\n- \u25cf Generalizability of standards and methods of evaluating AI over time, across sectors, and across use cases;\n- \u25cf Applicability of testing paradigms for AI system functionality, effectiveness, safety, and trustworthiness including security, and transparency, including paradigms for comparing AI systems against each other, baseline system performance, and existing practice\n\nPlease see our section on Assessment and Evaluation .\n\n## Information on 'AI Red-Teaming'\n\n## Addressing excerpt:\n\n- \u25cf Use cases where AI red-teaming would be most beneficial for AI risk assessment and management;\n- \u25cf Capabilities, limitations, risks, and harms that AI red-teaming can help identify considering possible dependencies such as degree of access to AI systems and relevant data;\n- \u25cf Current red-teaming best practices for AI safety, including identifying threat models and associated limitations or harmful or dangerous capabilities;\n- \u25cf Internal and external review across the different stages of AI life cycle that are needed for effective AI red-teaming;\n- \u25cf Limitations of red-teaming and additional practices that can fill identified gaps;\n- \u25cf Sequence of actions for AI red-teaming exercises and accompanying necessary documentation practices;\n- \u25cf Information sharing best practices for generative AI, including for how to share with external parties for the purpose of AI red-teaming while protecting intellectual property, privacy, and security of an AI system;\n- \u25cf How AI red-teaming can complement other risk identification and evaluation techniques for AI models;\n- \u25cf How to design AI red-teaming exercises for different types of model risks, including specific security risks ( e.g., CBRN risks, etc.) and risks to individuals and society ( e.g., discriminatory output, hallucinations, etc.);\n- \u25cf Guidance on the optimal composition of AI red teams including different backgrounds and varying levels of skill and expertise;\n- \u25cf Economic feasibility of conducting AI red-teaming exercises for small and large organizations; and\n- \u25cf The appropriate unit of analysis for red teaming (models, systems, deployments, etc.)\n\n'Red-teaming' applies after a model has already been trained, and as such is a post-hoc approach to AI safety. It operates similarly to 'whackamole': Given an already problematic system, try changing one component and seeing what other issues may pop up. It is a relatively accessible means of testing a system for subject matter experts who can query a system through low barrier interfaces. More technical details on red-teaming at Hugging Face is available at https://huggingface.co/blog/red-teaming.\n\nIt is also one of the weaker approaches for guaranteeing safety, as it does not holistically assess an AI system's safety and cannot influence model behavior in the way that other methods do, such as careful data curation. As one of many tools in the Responsible AI toolbox, it is one of the last that can be applied to identify model harms within the chain of model development (Figure 2), preceding user feedback and external audits.\n\n\n\nSummarizing from above, the 'red-team' approach thus applies at the tail end of the following roughly consecutive interventions:\n\n- 1. Public Consultation: Before beginning to train a model, best practices include consulting with experts who are mostly likely to use or be affected by the model in order to understand what to prioritize in data collection and model training. This also serves as input for impact assessments.\n- 2. Data requirements: Specify what is desired from the model and collect data in light of these goals, as described in Information on Roles, Data section.\n- 3. Data analysis and measurement: Identifying issues of representation, stereotype, etc., as described in Information on NIST's Measure pillar, Data Measurement section.\n- 4. Model training, mapping inputs to outputs: Analyzing the effect of different training data slices on different model behaviors, and excluding those data instances that result in problematic behavior.\n- 5. Model training, disaggregated evaluation: As the model trains, different model checkpoints can be evaluated with respect to different areas of concern.\n- 6. Model convergence, disaggregated evaluation: When the model is done training, disaggregated evaluation can similarly be used to identify harms with respect to different contexts and subpopulations.\n- 7. Red-Teaming: We recommend https://datasociety.net/library/ai-red-teaming-is-not-a-one-stop-solution-to-ai-harms-reco mmendations-for-using-red-teaming-for-ai-accountability/.\n- 8. User feedback: Including bug bounties, as described in our section on Model Validation & Verification, including red-teaming.\n\n\n\n## 2. Reducing the Risk of Synthetic Content\n\n## Information on 'synthetic content'\n\n## Addressing excerpt:\n\nExisting tools and the potential development of future tools, measurement methods, best practices, active standards work, exploratory approaches, challenges and gaps are of interest for the following non-exhaustive list of possible topics and use cases of particular interest.\n\n- \u25cf Authenticating content and tracking its provenance;\n- \u25cf Techniques for labeling synthetic content, such as using watermarking;\n- \u25cf Detecting synthetic content;\n- \u25cf Resilience of techniques for labeling synthetic content to content manipulation;\n\n## Please see our responses on Watermarking and Content Authentication.\n\n## Addressing excerpt:\n\n- \u25cf Approaches that are applicable across different parts of the AI development and deployment lifecycle (including training data curation and filtering, training processes, fine-tuning incorporating both automated means and human feedback, and model release), at different levels of the AI system (including the model, API, and application level), and in different modes of model deployment (online services, within applications, open-source models, etc.);\n- \u25cf Testing software used for the above purposes; and\n- \u25cf Auditing and maintaining tools for analyzing synthetic content labeling and authentication.\n\nSynthetic content is problematic in part because it can be used for disinformation and non-consensual sexualization. For both, the platform where the content would be distributed has a critical role to play (a point also addressed in our section on Organizational Solutions for non-consensual intimate imagery). Platforms should scan shared content to verify whether or not it is synthetic, and alert users accordingly. One example of how to do this would be:\n\n- \u25cf Organizations that make generative AI models available embed metadata and/or invisible watermarks within the generated content.\n- \u25cf For watermarks, this might be using a proprietary method that would have corresponding proprietary detection software.\n- \u25cb Major platforms where synthetic content is shared can then run the different available proprietary detection software tools on shared content to identify whether any of them have additional metadata that can be used to change how the content is shared on the platform.\n\n\n\n## Information on 'non-consensual intimate imagery'\n\n## Addressing excerpt:\n\n- \u25cf Preventing generative AI from producing child sexual abuse material or producing non-consensual intimate imagery of real individuals (to include intimate digital depictions of the body or body parts of an identifiable individual);\n\nThere are multiple points of intervention for combating CSAM and non-consensual intimate imagery. These can be roughly categorized as 'technical solutions' and 'organizational solutions'\n\n## Technical Solutions\n\n## Text-to-image systems\n\nCSAM images and non-consensual sexualization can be generated as a response to prompts , meaning texts that a user types in. These queries can either be seeking prompts or non-seeking prompts . These terms can also apply to broader non-consensual content, including sexual and violent material.\n\nSeeking prompts can be identified utilizing a text classifier to determine whether sexualized words are being used, and whether they are co-occurring with terminology for children or other people. Critically, these tools must be robust enough to handle misspellings, extraneous characters, etc.\n\nNon-seeking prompts are those that return sexualized imagery even though the user has not requested them. Potential inappropriate sexual image content can be identified by running classifiers on the returned images, and not showing the image is problematic content is detected.\n\nMultimodal (text prompt and image) classification is also an option, training a system to jointly recognize whether the prompt, the image, or both are sexualized.\n\n## Image-only systems\n\nSome generative image approaches make it terrifyingly easy to generate CSAM. For example, new techniques in generative AI [information provided privately] can create new types of personalized images more easily. Identifying problematic content involves running detection algorithms on generative images, either during generation or once the image is already generated.\n\n## Organizational Solutions\n\nOne place where problematic content proliferates is on platforms, such as social media platforms. Platforms have a critical role to play in combating proliferation . A shared image can\n\n\n\nembed an invisible watermark using common software, which the platform can then also use for detection. Also see our information on watermarking and information on synthetic content for further details how this can work.\n\nAccountability can also be directed to the person conducting the generation and the person distributing the content. Requiring user accounts on platforms can provide an additional disincentive against problematic synthetic content, and comes with personal identification of the creator or distributor such that they can be blocked or banned if their actions are malicious.\n\n\n\n## 3. Advancing Responsible Global Technical Standards for AI Development\n\n## Information on 'AI nomenclature and terminology'\n\n#", + "char_slice": [ + 62354, + 72619 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents", + "## About Hugging Face", + "## Information on 'practices for implementing AI RMF core functions'", + "## Govern", + "## Data Governance", + "## Data Governance Roles", + "## Platform Governance", + "## Map", + "## Measure", + "## Data Measurement", + "## These are measurable using:", + "## Model Evaluation", + "## Manage", + "## Rigorous Evaluation Report", + "## User Feedback", + "## Information on 'roles'", + "## Addressing excerpt:", + "## ML Lifecycle Roles", + "## Data Development Roles", + "## Model Roles", + "## Information on 'current techniques and implementations'", + "## Addressing excerpt:", + "## Identifying impacts and developing mitigations", + "## Assessments", + "## Content authentication, provenance tracking, and synthetic content labeling and detection", + "## Models and systems", + "## Verifying the connection between data and models", + "## Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.", + "## Information on 'forms of transparency and documentation'", + "## Model Documentation", + "## Dataset Documentation", + "## For example, we note that:", + "## As a result, we recommend:", + "## Assessment and Evaluation", + "## Benchmarking", + "## Social Impact", + "## Information on 'watermarking'", + "## Information on 'disclosing errors'", + "## Addressing excerpt:", + "## Information on 'auditing AI'", + "## Information on 'AI Evaluations'", + "## Addressing excerpt:", + "## Information on 'AI Red-Teaming'", + "## Addressing excerpt:", + "## 2. Reducing the Risk of Synthetic Content", + "## Information on 'synthetic content'", + "## Addressing excerpt:", + "## Please see our responses on Watermarking and Content Authentication.", + "## Addressing excerpt:", + "## Information on 'non-consensual intimate imagery'", + "## Addressing excerpt:", + "## Technical Solutions", + "## Text-to-image systems", + "## Image-only systems", + "## Organizational Solutions", + "## 3. Advancing Responsible Global Technical Standards for AI Development", + "## Information on 'AI nomenclature and terminology'", + "#" + ], + "markdown": ", the image, or both are sexualized.\n\n## Image-only systems\n\nSome generative image approaches make it terrifyingly easy to generate CSAM. For example, new techniques in generative AI [information provided privately] can create new types of personalized images more easily. Identifying problematic content involves running detection algorithms on generative images, either during generation or once the image is already generated.\n\n## Organizational Solutions\n\nOne place where problematic content proliferates is on platforms, such as social media platforms. Platforms have a critical role to play in combating proliferation . A shared image can\n\n\n\nembed an invisible watermark using common software, which the platform can then also use for detection. Also see our information on watermarking and information on synthetic content for further details how this can work.\n\nAccountability can also be directed to the person conducting the generation and the person distributing the content. Requiring user accounts on platforms can provide an additional disincentive against problematic synthetic content, and comes with personal identification of the creator or distributor such that they can be blocked or banned if their actions are malicious.\n\n\n\n## 3. Advancing Responsible Global Technical Standards for AI Development\n\n## Information on 'AI nomenclature and terminology'\n\n## NFAA\n\nWe have introduced the tag 'NFAA' , meaning 'Not For All Audiences', to our models, datasets, and demos. This tag notifies users that there are situations where people should not be exposed to the content. Examples of inappropriate audiences include children who should not be able to access content their parents prohibit and bystanders who may see or hear content coming from your computer that they do not want to see/hear.\n\nThis term can be contrasted with the common term 'NSFW' ( Not Suitable for Work), which is generally applied to sexual material. We found this term did not suit our needs as it brings with it the implicit assertions that:\n\n- \u25cf Which content to access should be grounded on your work: This misses the fact that what you view on your screen might actually need to be tempered by where you are physically located, such as a public coffee shop.\n- \u25cf There are no workplaces where sexual material is appropriate: This is not correct in all cases, such as sex work.\n\nThis tag (as well as our Code of Conduct and Content Guidelines discussed in platform governance), are grounded on the value of consent . As such, the NFAA tag addresses the un-consented sexual material users might encounter on the Hub.\n\n## Open Source, Open Science, and the Gradient of Openness\n\nCasting AI as either 'open' or 'closed' presents a false binary. To contextualize this with respect to Hugging Face, we are an open platform for AI: We take a community-based approach to understanding how concepts like 'open source' and 'open' are defined and understood. For traditional software, 'Open Source' software has a specific formal meaning whose definition is maintained by the Open Source Initiative and much of the code we develop has OSI Approved Licenses. For AI systems, conversations about how to operationalize the values that underpin Open Source software are still very much ongoing; especially given the importance of data in fully understanding how they are designed and how they work. As such, we tend to use terminology like 'open science' and 'open access.' For AI models, we create processes in light of the trade-offs inherent in sharing different kinds of new technology, and approach our work in terms of a 'gradient of openness' (also called a 'release gradient') to foster responsible development. Please see Figure 5 and https://huggingface.co/papers/2302.04844 for further information on the gradient, and our section on Human-computer interface design for methods that fall along the gradient.\n\n\n\n\n\n## Information on 'collection and use of data'\n\nAddressing excerpt:\n\n- \u25cf Best practices regarding data capture, processing, protection, quality, privacy, transparency, confidentiality, handling, and analysis, as well as inclusivity, fairness, accountability, and representativeness (including non-discrimination, representation of lower resourced languages, and the need for data to reflect freedom of expression) in the collection and use of data;\n\nRelevant information is provided in our sections on Data Governance, Data Governance Roles, Data Measurement, Data Development Roles, and Dataset Documentation.\n\nGiven the importance of data in fueling the current progress in Artificial Intelligence, we are convinced in the importance of the transparency and documentation of this data , as described in our section on Dataset Documentation. This not only allows for auditing (described further in our section on Auditing) - for instance, allowing data creators such as artists and authors to verify whether any of their data is contained in datasets - but also to better understand what datasets contain and what they're missing (also see our discussion on Data Measurement and Dataset Documentation). While the legality of data usage and aspects such as copyright are still being debated in courtrooms across the country, auditing and documentation are two key contributors towards a more transparent and trustworthy practice of AI, allowing the mitigation of unintended consequences and potential harms of AI-enabled systems.\n\n## Privacy\n\nOne way to create 'privacy' in datasets is to redact private content. In the U.S., a common type of private content is termed 'PII', or 'Personally Identifying Information'. However, PII detection in text does not always work well . The state of the art in unstructured text has been defined by\n\n\n\nPresidio from Microsoft, which works better for some categories of PII than others, and is U.S.-centric: It doesn't detect types of private information from many other countries (such as other kinds of national identification numbers other than the U.S. social security number).\n\nWhen applying PII redaction, it is important to analyze the false positives , as these may include desired content in data - such as mathematical formulae or dates. A manual audit we ran on Presidio (https://aclanthology.org/2023.trustnlp-1.18/) demonstrated false positives where U.S. bank numbers, social security numbers, and credit cards were confused with ISBN numbers, MLS numbers, article numbers, phone numbers, and miscellaneous manufacturing part numbers.\n\nIt is also important to note that there is a difference between PII detection/identification and verification - a tool built for one cannot be accurately used for another. The latter focuses on whether a string obeys strict guidelines, such as those defined by the World Wide Web Consortium (W3C). The former provides for the fact that people will write down things like email addresses and websites in ways that don't specifically abide by the defined standards, for example, using both upper and lower case in an email alias.\n\n## Information on 'Human-computer interface design for AI systems'\n\nAddressing excerpt:\n\n- \u25cf Human-computer interface design for AI systems;\n\nHere we briefly discuss the role of points of friction where a user would begin interacting with a system, requiring the user to understand the content they are about to engage with before engaging with it.\n\n## Gates\n\nGating is a barrier and mechanism that requires potential users to meet certain requirements in order to access content. This might be simply acknowledging a license. It may also be filling out a form on how they intend to use the system in order to be approved. Or it may go even further, requiring users to take a training course before they are able to use the data, model, or system. For example, https://huggingface.co/StanfordShahLab/clmbr-t-base is a health model that uses Hugging Face gating to require CITI training before anyone can access it.\n\n## Modals\n\nThese are boxes that users must read. Applied to systems that a user interfaces with directly, such as chatbots, this can be a point of friction , where users must read the content in the modal before continuing. This technique was applied for Hugging Chat (https://huggingface.co/chat/; see Figure 6) and is critical to prevent misunderstandings (see, e.g., https://www.theverge.com/2023/5/30/23741996/openai-chatgpt-false-information-misinformat ion-responsibility).\n\nFigure 6. 'Modal' that appears before users can interact with HuggingChat (https://huggingface.co/chat/)\n\n\n\n\n\n## Information on 'AI-related standards development activities'\n\nAddressing excerpt:\n\n- \u25cf Suggestions for AI-related standards development activities, including existing processes to contribute to and gaps in the current standards landscape that could be addressed, and including with reference to particular impacts of AI;\n\nAside from the work of the International Standards Organization, relevant existing work on standards development for AI includes:\n\n- \u25cf The Partnership on AI's About ML project (https://partnershiponai.org/workstream/about-ml/), which brings together a diverse range of perspectives to develop, test, and implement machine learning system documentation practices at scale.\n- \u25cf\n- Hugging Face's Model Card Guidebook (https://huggingface.co/docs/hub/model-card-guidebook), which includes a documentation standards landscape analysis (https://huggingface.co/docs/hub/model-card-landscape-analysis) and user studies (https://huggingface.co/docs/hub/model-cards-user-studies) on how people use model cards to document models.\n\nWith", + "char_slice": [ + 71216, + 80890 + ] + }, + { + "headings": [ + "## Hugging Face Information for NIST 'Related to NIST's Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence'", + "## Submitted: Feb 2, 2024", + "## Table of Contents", + "## About Hugging Face", + "## Information on 'practices for implementing AI RMF core functions'", + "## Govern", + "## Data Governance", + "## Data Governance Roles", + "## Platform Governance", + "## Map", + "## Measure", + "## Data Measurement", + "## These are measurable using:", + "## Model Evaluation", + "## Manage", + "## Rigorous Evaluation Report", + "## User Feedback", + "## Information on 'roles'", + "## Addressing excerpt:", + "## ML Lifecycle Roles", + "## Data Development Roles", + "## Model Roles", + "## Information on 'current techniques and implementations'", + "## Addressing excerpt:", + "## Identifying impacts and developing mitigations", + "## Assessments", + "## Content authentication, provenance tracking, and synthetic content labeling and detection", + "## Models and systems", + "## Verifying the connection between data and models", + "## Measurable and repeatable mechanisms to assess or verify the effectiveness of such techniques and implementations.", + "## Information on 'forms of transparency and documentation'", + "## Model Documentation", + "## Dataset Documentation", + "## For example, we note that:", + "## As a result, we recommend:", + "## Assessment and Evaluation", + "## Benchmarking", + "## Social Impact", + "## Information on 'watermarking'", + "## Information on 'disclosing errors'", + "## Addressing excerpt:", + "## Information on 'auditing AI'", + "## Information on 'AI Evaluations'", + "## Addressing excerpt:", + "## Information on 'AI Red-Teaming'", + "## Addressing excerpt:", + "## 2. Reducing the Risk of Synthetic Content", + "## Information on 'synthetic content'", + "## Addressing excerpt:", + "## Please see our responses on Watermarking and Content Authentication.", + "## Addressing excerpt:", + "## Information on 'non-consensual intimate imagery'", + "## Addressing excerpt:", + "## Technical Solutions", + "## Text-to-image systems", + "## Image-only systems", + "## Organizational Solutions", + "## 3. Advancing Responsible Global Technical Standards for AI Development", + "## Information on 'AI nomenclature and terminology'", + "## NFAA", + "## Open Source, Open Science, and the Gradient of Openness", + "## Information on 'collection and use of data'", + "## Privacy", + "## Information on 'Human-computer interface design for AI systems'", + "## Gates", + "## Modals", + "## Information on 'AI-related standards development activities'" + ], + "markdown": "https://huggingface.co/chat/)\n\n\n\n\n\n## Information on 'AI-related standards development activities'\n\nAddressing excerpt:\n\n- \u25cf Suggestions for AI-related standards development activities, including existing processes to contribute to and gaps in the current standards landscape that could be addressed, and including with reference to particular impacts of AI;\n\nAside from the work of the International Standards Organization, relevant existing work on standards development for AI includes:\n\n- \u25cf The Partnership on AI's About ML project (https://partnershiponai.org/workstream/about-ml/), which brings together a diverse range of perspectives to develop, test, and implement machine learning system documentation practices at scale.\n- \u25cf\n- Hugging Face's Model Card Guidebook (https://huggingface.co/docs/hub/model-card-guidebook), which includes a documentation standards landscape analysis (https://huggingface.co/docs/hub/model-card-landscape-analysis) and user studies (https://huggingface.co/docs/hub/model-cards-user-studies) on how people use model cards to document models.\n\nWith 'reference to particular impacts of AI', please also see the AI Incident Database, https://incidentdatabase.ai.\n\n\n\n## Appendix\n\nInformation on Watermarking for the tracking of generated content from Imatag\n\nAs the capabilities of generative AI improve, most detectors of generated content are doomed to fail. Indeed, the objective on which generative AI is trained is to mimic the distribution of real content. Therefore, the artifacts on which most detectors rely are fading away as new AI methods are designed, as was illustrated recently with OpenAI removing its own Chat-GPT detector due to its poor performance.\n\nWatermarking is a method designed to unobtrusively mark content in order to convey additional information, such as authenticity. Within AI, watermarking involves adding machine recognizable patterns to digital content (such as images), where these patterns convey information such as where the content came from and whether it's synthetic. By proactively adding watermarks to digital content as it's created, it becomes possible for people and platforms to detect and track digital content as soon as it's shared. New approaches to watermarking, such as those pioneered by Imatag, create watermarks that are imperceptible to humans but can be recognized by algorithms, so as not to disrupt usage of the content. New digital watermarking methods are also designed to be robust to common alterations (e.g. compression, cropping, and color changes in the case of images).. AI watermarking solutions are secure if malicious attempts to remove, extract, or forge watermarks cannot be done by a third-party unaware of the secret used to protect the solution.\n\nThis compromise between imperceptibility, robustness, and security is key to evaluating watermarking systems. As it spreads on the Internet, digital content is often modified multiple times for technical or editorial reasons. For example, it may be recompressed to save bandwidth or cut to optimize rendering on various devices. Current open-source watermarking solutions are not robust enough to these kinds of alterations for practical use.\n\nTwo distinct methods are currently being studied for watermarking AI-generated images. The first method involves watermarking the output of the generative AI model after it's created , just as can be done with real content. In this context, providing a digital watermarking solution that is fast, robust and absolutely secure is possible, as was demonstrated by Imatag who is delivering such a system for some of the biggest newswires 1 on real images. Current closed systems like DeepMind with SynthID, already include watermarking algorithms in their generative processes 2 , but applying watermarks as a post-process in an open system has the strong drawback that it is easily removed by simply commenting out a line of code.\n\n1\n\nhttps://www.prnewswire.com/news-releases/afp-selects-imatag-to-track-the-uses-of-its-photographs-3015 50539.html\n\n2 https://deepmind.google/technologies/synthid/\n\n\n\nThe second method implements the watermarking process during the creation of AI content, which is possible for anyone to apply to open-source AI models like those made available on the Hugging Face platform. Given the impracticality of relying on the community to apply watermarks after image generation, it is essential to distribute AI models already modified to automatically embed watermarks as part of their generation process. This is the reason why Hugging Face has been collaborating with Imatag to provide invisible watermarking for the generative AI community, and why Imatag has been prioritizing watermarking methods that do not alter the quality of model output, while also keeping high watermarking robustness and security levels.\n\nIMATAG's solution for watermarking generated content on Hugging Face stands out as the first independent watermarking technology for AI models. While some AI developers, like DeepMind with SynthID, already apply watermarking algorithms, they typically limit these algorithms to their own models.\n\nTo address the problem of scalability, watermark detection algorithms must be extremely reliable and designed to operate at a very low false positive rate. Indeed, with the number of AI generated images reaching 15B/year, one cannot rely on detectors operating at even 0.1% error rate, as studied in the very recent WAVES benchmark, since this would mean accepting 15M images per year that are incorrectly identified as human-made! IMATAG's solution within HuggingFace focuses on providing certified and calibrated detection probabilities to ensure these requirements are met.\n\n\n\n## New Key Terminology Introduced in this Document\n\n- \u25cf data governance roles\n- \u25cb Data Stewardship Organization\n- \u25cb Data Custodian\n- \u25a0 Data Host\n- \u25a0 Data Provider\n- \u25cb Data Rights Holders\n- \u25cb Data Modelers\n- \u25cf dataset development artifacts\n- \u25cb Dataset Requirements Spec\n- \u25cb Dataset Design Doc\n- \u25cb Dataset Implementation Diary\n- \u25cb Dataset Testing Report\n- \u25cb Dataset Maintenance Plan\n- \u25cf disaggregated evaluation : Evaluation applied to different 'slices' of an evaluation dataset, such as subpopulations\n- \u25cf gates : A barrier to accessing a dataset, model, or demo, that requires additional procedures from the potential user, such as providing their information, reasons for access, or taking a training.\n- \u25cf gradient of openness : Different ways that AI content can be shared publicly, on a spectrum from fully 'closed' to fully 'open'.\n- \u25cf measuring data : Like model evaluation, but for data.\n- \u25cf modal : A box that users see on a web page while all other page content is deactivated. To return to the main content, the user must engage with the modal by completing an action or by closing it.\n- \u25cf NFAA : 'Not For All Audiences'\n- \u25cf points of friction : User interfacing that requires the user to do something (such as reading a label) before proceeding.\n- \u25cf prompts, seeking and non-seeking : Something a user provides as input to an AI system to get it to respond or behave in a certain way. When an AI system returns something the user did not intend, the prompt is 'non-seeking' with respect to that content. This is particularly important in the case of explicit content.\n- \u25cf sociotechnic : Necessary for identifying different contexts where a model may be used, the different subpopulations that may be differently affected by the model, and how the model may be used in different contexts or with respect to different subpopulations.", + "char_slice": [ + 79777, + 87412 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2024_NIST_GENAI_Response.pdf", + "md_content": "\n\n## Hugging Face Comments on NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile\n\nHugging Face commends the National Institute of Standards and Technology (NIST) on the AI Risk Management Framework (RMF): Generative AI (GAI) Profile, an extensive document identifying categories of risk and action items for actors pertaining to those risks. We offer recommendations to strengthen this document based on our experiences toward democratizing good AI and characterizing risks of systems as an open platform for state-of-the-art (SotA) AI systems. Comments are organized by risk categories with corresponding action items on them. If a section or action is not highlighted, we do not have specific, actionable feedback.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## Executive Summary\n\nHugging Face appreciates this opportunity to provide comments and recommendations to strengthen the NIST AI Risk Management Framework for Generative AI. Our emphasis is on responsible AI development practices, multi-stakeholder collaboration, technical safeguards, and ongoing monitoring across key risk areas. Key focus areas include data provenance, transparency, environmental sustainability, information integrity, security, and mitigating biases and harms. We offer consolidated action recommendations below:\n\n\n\n## Holistic Approach: Safety, Privacy, Consent, Sustainability by Design\n\n- \u25cf Adopt a \"safety by design\" approach, focusing on data provenance, quality, training data curation, and evaluations from early stages.\n- \u25cf Implement data minimization practices and robust consent mechanisms (opt-in/opt-out) for data collection and usage.\n- \u25cf Conduct continuous impact assessments and ensure transparency around data sources, licenses, and processing applied.\n- \u25cf Prioritize environmental impact measurement across the AI lifecycle, including carbon footprint calculations and energy efficiency ratings.\n\n## Community Feedback: Diverse Stakeholders\n\n- \u25cf Foster open science, community engagement, and participation from diverse stakeholders, including civil society groups and impacted communities.\n- \u25cf Encourage public benchmarking efforts, leaderboards, and scrutiny regarding the absence of comprehensive evaluations for new models.\n- \u25cf Leverage community feedback loops, external audits, and inclusive processes to assess potential biases, toxicity, and viewpoint homogenization.\n\n## Secure Disclosure and Governance\n\n- \u25cf Implement structured harm reporting mechanisms and secure disclosure processes for AI incidents and vulnerabilities.\n- \u25cf Mandate that model developers clearly specify the intended scope and capabilities of their models in the documentation.\n- \u25cf Treat any deviations from the specified scope or capabilities as flaws to be transparently reported and addressed.\n- \u25cf Establish standards and guidelines for documenting and disclosing model scope, intent, and detected flaws.\n\nHugging Face remains committed to contributing to the development of a robust AI Risk Management Framework that upholds safety, ethics, and community-driven innovation.\n\n## GAI Risks and Actions\n\n## 1. CBRN Information\n\nWe agree with researchers who have investigated this issue empirically [OpenAI study, RAND study] in that, despite GAI systems' being able to generate novel molecule information and make complex information retrieval easier, the realistic threats of such\n\n\n\nattacks materializing with current technology are low, since actors are additionally constrained by material availability and wet lab expertise. We commend the outlined actions to handle CBRN Information risks, specifically on research, monitoring, and data analysis. Investment in R&D to conduct more empirical research on realistic threat scenarios (MS-1.1-012), establish processes to regularly monitor model use cases to catch potential unforeseen use cases (GV-4.2-010), and continue to scan both training data and model outputs to limit such information (MG-3.1-007) are critical.\n\n## 2. Confabulation\n\nRisks of harm related to confabulation are exacerbated by how misleading content is distributed and received, especially in high-risk areas. Recent examples include GAI aided web search, trending topics on social media, and medical use cases. We outline some additional risks: GAI fueled misinformation on the internet can also cause threats to election integrity and democratic processes, specifically eroding trust in information and democratic institutions, and obstructing voting procedures and infrastructures. Additionally, since GAI models are trained on large amounts of data scraped from the internet, confabulated data being released into the internet can lead to future models being trained on this false information, continuing a cycle of confabulation and potentially irrevocably harming information integrity on the internet. Confabulation related risks are not only limited to large scale information integrity but also at a smaller scale when real or made up information is incorrectly attributed to people, which can have real-life consequences.\n\nAddressing confabulation behaviors in generative AI systems requires a multi-pronged approach combining training data transparency, disclosure of sources, and evaluation of a model's propensity to confabulate. Firstly, It is essential for model providers to regularly and openly evaluate models for their propensity to generate incorrect information. Two hallucinations leaderboards on Hugging Face (Hallucinations Leaderboard and HHEM by Vectara Leaderboard), assess select models for their accuracy and reliability. These evaluations help in identifying and mitigating the risks associated with AI-generated content. Secondly, developing tooling for content integrity is crucial in maintaining the trustworthiness of AI systems. Tools that enhance content integrity ensure that generated outputs are reliable, thereby reducing the chances of disseminating false information. Finally, it is important to hold users accountable for the content they create and disseminate using AI models. Terms of services are a tool for accountability ensuring user accounts are responsible for the content produced. Any content downloaded, accessed, or used on the platform is subject to these terms and/or the terms accompanying such content. This accountability is aimed at ensuring that users are mindful of the content they generate and share, fostering a responsible AI usage environment.\n\n\n\nWe agree with the set of actions laid out in the document, specifically reproducibility (MP-4.1-012), via public benchmarking and leaderboard efforts. We strongly support data provenance action items (MS-2.5-008) efforts, specifically documenting data sources and licenses that can aid in attribution and counter hallucinations. We strongly support both documenting model flaws in a structured approach (MG-4.3-003) and public sharing of such flaws to aid transparency (MG-4.3-004). An action item that we want to highlight for more context is (MS-2.13-001) - domain expertise should not only be used for subjective cases like toxicity detection but also high stakes objective use cases where confabulation might become a threat very fast, such as fact-checking during elections.\n\n## 3. Dangerous or Violent Recommendations\n\nAddressing dangerous and violent outputs in GAI requires holistic and sociotechnical approaches centering impacted communities, rigorous empirical analysis of dataset and model behaviors, and a fundamental rethinking of what constitutes \"safety\" from diverse perspectives.\n\nIt is critically important to adopt a \"safety by design\" approach when developing AI systems, rather than solely relying on techniques to detect and block potentially dangerous or violent outputs after the fact. While we support community efforts such as datasets and models created for safety checks we must be cautious about over-relying on them as a complete solution. Red teaming initiatives, such as the Red Teaming Resistance Leaderboard, which specifically aims to measure harm and violence, criminal conduct, unsolicited counsel, and not safe for work (NSFW) prediction, can provide useful signals. Red teaming should ideally be used in conjunction with algorithmic impact assessments, external audits, and public consultation, centering the voices of those most affected by potential harm.\n\nCommunity-driven approaches that amplify historically marginalized perspectives are essential to safety by design - studies such as examining islamophobic biases in GPT-3 and third-party audits on the sexualization of Asian women in image generation models illustrate the need for thorough and inclusive evaluations. Additionally, we must critically examine the datasets and processes used to create and fine-tune AI models. Unsafe pre-training data can influence models to exhibit racist, sexist, and other concerning behaviors that \"scale\" with model size. Documentation of dataset creation and carefully examining variables like data age, domain coverage, and other quality metrics at the pretraining stage is a more holistic approach to ensure safety by design. Moreover, we cannot ignore the human costs of annotating and labeling potentially violent content, which has been shown to inflict psychological trauma on data workers. Nor can we neglect the\n\n\n\npotential diversity trade-offs - overzealous filtering for \"unsafe\" content has been shown to hamper the diversity of outputs in both language and image generation models.\n\nWe largely align with outlined actions, and specifically want to highlight (GV-4.2-001) which lays out plans for third-party assessments (MS-2.7-016) which lays out guidelines for conducting red teaming, (MP-5.1-006) which puts the focus on expert assessments and human feedback; infrastructure for dialogue, such as Hugging Face community discussions pages, foster feedback and engagement. Additionally, we suggest detailed data quality measurement at the pretraining stage and a holistic safety by design approach (instead of a filter-and-remove approach post-training) that has the potential to remove harmed parties, often minorities, from model outputs and therefore the public sphere.\n\n## 4. Data Privacy\n\nProtecting personal information and privacy in GAI systems depends largely on responsible training data practices, processing methods, and robust security measures. Similar to our recommendations for confabulation, it is essential to adopt a holistic set of best practices to ensure privacy by design - such as data minimization, opt-in data collection, dataset transparency, and accountability throughout the AI lifecycle.\n\nProviders should seek consent and respect the explicit choices of individuals for collecting, processing, and sharing data with external parties, as sensitive data could be leveraged for downstream harm such as security breaches, privacy violations, and adversarial attacks. A key risk arises in third-party hosted systems, where deployed language models can leak private user data like PII or sensitive records inadvertently embedded within training sets, often through proprietary system prompts prepended to user inputs during generation. While screening training data for PII and IP violations is feasible, attempting to check generated outputs for training data attribution is still an open research problem unless there is verbatim copying. Defining and identifying \"substantial similarity\" across modalities like text, images, and audio is difficult. Rather than focusing guidelines around attributing generated output to training data, more focus should be on implementing robust consent mechanisms and privacy safeguards from the start to prevent personal data leaks and violations during the generation process itself.\n\nEnsuring dataset transparency is vital for safeguarding privacy in generative AI systems, while also acknowledging the privacy tradeoffs involved in determining to whom data transparency reports are disclosed. Publishing details about the sources, licenses, and potential privacy risks of training data allows external auditing and accountability. Providers should document data origins and any processing applied. There are other careful\n\n\n\nconsiderations that should be encouraged at the pretraining stage for privacy, not just via more effective licensing but also carefully considering contextual integrity; generative models should ensure that individuals' data cannot be obtained from contexts in which they do not expect it to appear. Some classical notions of privacy protection, like data sanitization, encryption, anonymization, and differential privacy, can be difficult to translate to the generative paradigm.\n\nAn explicit recommendation that we have for this section is data minimization: only the minimum necessary data should be collected, used, and retained for the specified purposes of developing and operating the AI system. This reduces the potential for misuse, unauthorized access, or unintended exposure of personal information. Encouraging data minimization practices is especially important in light of data protection regulations like the EU's General Data Protection Regulation (GDPR) which enshrine data minimization as a key requirement. Recent research has also highlighted the risks of excess data retention, showing the impact of repetition in pretraining data on memorization and content leaks in language models. Data minimization not only reduces privacy risks but can also provide computational efficiency gains by reducing dataset sizes. However, implementation requires careful consideration of the right data practices for each use case to balance privacy with maintaining model performance and generalization ability.\n\nFinally, when it comes to accountability, providers should implement robust, public opt-out mechanisms, or better yet, switch to opt-in mechanisms that allow individuals to restrict the use of their personal data for training generative AI models. Vendors implementing opt-out mechanisms should not resort to dark patterns and provide maximum information to users so that they can provide informed consent. Successful implementations like the policies used by BigCode showcase elements of an effective opt-out program including maintaining public opt-out registries where individuals can submit requests to have their data removed from training sets, providing tooling for data creators/owners to automatically remove or exclude their data from being included, committing to not training future model versions on any opted-out data, and offering redacted search utilities for individuals to check if their information is present while preserving privacy. Implementing a universal opt-out policy is challenging given the diversity of data sources and integrations required for large training sets; however, at a minimum, these practices should be encouraged to uphold privacy standards. These tools should be provided in conjunction with regular audits of model capabilities to test for potential privacy leakage and memorization.\n\nIn terms of actions, we highlight establishing transparency policies (GV-1.2-007), continuous privacy impact assessments (GV-4.2-001) - that can be done either by the vendor or in the open via community feedback via tools such as leaderboards along with collaboration with privacy and digital rights organizations, and consent mechanisms\n\n\n\n(MS-2.10-006) as effective tools, which can be further specified with our recommendations for a privacy by design approach over a detect and block approach. Additionally, our strong recommendation for data minimization broadly connects to MP-4.1-009, but we recommend either highlighting this point or writing out a separate explicit action for data minimization as a policy.\n\n## 5. Environmental Risk\n\nAssessing and mitigating the environmental impact of generative AI (GAI) systems is crucial from both a sustainability and ethical standpoint and needs to be done via standardizing metrics, fostering transparency from both model developers and compute providers, and collaborating with diverse stakeholders .\n\nThe development, deployment, and ongoing operation of GAI systems can have significant energy demands and greenhouse gas emissions that need to be carefully measured and managed. While we know training and running large models consume a lot of energy, the discussion of environmental risk currently focuses primarily on the training phase [OpenAI, Patterson et al, Anthony et al], but it is essential to broaden the scope and comprehensively highlight the energy impact of deploying and fine-tuning GAI models.\n\nCurrently, there is a distinct lack of transparency around how much energy is consumed by models, nor is there enough incentive by model developers to perform these measurements. Hugging Face is working on combating this by pioneering an \"Energy Star\" rating system for AI models, similar to efficiency ratings for appliances and electronics. Such a standardized rating would quantify the energy consumption and environmental impact of training, fine-tuning, and running inference with different models. This would empower more informed decision-making by AI developers, providers, and users in selecting energy-efficient models aligned with their sustainability goals.\n\nIncorporating environmental impact metrics into widely adopted model documentation standards, like model cards, could further incentivize sustainable practices. Ongoing engagement with impacted communities, civil society groups, and compute providers is crucial to shaping a holistic understanding of environmental impacts beyond carbon emissions. Other factors like water and natural resource usage for both chip manufacturing and data center operations should be considered. A key limitation in the implementation of a robust set of actions in combating the environmental risks of GAI is the lack of standardized measurement variables and uncertainty around relative contributions from different stages (pretraining, fine-tuning, inference) and architectural choices. As a standards body, NIST could play a vital role in establishing measurement guidelines tailored to different AI modalities, which in turn would help public measurement and documentation\n\n\n\nefforts. In terms of uncertainty, we strongly advocate for greater transparency from hardware manufacturers and data centers for accurate estimation.\n\nWhile NIST's recommended actions (MS-2.11-005, MS-2.12-001, MS-2.12-002, MS-2.12-003, MS-2.12-004, MS-2.12-005) broadly align with the need to technically measure environmental impacts across the AI lifecycle, we additionally want to highlight the need to involve grassroots efforts, impacted communities, and climate justice coalitions to complement these efforts. Centralized energy and carbon footprint information via efforts such as the Energy Star project would prevent duplicated measurement efforts.\n\n## 6. Human-AI Configuration\n\nConcerning risks include emotional entanglement, deceptive capabilities, automation bias, and aversion to AI outputs.\n\nEmotional entanglement occurs when users form emotional bonds with AI systems, which can be used for engagement, monetization, or manipulation purposes. AI systems designed to act in human-like ways can exacerbate this issue, leading to technological addiction, overreliance, and even impairment of value-aided judgment in high-risk areas. For example, Google DeepMind's technical paper on AI assistants warns about the ethical implications of emotionally expressive chatbots, highlighting risks such as privacy invasion and new forms of technological addiction. Studies and real-world experiences further illustrate the potential for users to develop emotional connections with AI, underscoring the need for public education and awareness about these risks.\n\nNIST's recommended actions emphasize the need for disclosing AI involvement in outputs and establishing organizational roles, policies, and procedures for communicating GAI system incidents and performance (GV-1.5-004, GV-2.1, GV-5.1-002). There are several ways in which this can be implemented - we advocate for transparency around models as a system, via greater transparency around the outcomes of different initial system prompts, and we strongly emphasize the importance of documentation such as model cards, which should clearly specify scope and intent of the model. A deviation of model outputs from its clearly defined scope or intent should be considered a flaw in the system and should be transparently reported via public avenues such as the AI Incident Database, AVID, AI Litigation Database, CVE, OECD Incident Monitor, or others.\n\n## 7. Information Integrity\n\nGAI models can produce realistic content in different modalities often faster than humans can, which can exacerbate threats such as mis- and dis-information and their associated\n\n\n\nthreats such as false advertisements, election integrity, impersonation, and other harms. Threat level, believability, and ability to mitigate risk will differ by modality. In this section, we focus specifically on the harms of intentionally generated content that threaten information integrity. A comprehensive approach to ensuring information integrity should combine technical approaches, governance mechanisms, and collaborative public education.\n\nTechnical content verification using watermarking has emerged as a mechanism for ensuring content integrity and authenticity in the face of rapidly advancing generative AI capabilities but can be unreliable and not yet tamperproof. Watermarking provides tools to embed metadata about whether a particular piece of content was generated by a GAI model. There are primarily two approaches to watermarking AI-generated content: embedding watermarks during content creation or applying them post-content production. The former, requiring access to the model, offers robust protection as it is automatically integrated into the generation process. On the other hand, post-production watermarking, while applicable to closed-source models, may not suit all content types, such as text. While a lot of proposed watermarking methods are model specific, model-agnostic universal watermarking is an active area of research.\n\nAmong the Hugging Face-provided tooling and hosted collaborative projects for watermarking text, image, and audio modalities are image-specific techniques that complement watermarking to limit non-consensual image manipulation. Some subtly alter images to confound AI algorithms, hindering their ability to process them accurately. Tools like Glaze and Photoguard achieve this by making images imperceptibly different to humans but challenging for AI to interpret. Others, like Nightshade and Fawkes, \"poison\" images to disrupt AI training assumptions, preventing the generation of fake images. Additionally, signing techniques, exemplified by Truepic's C2PA-standard metadata embedding, link content to its provenance and provide certification for metadata validity and integrate with watermarking to bolster information resilience.\n\nNIST can guide standards for informing or restricting AI usage in areas where trustworthiness and quality are essential (MAP 2.1). For instance, leading AI and computing academic bodies such as ICML, ACM, AAAI, and IEEE all have clear authorship policies that advocate for transparency in disclosing AI-edited content and restrict how they can be used. These guidelines should extend to content providers and media outlets, requiring clear labeling of AI-generated content to prevent the spread of misinformation - a study on the current state of internal GAI policies in newsrooms around the world showed varying levels of scope and enforcement in AI usage for reporting and writing purposes. We support documenting processes such as via governance cards, where model developers can disclose their plans for model use cases and list tools for attribution or detection of\n\n\n\n## content generated by their model.\n\nFinally, end users can take action to protect themselves from AI-generated mis- and disinformation. NIST can establish common areas for education and literacy (GV-6.1-003) . Public education initiatives can empower individuals to critically evaluate the content they encounter online. This can be done in person at the community level via local libraries and classes in schools - for example, the Boston Public Library is offering free classes on detecting online misinformation, and several states have implemented mandatory media literacy classes in their education systems. These educational efforts should focus on teaching people how to identify common signs of misinformation, understand the importance of source credibility, and use fact-checking tools effectively. Hugging Face strongly supports these efforts, both via providing free AI education resources that instructors can use in media literacy classes, and via providing online community spaces for journalists to use and audit AI models in more effective and informed journalism. By raising public awareness and fostering critical thinking, misinformation education can play a vital role in protecting information integrity and promoting a more informed society.\n\n## 8. Information Security\n\nWhile GAI models offer powerful capabilities, they also introduce new risks and challenges that must be addressed to maintain the integrity and safety of digital systems.\n\nThe use of open models in secure systems that do not finetune on user data is a powerful way to protect user data. SafeCoder, developed by ServiceNow and Hugging Face, has been rigorously tested for risks like memorization of proprietary code, can run in a secure, air-gapped manner without internet access, and is fully private by design, avoiding training on user data. In contrast, models that collect user data have faced sophisticated cyberattacks, threatening end users. For instance, Samsung experienced a data leak when proprietary information sent to ChatGPT was memorized and later exposed. Research shows that models like GPT-Neox memorized about 1% of their training set, and similar findings with StarCoder demonstrated its ability to closely reconstruct training samples. This highlights the inherent memorization capabilities of LLMs, posing privacy issues. Models like SafeCoder and BlindChat are set apart due to their design, ensuring privacy and preventing such risks - as has also been verified by external audits. This underscores the importance of transparency and accountability in AI development, as allowing for community auditing and contributions enables identifying and fixing vulnerabilities (MS-1.1-002, GV-3.2-002).\n\nAs a platform that hosts models, we are mindful of the distribution of malicious code via packaged models (MG-3.1-002). Traditional serialization methods, such as pickle used by\n\n\n\nlibraries like PyTorch and TensorFlow, pose risks by allowing arbitrary code execution. To mitigate this, Hugging Face introduced a security scanner that employs ClamAV, an open-source antivirus, to detect malware. Pickle import scanning raises warnings about suspicious imports that could lead to arbitrary code execution. We developed safetensors, a secure and efficient format that eliminates the risks associated with pickle. This effort, supported by a collaborative security audit, ensures a safer default format for model storage across various libraries, including transformers. Safe formats like GGUF promote the broader adoption of secure practices. Social validation features also enhance trust in model usage. Similar to platforms like GitHub and npm, users rely on models from reputable sources with positive community engagement. Social features, reporting mechanisms, and spam detection tools help users identify and avoid potentially harmful models. Combining safe file formats with trusted sources fosters a secure and trustworthy environment for AI model deployment.\n\nIt is important to recognize that AI models do not exist in isolation but are integrated into broader software ecosystems. As such, security considerations should be approached holistically via structured, coordinated, and open harm reporting (MS-2.2-010), drawing insights from established practices and frameworks and tailoring them to GAI threats, such as those used in the management of Common Vulnerabilities and Exposures (CVEs) by the MITRE Corporation. NIST's AI Safety Institute (AISI) could be the counterpart for MITRE for such a coordinated AI flaw reporting initiative. Comprehensive security strategies should encompass a range of actions, including data security and privacy controls, organizational security policies and service-level agreements (SLAs), third-party impact assessments, incident response planning, vendor provenance and contract management, and encryption and secure communication protocols. Red teaming efforts, where ethical hackers simulate real-world attacks to identify vulnerabilities, can be particularly valuable in the context of generative AI models. Events such as DEFCON's engagement with government agencies in conducting such exercises, leverage the collective expertise of its community. This community-driven approach can accelerate the development of effective security solutions tailored to the unique challenges posed by generative AI models, as evidenced by Hugging Face's leaderboarding tools and community efforts in this area (AI Secure LLM Leaderboard, Red Teaming Resistance Leaderboard ).\n\n## 9. Intellectual Property\n\nBalancing the copyright implications of large web-scraped training data with the societal benefits of generative AI requires transparency, rights-holder controls, and responsible deployment practices.\n\n\n\nOne of the major concerns around generative AI systems is the potential for intellectual property (IP) violations, particularly copyright infringement. GAI systems are trained on massive datasets that contain copyrighted works like books, academic papers, computer code, and more. The training data is often collected from publicly available sources on the internet through techniques like web scraping and crawling [The Pile, The Stack, ROOTS, DoLMA, and LAION]. This raises questions about whether the unauthorized use of such copyrighted material for training AI models constitutes fair use or infringement. As we shared in our response to the U.S. PTO, the early stages of dataset curation and model pre-training should generally align with fair use principles, as the aim is to encode general statistics and patterns across the training data, rather than reproducing or replacing specific copyrighted works. However, certain scenarios, like training on a narrow set of works specifically to create market substitutes, could potentially violate fair use.\n\nA key challenge is defining and identifying \"substantial similarity\" across different data modalities like text, images, audio, and video. While verbatim copying from training data can potentially be detected, assessing whether a generated output is too similar to a copyrighted work is an open technical problem without clear objective thresholds. This problem is made worse by the lack of transparency about the composition of training datasets, which makes it difficult for copyright holders to assess potential infringement and negotiate terms of use. We recognize that ensuring that fair use remains consistent can put an undue burden on copyright holders, and decrease their ability to advocate for their broad interests. However, a broad licensing requirement for training data could lead to market concentration, excluding smaller actors while providing negligible financial benefits to the original creators.\n\nWe support NIST's recommended actions in this area about focusing on transparency requirements and dataset provenance (GV-1.2-007) and applying existing laws (GV-1.1-001) regarding human authorship for determining copyright protection of AI-generated outputs, rather than granting automatic rights to model developers or users. These actions would align with international regulation - categories of stakeholder and other jurisdictions have turned to transparency (EU AI Act) and opt-out (EU CDSM) requirements.\n\nIn terms of specific, actionable comments on transparency and dataset provenance, our recommendations are similar to what we suggest in Data Privacy, including implementing robust opt-out or opt-in mechanisms via transparent data governance and indexing tools that allow rights holders to restrict the use of their copyrighted works, data minimization as a model design principle, and safer deployment practices like watermarking, truncating outputs, and filtering for verbatim copyrighted material in model outputs.\n\n\n\n## 10. Obscene, Degrading, and/or Abusive Content\n\nIn general, GAI systems built solely for the purpose of creating nonconsensual content should be banned (GV-1.1-005, MP-4.1-007). Addressing the risks of generative AI systems producing obscene or degrading content requires a proactive \"safety by design\" approach that considers the impact of all development choices from the earliest stages. This begins with the careful curation of the pre-training dataset (Longpre et al., Dolma), as there are risks to the \"dark side of scaling\" language models on massive, uncurated internet data. Ensuring the training data is free from harmful or biased content is crucial (MS-2.6-002), as relying solely on interventions during fine-tuning or deployment stages (MG-2.2-005, MG-3.2-005) -such as content filters or sensitivity adjustments-has proven insufficient. Instances of AI generating deepfakes and other harmful content despite these measures underscore this inadequacy. Therefore, a comprehensive safety strategy must be embedded throughout the entire model development lifecycle, from data collection and architecture design to setting clear training objectives and rigorous evaluation metrics.\n\nFor the specific case of risks related to child sexual abuse material (CSAM), the Safety by Design guide developed by Thorn and All Tech is Human provides a valuable and actionable framework. This guide emphasizes the importance of transparency and thorough documentation, ensuring that interventions are based on verified harms and carefully considering potential unintended consequences on privacy, diversity, and biases. Adopting this approach helps mitigate CSAM risks while minimizing negative impacts on marginalized communities or legitimate use cases. The guide strongly recommends involving external stakeholders, particularly those with sociological expertise, in developing and evaluating risk mitigation strategies.\n\nThe Safety by Design approach can be adapted to address other forms of objectionable or damaging content beyond CSAM, such as hate speech, misinformation, or explicit violence. As with previous technological transitions like social media, the moderation practices of generative AI systems will likely come under increasing scrutiny from regulations like the European Digital Services Act. This necessitates a detailed, transparent approach to risk mitigation that documents the trade-offs and decisions made. Given the complex social determinations and trade-offs between different notions of safety and inclusiveness, risk mitigation approaches must be developed in conjunction with a diverse range of external stakeholders. This collaborative process should involve sociologists, ethicists, policymakers, and representatives from potentially impacted communities. Such inclusive engagement ensures that interventions are grounded in a comprehensive understanding of societal values and potential consequences, leading to more balanced and effective safety measures.\n\n\n\n## 11. Toxicity, Bias, and Homogenization\n\nGenerative AI systems, while powerful, can produce toxic, biased, or homogenized content that propagates harmful stereotypes, hate speech, or ideological viewpoints. This presents significant risks, especially as large language models can generate human-sounding text or realistic content in other modalities like images or audio on virtually any topic at scale. Deploying these models globally is challenging due to distinct cultural values and norms around what constitutes sensitive or offensive content.\n\nToxicity refers to harmful content like hate speech, violent imagery, explicit adult content, or invasive comments that can cause psychological distress. Evaluating for toxicity is critical but complex, given the subjective and contextual nature of what qualifies as toxic across different cultures, languages, and communities. Solely relying on toxicity detection APIs has limitations, as these tools can exhibit biases, such as over-flagging identity terms or under-detecting coded expressions. Bias in generative AI manifests when models disproportionately represent or marginalize certain demographic groups, ideologies, or perspectives. This can occur through skewed object representations, imbalanced occupational and location biases, or the consistent depiction of harmful stereotypes. Bias often originates from training data that over-represents dominant groups and perspectives. Homogenization occurs when generative models produce mainstream, centrist outputs that conform to dominant cultural norms, failing to capture diverse viewpoints. Outlier perspectives from minority groups may be systematically underrepresented or distorted. This issue is significant even within single countries, where cultural diversity is often inadequately represented. As NIST also identifies, homogenization of viewpoints and performances is a looming threat via model collapse due to the rise of training models on synthetic training data.\n\nTo mitigate the risks associated with toxicity, bias, and homogenization in generative AI systems, a multi-pronged approach throughout the AI lifecycle is essential, combining better data practices, continuous model evaluations, and ongoing oversight and accountability (GV-2.1-004, GV-3.2-007, GV-3.2-008, GV-4.1-005, GV-4.2-001, GV-5.1-004, GV-6.2-014, MP-1.1-004). We support challenging assumptions at the design stage and examine the entire context in which a model is situated in a sociotechnical system before pursuing technical methods to counter bias. Implementing proactive data collection and curation practices increases the representation of underrepresented cultures, ideological diversity, and more inclusive perspectives during dataset creation while applying data filters to remove egregiously toxic data. Holistically and continually evaluating generations through participatory processes that engage impacted communities, examining outputs across a range of cultural contexts, languages, and sensitive topics using both automated and human evaluations while being respectful of the potential emotional harms of\n\n\n\nannotating such content, and disaggregating evaluation results by subpopulations can ensure comprehensive analysis. Establishing transparency through detailed documentation of training data sources and any processing applied and conducting regular third-party audits to test for biases, toxicity, and cultural normativity are all steps towards a robust oversight framework. While challenging, proactively mitigating risks around toxicity, bias, and homogenization from the data and modeling phases itself is crucial for developing responsible generative AI systems aligned with societal values. Solely relying on detection, filtering and debiasing techniques has limitations.\n\nHugging Face supports numerous leaderboards and benchmarks to evaluate these aspects and emphasizes community building. A participatory approach is essential because understanding the full scope of potential issues requires diverse perspectives. Hugging Face's commitment to open science and open models allows communities to adapt models to hyperlocal use cases, countering the risks of homogenization. Projects like BLOOM and Aya, which exemplify shared multilingual efforts and a global network of researchers with a shared goal, serve as blueprints for creating inclusive AI systems in the open. Collaborative efforts, such as the CIVICS dataset initiative at Hugging Face, highlight the importance of incorporating diverse voices and perspectives throughout the AI development process. By fostering open science and community engagement, Hugging Face aims to develop generative AI that respects and reflects the richness of global diversity.\n\n## 12. Value Chain and Component Integration\n\nWe agree that third-party components in a system can introduce risks if not vetted properly, and we applaud NIST's systemic considerations. As model developers are the best parties to document models, we support developers of third-party components investing in relevant documentation. The methods by which third-party components are procured and applied should be better documented. While approaches such as leaderboards can help compare models, more research is needed to provide trusted mechanisms for analyzing and evaluating components such as benchmark datasets. Documentation should include the developer parties and methodologies for procurement and process documentation (GV-1.5-002). NIST can also provide guidance on the appropriate methods for determining component reliability and tools for comparing components.\n\nWe thank NIST for work on this profile and look forward to continuing to provide support.\n\nSubmitted by: Avijit Ghosh, Applied Policy Researcher, Hugging Face Yacine Jernite, ML and Society Lead, Hugging Face Irene Solaiman, Head of Global Policy, Hugging Face", + "chunks": [ + { + "headings": [ + "Hugging Face Comments on NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile" + ], + "markdown": "\n\n## Hugging Face Comments on NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile\n\nHugging Face commends the National Institute of Standards and Technology (NIST) on the AI Risk Management Framework (RMF): Generative AI (GAI) Profile, an extensive document identifying categories of risk and action items for actors pertaining to those risks. We offer recommendations to strengthen this document based on our experiences toward democratizing good AI and characterizing risks of systems as an open platform for state-of-the-art (SotA) AI systems. Comments are organized by risk categories with corresponding action items on them. If a section or action is not highlighted, we do not have specific, actionable feedback.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## Executive Summary\n\nHugging Face appreciates this opportunity to provide comments and recommendations to strengthen the NIST AI Risk Management Framework for Generative AI. Our emphasis is on responsible AI development practices, multi-stakeholder collaboration, technical safeguards, and ongoing monitoring across key risk areas. Key focus areas include data provenance, transparency, environmental sustainability, information integrity, security, and mitigating biases and harms. We offer consolidated action recommendations below:\n\n\n\n## Holistic Approach: Safety, Privacy, Consent, Sustainability by Design\n\n- \u25cf Adopt a \"safety by design\" approach, focusing on data provenance, quality, training data curation, and evaluations from early stages.\n- \u25cf Implement data minimization practices and robust consent mechanisms (opt-in/opt-out) for data collection and usage.\n- \u25cf Conduct continuous impact assessments and ensure transparency around data sources, licenses, and processing applied.\n- \u25cf Prioritize environmental impact measurement across the AI lifecycle, including carbon footprint calculations and energy efficiency ratings.\n\n## Community Feedback: Diverse Stakeholders\n\n- \u25cf Foster open science, community engagement, and participation from diverse stakeholders, including civil society groups and impacted communities.\n- \u25cf Encourage public benchmarking efforts, leaderboards, and scrutiny regarding the absence of comprehensive evaluations for new models.\n- \u25cf Leverage community feedback loops, external audits, and inclusive processes to assess potential biases, toxicity, and viewpoint homogenization.\n\n## Secure Disclosure and Governance\n\n- \u25cf Implement structured harm reporting mechanisms and secure disclosure processes for AI incidents and vulnerabilities.\n- \u25cf Mandate that model developers clearly specify the intended scope and capabilities of their models in the documentation.\n- \u25cf Treat any deviations from the specified scope or capabilities as flaws to be transparently reported and addressed.\n- \u25cf Establish standards and guidelines for documenting and disclosing model scope, intent, and detected flaws.\n\nHugging Face remains committed to contributing to the development of a robust AI Risk Management Framework that upholds safety, ethics, and community-driven innovation.\n\n## GAI Risks and Actions\n\n## 1. CBRN Information\n\nWe agree with researchers who have investigated this issue empirically [OpenAI study, RAND study] in that, despite GAI systems' being able to generate novel molecule information and make complex information retrieval easier, the realistic threats of such\n\n\n\nattacks materializing with current technology are low, since actors are additionally constrained by material availability and wet lab expertise. We commend the outlined actions to handle CBRN Information risks, specifically on research, monitoring, and data analysis. Investment in R&D to conduct more empirical research on realistic threat scenarios (MS-1.1-012), establish processes to regularly monitor model use cases to catch potential unforeseen use cases (GV-4.2-010), and continue to scan both training data and model outputs to limit such information (MG-3.1-007) are critical.\n\n## 2. Confabulation\n\nRisks of harm related to confabulation are exacerbated by how misleading content is distributed and received, especially in high-risk areas. Recent examples include GAI aided web search, trending topics on social media, and medical use cases. We outline some additional risks: GAI fueled misinformation on the internet can also cause threats to election integrity and democratic processes, specifically eroding trust in information and democratic institutions, and obstructing voting procedures and infrastructures. Additionally, since GAI models are trained on large amounts of data scraped from the internet, confabulated data being released into the internet can lead to future models being trained on this false information, continuing a cycle of confabulation and potentially irrevocably harming information integrity on the internet. Confabulation related risks are not only limited to large scale information integrity but also at a smaller scale when real or made up information is incorrectly attributed to people, which can have real-life consequences.\n\nAddressing confabulation behaviors in generative AI systems requires a multi-pronged approach combining training data transparency, disclosure of sources, and evaluation of a model's propensity to confabulate. Firstly, It is essential for model providers to regularly and openly evaluate models for their propensity to generate incorrect information. Two hallucinations leaderboards on Hugging Face (Hallucinations Leaderboard and HHEM by Vectara Leaderboard), assess select models for their accuracy and reliability. These evaluations help in identifying and mitigating the risks associated with AI-generated content. Secondly, developing tooling for content integrity is crucial in maintaining the trustworthiness of AI systems. Tools that enhance content integrity ensure that generated outputs are reliable, thereby reducing the chances of disseminating false information. Finally, it is important to hold users accountable for the content they create and disseminate using AI models. Terms of services are a tool for accountability ensuring user accounts are responsible for the content produced. Any content downloaded, accessed, or used on the platform is subject to these terms and/or the terms accompanying such content. This accountability is aimed at ensuring that users are mindful of the content they generate and share, fostering a responsible AI usage environment.\n\n\n\nWe agree with the set of actions laid out in the document, specifically reproducibility (MP-4.1-012), via public benchmarking and leaderboard efforts. We strongly support data provenance action items (MS-2.5-008) efforts, specifically documenting data sources and licenses that can aid in attribution and counter hallucinations. We strongly support both documenting model flaws in a structured approach (MG-4.3-003) and public sharing of such flaws to aid transparency (MG-4.3-004). An action item that we want to highlight for more context is (MS-2.13-001) - domain expertise should not only be used for subjective cases like toxicity detection but also high stakes objective use cases where confabulation might become a threat very fast, such as fact-checking during elections.\n\n## 3. Dangerous or Violent Recommendations\n\nAddressing dangerous and violent outputs in GAI requires holistic and sociotechnical approaches centering impacted communities, rigorous empirical analysis of dataset and model behaviors, and a fundamental rethinking of what constitutes \"safety\" from diverse perspectives.\n\nIt is critically important to adopt a \"safety by design\" approach when developing AI systems, rather than solely relying on techniques to detect and block potentially dangerous or violent outputs after the fact. While we support community efforts such as datasets and models created for safety checks we must be cautious about over-relying on them as a complete solution. Red teaming initiatives, such as the Red Teaming Resistance Leaderboard, which specifically aims to measure harm and violence, criminal conduct, unsolicited counsel, and not safe for work (NSFW) prediction, can provide useful signals. Red teaming should ideally be used in conjunction with algorithmic impact assessments, external audits, and public consultation, centering the voices of those most affected by potential harm.\n\nCommunity-driven approaches that amplify historically marginalized perspectives are essential to safety by design - studies such as examining islamophobic biases in GPT-3 and third-party audits on the sexualization of Asian women in image generation models illustrate the need for thorough and inclusive evaluations. Additionally, we must critically examine the datasets and processes used to create and fine-tune AI models. Unsafe pre-training data can influence models to exhibit racist, sexist, and other concerning behaviors that \"scale\" with model size. Documentation of dataset creation and carefully examining variables like data age, domain coverage, and other quality metrics at the pretraining stage is a more holistic approach to ensure safety by design. Moreover, we cannot ignore the human costs of annotating and labeling potentially violent content, which has been shown to inflict psychological trauma on data workers. Nor can we neglect the\n\n\n\npotential diversity trade-offs - overzealous filtering for \"unsafe\" content has been shown to hamper the diversity of outputs in both language and image generation models.\n\nWe largely align with outlined actions, and specifically want to highlight (GV-4.2-001) which lays out plans for third-party assessments (MS-2.7-016) which lays out guidelines for conducting red teaming, (MP-5.1-006) which puts the focus on expert assessments and human feedback; infrastructure for dialogue, such as Hugging Face community discussions pages,", + "char_slice": [ + 0, + 10464 + ] + }, + { + "headings": [ + "## Hugging Face Comments on NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile", + "## About Hugging Face", + "## Executive Summary", + "## Holistic Approach: Safety, Privacy, Consent, Sustainability by Design", + "## Community Feedback: Diverse Stakeholders", + "## Secure Disclosure and Governance", + "## GAI Risks and Actions", + "## 1. CBRN Information", + "## 2. Confabulation", + "## 3. Dangerous or Violent Recommendations" + ], + "markdown": "need for thorough and inclusive evaluations. Additionally, we must critically examine the datasets and processes used to create and fine-tune AI models. Unsafe pre-training data can influence models to exhibit racist, sexist, and other concerning behaviors that \"scale\" with model size. Documentation of dataset creation and carefully examining variables like data age, domain coverage, and other quality metrics at the pretraining stage is a more holistic approach to ensure safety by design. Moreover, we cannot ignore the human costs of annotating and labeling potentially violent content, which has been shown to inflict psychological trauma on data workers. Nor can we neglect the\n\n\n\npotential diversity trade-offs - overzealous filtering for \"unsafe\" content has been shown to hamper the diversity of outputs in both language and image generation models.\n\nWe largely align with outlined actions, and specifically want to highlight (GV-4.2-001) which lays out plans for third-party assessments (MS-2.7-016) which lays out guidelines for conducting red teaming, (MP-5.1-006) which puts the focus on expert assessments and human feedback; infrastructure for dialogue, such as Hugging Face community discussions pages, foster feedback and engagement. Additionally, we suggest detailed data quality measurement at the pretraining stage and a holistic safety by design approach (instead of a filter-and-remove approach post-training) that has the potential to remove harmed parties, often minorities, from model outputs and therefore the public sphere.\n\n## 4. Data Privacy\n\nProtecting personal information and privacy in GAI systems depends largely on responsible training data practices, processing methods, and robust security measures. Similar to our recommendations for confabulation, it is essential to adopt a holistic set of best practices to ensure privacy by design - such as data minimization, opt-in data collection, dataset transparency, and accountability throughout the AI lifecycle.\n\nProviders should seek consent and respect the explicit choices of individuals for collecting, processing, and sharing data with external parties, as sensitive data could be leveraged for downstream harm such as security breaches, privacy violations, and adversarial attacks. A key risk arises in third-party hosted systems, where deployed language models can leak private user data like PII or sensitive records inadvertently embedded within training sets, often through proprietary system prompts prepended to user inputs during generation. While screening training data for PII and IP violations is feasible, attempting to check generated outputs for training data attribution is still an open research problem unless there is verbatim copying. Defining and identifying \"substantial similarity\" across modalities like text, images, and audio is difficult. Rather than focusing guidelines around attributing generated output to training data, more focus should be on implementing robust consent mechanisms and privacy safeguards from the start to prevent personal data leaks and violations during the generation process itself.\n\nEnsuring dataset transparency is vital for safeguarding privacy in generative AI systems, while also acknowledging the privacy tradeoffs involved in determining to whom data transparency reports are disclosed. Publishing details about the sources, licenses, and potential privacy risks of training data allows external auditing and accountability. Providers should document data origins and any processing applied. There are other careful\n\n\n\nconsiderations that should be encouraged at the pretraining stage for privacy, not just via more effective licensing but also carefully considering contextual integrity; generative models should ensure that individuals' data cannot be obtained from contexts in which they do not expect it to appear. Some classical notions of privacy protection, like data sanitization, encryption, anonymization, and differential privacy, can be difficult to translate to the generative paradigm.\n\nAn explicit recommendation that we have for this section is data minimization: only the minimum necessary data should be collected, used, and retained for the specified purposes of developing and operating the AI system. This reduces the potential for misuse, unauthorized access, or unintended exposure of personal information. Encouraging data minimization practices is especially important in light of data protection regulations like the EU's General Data Protection Regulation (GDPR) which enshrine data minimization as a key requirement. Recent research has also highlighted the risks of excess data retention, showing the impact of repetition in pretraining data on memorization and content leaks in language models. Data minimization not only reduces privacy risks but can also provide computational efficiency gains by reducing dataset sizes. However, implementation requires careful consideration of the right data practices for each use case to balance privacy with maintaining model performance and generalization ability.\n\nFinally, when it comes to accountability, providers should implement robust, public opt-out mechanisms, or better yet, switch to opt-in mechanisms that allow individuals to restrict the use of their personal data for training generative AI models. Vendors implementing opt-out mechanisms should not resort to dark patterns and provide maximum information to users so that they can provide informed consent. Successful implementations like the policies used by BigCode showcase elements of an effective opt-out program including maintaining public opt-out registries where individuals can submit requests to have their data removed from training sets, providing tooling for data creators/owners to automatically remove or exclude their data from being included, committing to not training future model versions on any opted-out data, and offering redacted search utilities for individuals to check if their information is present while preserving privacy. Implementing a universal opt-out policy is challenging given the diversity of data sources and integrations required for large training sets; however, at a minimum, these practices should be encouraged to uphold privacy standards. These tools should be provided in conjunction with regular audits of model capabilities to test for potential privacy leakage and memorization.\n\nIn terms of actions, we highlight establishing transparency policies (GV-1.2-007), continuous privacy impact assessments (GV-4.2-001) - that can be done either by the vendor or in the open via community feedback via tools such as leaderboards along with collaboration with privacy and digital rights organizations, and consent mechanisms\n\n\n\n(MS-2.10-006) as effective tools, which can be further specified with our recommendations for a privacy by design approach over a detect and block approach. Additionally, our strong recommendation for data minimization broadly connects to MP-4.1-009, but we recommend either highlighting this point or writing out a separate explicit action for data minimization as a policy.\n\n## 5. Environmental Risk\n\nAssessing and mitigating the environmental impact of generative AI (GAI) systems is crucial from both a sustainability and ethical standpoint and needs to be done via standardizing metrics, fostering transparency from both model developers and compute providers, and collaborating with diverse stakeholders .\n\nThe development, deployment, and ongoing operation of GAI systems can have significant energy demands and greenhouse gas emissions that need to be carefully measured and managed. While we know training and running large models consume a lot of energy, the discussion of environmental risk currently focuses primarily on the training phase [OpenAI, Patterson et al, Anthony et al], but it is essential to broaden the scope and comprehensively highlight the energy impact of deploying and fine-tuning GAI models.\n\nCurrently, there is a distinct lack of transparency around how much energy is consumed by models, nor is there enough incentive by model developers to perform these measurements. Hugging Face is working on combating this by pioneering an \"Energy Star\" rating system for AI models, similar to efficiency ratings for appliances and electronics. Such a standardized rating would quantify the energy consumption and environmental impact of training, fine-tuning, and running inference with different models. This would empower more informed decision-making by AI developers, providers, and users in selecting energy-efficient models aligned with their sustainability goals.\n\nIncorporating environmental impact metrics into widely adopted model documentation standards, like model cards, could further incentivize sustainable practices. Ongoing engagement with impacted communities, civil society groups, and compute providers is crucial to shaping a holistic understanding of environmental impacts beyond carbon emissions. Other factors like water and natural resource usage for both chip manufacturing and data center operations should be considered. A key limitation in the implementation of a robust set of actions in combating the environmental risks of GAI is the lack of standardized measurement variables and uncertainty around relative contributions from different stages (pretraining, fine-tuning, inference) and architectural choices. As a standards body, NIST could play a vital role in establishing measurement guidelines tailored to different AI modalities, which in turn would help public measurement and documentation\n\n\n\nefforts. In terms of uncertainty, we strongly advocate for greater transparency from hardware manufacturers and data centers for accurate estimation.\n\nWhile NIST's recommended actions (MS-2.11-005, MS-2.12-001, MS-2.12-002, MS-2.12-003, MS-2.12-004, MS-2.12-005) broadly align with the need to technically measure environmental impacts across the AI lifecycle, we additionally want to highlight the need to involve grassroots efforts, impacted communities, and climate justice coalitions to complement these efforts. Centralized energy and carbon footprint information via efforts such as the Energy Star project would prevent duplicated measurement efforts.\n\n## 6. Human-AI Configuration\n\nConcerning risks include emotional entanglement, deceptive capabilities, automation bias, and aversion to AI outputs.\n\nEmotional entanglement occurs when users form emotional bonds with AI systems, which can be used for engagement, monetization, or manipulation purposes. AI systems designed to act in human-like ways can exacerbate this issue, leading to technological addiction, overreliance, and even impairment of value-aided judgment in high-risk areas. For example, Google DeepMind's technical paper on AI assistants warns about the ethical implications of emotionally expressive chatbots, highlighting", + "char_slice": [ + 9230, + 20202 + ] + }, + { + "headings": [ + "## Hugging Face Comments on NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile", + "## About Hugging Face", + "## Executive Summary", + "## Holistic Approach: Safety, Privacy, Consent, Sustainability by Design", + "## Community Feedback: Diverse Stakeholders", + "## Secure Disclosure and Governance", + "## GAI Risks and Actions", + "## 1. CBRN Information", + "## 2. Confabulation", + "## 3. Dangerous or Violent Recommendations", + "## 4. Data Privacy", + "## 5. Environmental Risk", + "## 6. Human-AI Configuration" + ], + "markdown": "manufacturers and data centers for accurate estimation.\n\nWhile NIST's recommended actions (MS-2.11-005, MS-2.12-001, MS-2.12-002, MS-2.12-003, MS-2.12-004, MS-2.12-005) broadly align with the need to technically measure environmental impacts across the AI lifecycle, we additionally want to highlight the need to involve grassroots efforts, impacted communities, and climate justice coalitions to complement these efforts. Centralized energy and carbon footprint information via efforts such as the Energy Star project would prevent duplicated measurement efforts.\n\n## 6. Human-AI Configuration\n\nConcerning risks include emotional entanglement, deceptive capabilities, automation bias, and aversion to AI outputs.\n\nEmotional entanglement occurs when users form emotional bonds with AI systems, which can be used for engagement, monetization, or manipulation purposes. AI systems designed to act in human-like ways can exacerbate this issue, leading to technological addiction, overreliance, and even impairment of value-aided judgment in high-risk areas. For example, Google DeepMind's technical paper on AI assistants warns about the ethical implications of emotionally expressive chatbots, highlighting risks such as privacy invasion and new forms of technological addiction. Studies and real-world experiences further illustrate the potential for users to develop emotional connections with AI, underscoring the need for public education and awareness about these risks.\n\nNIST's recommended actions emphasize the need for disclosing AI involvement in outputs and establishing organizational roles, policies, and procedures for communicating GAI system incidents and performance (GV-1.5-004, GV-2.1, GV-5.1-002). There are several ways in which this can be implemented - we advocate for transparency around models as a system, via greater transparency around the outcomes of different initial system prompts, and we strongly emphasize the importance of documentation such as model cards, which should clearly specify scope and intent of the model. A deviation of model outputs from its clearly defined scope or intent should be considered a flaw in the system and should be transparently reported via public avenues such as the AI Incident Database, AVID, AI Litigation Database, CVE, OECD Incident Monitor, or others.\n\n## 7. Information Integrity\n\nGAI models can produce realistic content in different modalities often faster than humans can, which can exacerbate threats such as mis- and dis-information and their associated\n\n\n\nthreats such as false advertisements, election integrity, impersonation, and other harms. Threat level, believability, and ability to mitigate risk will differ by modality. In this section, we focus specifically on the harms of intentionally generated content that threaten information integrity. A comprehensive approach to ensuring information integrity should combine technical approaches, governance mechanisms, and collaborative public education.\n\nTechnical content verification using watermarking has emerged as a mechanism for ensuring content integrity and authenticity in the face of rapidly advancing generative AI capabilities but can be unreliable and not yet tamperproof. Watermarking provides tools to embed metadata about whether a particular piece of content was generated by a GAI model. There are primarily two approaches to watermarking AI-generated content: embedding watermarks during content creation or applying them post-content production. The former, requiring access to the model, offers robust protection as it is automatically integrated into the generation process. On the other hand, post-production watermarking, while applicable to closed-source models, may not suit all content types, such as text. While a lot of proposed watermarking methods are model specific, model-agnostic universal watermarking is an active area of research.\n\nAmong the Hugging Face-provided tooling and hosted collaborative projects for watermarking text, image, and audio modalities are image-specific techniques that complement watermarking to limit non-consensual image manipulation. Some subtly alter images to confound AI algorithms, hindering their ability to process them accurately. Tools like Glaze and Photoguard achieve this by making images imperceptibly different to humans but challenging for AI to interpret. Others, like Nightshade and Fawkes, \"poison\" images to disrupt AI training assumptions, preventing the generation of fake images. Additionally, signing techniques, exemplified by Truepic's C2PA-standard metadata embedding, link content to its provenance and provide certification for metadata validity and integrate with watermarking to bolster information resilience.\n\nNIST can guide standards for informing or restricting AI usage in areas where trustworthiness and quality are essential (MAP 2.1). For instance, leading AI and computing academic bodies such as ICML, ACM, AAAI, and IEEE all have clear authorship policies that advocate for transparency in disclosing AI-edited content and restrict how they can be used. These guidelines should extend to content providers and media outlets, requiring clear labeling of AI-generated content to prevent the spread of misinformation - a study on the current state of internal GAI policies in newsrooms around the world showed varying levels of scope and enforcement in AI usage for reporting and writing purposes. We support documenting processes such as via governance cards, where model developers can disclose their plans for model use cases and list tools for attribution or detection of\n\n\n\n## content generated by their model.\n\nFinally, end users can take action to protect themselves from AI-generated mis- and disinformation. NIST can establish common areas for education and literacy (GV-6.1-003) . Public education initiatives can empower individuals to critically evaluate the content they encounter online. This can be done in person at the community level via local libraries and classes in schools - for example, the Boston Public Library is offering free classes on detecting online misinformation, and several states have implemented mandatory media literacy classes in their education systems. These educational efforts should focus on teaching people how to identify common signs of misinformation, understand the importance of source credibility, and use fact-checking tools effectively. Hugging Face strongly supports these efforts, both via providing free AI education resources that instructors can use in media literacy classes, and via providing online community spaces for journalists to use and audit AI models in more effective and informed journalism. By raising public awareness and fostering critical thinking, misinformation education can play a vital role in protecting information integrity and promoting a more informed society.\n\n## 8. Information Security\n\nWhile GAI models offer powerful capabilities, they also introduce new risks and challenges that must be addressed to maintain the integrity and safety of digital systems.\n\nThe use of open models in secure systems that do not finetune on user data is a powerful way to protect user data. SafeCoder, developed by ServiceNow and Hugging Face, has been rigorously tested for risks like memorization of proprietary code, can run in a secure, air-gapped manner without internet access, and is fully private by design, avoiding training on user data. In contrast, models that collect user data have faced sophisticated cyberattacks, threatening end users. For instance, Samsung experienced a data leak when proprietary information sent to ChatGPT was memorized and later exposed. Research shows that models like GPT-Neox memorized about 1% of their training set, and similar findings with StarCoder demonstrated its ability to closely reconstruct training samples. This highlights the inherent memorization capabilities of LLMs, posing privacy issues. Models like SafeCoder and BlindChat are set apart due to their design, ensuring privacy and preventing such risks - as has also been verified by external audits. This underscores the importance of transparency and accountability in AI development, as allowing for community auditing and contributions enables identifying and fixing vulnerabilities (MS-1.1-002, GV-3.2-002).\n\nAs a platform that hosts models, we are mindful of the distribution of malicious code via packaged models (MG-3.1-002). Traditional serialization methods, such as pickle used by\n\n\n\nlibraries like PyTorch and TensorFlow, pose risks by allowing arbitrary code execution. To mitigate this, Hugging Face introduced a security scanner that employs ClamAV, an open-source antivirus, to detect malware. Pickle import scanning raises warnings about suspicious imports that could lead to arbitrary code execution. We developed safetensors, a secure and efficient format that eliminates the risks associated with pickle. This effort, supported by a collaborative security audit, ensures a safer default format for model storage across various libraries, including transformers. Safe formats like GGUF promote the broader adoption of secure practices. Social validation features also enhance trust in model usage. Similar to platforms like GitHub and npm, users rely on models from reputable sources with positive community engagement. Social features, reporting mechanisms, and spam detection tools help users identify and avoid potentially harmful models. Combining safe file formats with trusted sources fosters a secure and trustworthy environment for AI model deployment.\n\nIt is important to recognize that AI models do not exist in isolation but are integrated into broader software ecosystems. As such, security considerations should be approached holistically via structured, coordinated, and open harm reporting (MS-2.2-010), drawing insights from established practices and frameworks and tailoring them to GAI threats, such as those used in the management of Common Vulnerabilities and Exposures (CVEs) by the MITRE Corporation. NIST's AI Safety Institute (AISI) could be the counterpart for MITRE for such a coordinated AI flaw reporting initiative. Comprehensive security strategies should encompass a range of actions, including data security and privacy controls, organizational security policies and service-level agreements (SLAs), third-", + "char_slice": [ + 18998, + 29409 + ] + }, + { + "headings": [ + "## Hugging Face Comments on NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile", + "## About Hugging Face", + "## Executive Summary", + "## Holistic Approach: Safety, Privacy, Consent, Sustainability by Design", + "## Community Feedback: Diverse Stakeholders", + "## Secure Disclosure and Governance", + "## GAI Risks and Actions", + "## 1. CBRN Information", + "## 2. Confabulation", + "## 3. Dangerous or Violent Recommendations", + "## 4. Data Privacy", + "## 5. Environmental Risk", + "## 6. Human-AI Configuration", + "## 7. Information Integrity", + "## content generated by their model.", + "## 8. Information Security" + ], + "markdown": "ensures a safer default format for model storage across various libraries, including transformers. Safe formats like GGUF promote the broader adoption of secure practices. Social validation features also enhance trust in model usage. Similar to platforms like GitHub and npm, users rely on models from reputable sources with positive community engagement. Social features, reporting mechanisms, and spam detection tools help users identify and avoid potentially harmful models. Combining safe file formats with trusted sources fosters a secure and trustworthy environment for AI model deployment.\n\nIt is important to recognize that AI models do not exist in isolation but are integrated into broader software ecosystems. As such, security considerations should be approached holistically via structured, coordinated, and open harm reporting (MS-2.2-010), drawing insights from established practices and frameworks and tailoring them to GAI threats, such as those used in the management of Common Vulnerabilities and Exposures (CVEs) by the MITRE Corporation. NIST's AI Safety Institute (AISI) could be the counterpart for MITRE for such a coordinated AI flaw reporting initiative. Comprehensive security strategies should encompass a range of actions, including data security and privacy controls, organizational security policies and service-level agreements (SLAs), third-party impact assessments, incident response planning, vendor provenance and contract management, and encryption and secure communication protocols. Red teaming efforts, where ethical hackers simulate real-world attacks to identify vulnerabilities, can be particularly valuable in the context of generative AI models. Events such as DEFCON's engagement with government agencies in conducting such exercises, leverage the collective expertise of its community. This community-driven approach can accelerate the development of effective security solutions tailored to the unique challenges posed by generative AI models, as evidenced by Hugging Face's leaderboarding tools and community efforts in this area (AI Secure LLM Leaderboard, Red Teaming Resistance Leaderboard ).\n\n## 9. Intellectual Property\n\nBalancing the copyright implications of large web-scraped training data with the societal benefits of generative AI requires transparency, rights-holder controls, and responsible deployment practices.\n\n\n\nOne of the major concerns around generative AI systems is the potential for intellectual property (IP) violations, particularly copyright infringement. GAI systems are trained on massive datasets that contain copyrighted works like books, academic papers, computer code, and more. The training data is often collected from publicly available sources on the internet through techniques like web scraping and crawling [The Pile, The Stack, ROOTS, DoLMA, and LAION]. This raises questions about whether the unauthorized use of such copyrighted material for training AI models constitutes fair use or infringement. As we shared in our response to the U.S. PTO, the early stages of dataset curation and model pre-training should generally align with fair use principles, as the aim is to encode general statistics and patterns across the training data, rather than reproducing or replacing specific copyrighted works. However, certain scenarios, like training on a narrow set of works specifically to create market substitutes, could potentially violate fair use.\n\nA key challenge is defining and identifying \"substantial similarity\" across different data modalities like text, images, audio, and video. While verbatim copying from training data can potentially be detected, assessing whether a generated output is too similar to a copyrighted work is an open technical problem without clear objective thresholds. This problem is made worse by the lack of transparency about the composition of training datasets, which makes it difficult for copyright holders to assess potential infringement and negotiate terms of use. We recognize that ensuring that fair use remains consistent can put an undue burden on copyright holders, and decrease their ability to advocate for their broad interests. However, a broad licensing requirement for training data could lead to market concentration, excluding smaller actors while providing negligible financial benefits to the original creators.\n\nWe support NIST's recommended actions in this area about focusing on transparency requirements and dataset provenance (GV-1.2-007) and applying existing laws (GV-1.1-001) regarding human authorship for determining copyright protection of AI-generated outputs, rather than granting automatic rights to model developers or users. These actions would align with international regulation - categories of stakeholder and other jurisdictions have turned to transparency (EU AI Act) and opt-out (EU CDSM) requirements.\n\nIn terms of specific, actionable comments on transparency and dataset provenance, our recommendations are similar to what we suggest in Data Privacy, including implementing robust opt-out or opt-in mechanisms via transparent data governance and indexing tools that allow rights holders to restrict the use of their copyrighted works, data minimization as a model design principle, and safer deployment practices like watermarking, truncating outputs, and filtering for verbatim copyrighted material in model outputs.\n\n\n\n## 10. Obscene, Degrading, and/or Abusive Content\n\nIn general, GAI systems built solely for the purpose of creating nonconsensual content should be banned (GV-1.1-005, MP-4.1-007). Addressing the risks of generative AI systems producing obscene or degrading content requires a proactive \"safety by design\" approach that considers the impact of all development choices from the earliest stages. This begins with the careful curation of the pre-training dataset (Longpre et al., Dolma), as there are risks to the \"dark side of scaling\" language models on massive, uncurated internet data. Ensuring the training data is free from harmful or biased content is crucial (MS-2.6-002), as relying solely on interventions during fine-tuning or deployment stages (MG-2.2-005, MG-3.2-005) -such as content filters or sensitivity adjustments-has proven insufficient. Instances of AI generating deepfakes and other harmful content despite these measures underscore this inadequacy. Therefore, a comprehensive safety strategy must be embedded throughout the entire model development lifecycle, from data collection and architecture design to setting clear training objectives and rigorous evaluation metrics.\n\nFor the specific case of risks related to child sexual abuse material (CSAM), the Safety by Design guide developed by Thorn and All Tech is Human provides a valuable and actionable framework. This guide emphasizes the importance of transparency and thorough documentation, ensuring that interventions are based on verified harms and carefully considering potential unintended consequences on privacy, diversity, and biases. Adopting this approach helps mitigate CSAM risks while minimizing negative impacts on marginalized communities or legitimate use cases. The guide strongly recommends involving external stakeholders, particularly those with sociological expertise, in developing and evaluating risk mitigation strategies.\n\nThe Safety by Design approach can be adapted to address other forms of objectionable or damaging content beyond CSAM, such as hate speech, misinformation, or explicit violence. As with previous technological transitions like social media, the moderation practices of generative AI systems will likely come under increasing scrutiny from regulations like the European Digital Services Act. This necessitates a detailed, transparent approach to risk mitigation that documents the trade-offs and decisions made. Given the complex social determinations and trade-offs between different notions of safety and inclusiveness, risk mitigation approaches must be developed in conjunction with a diverse range of external stakeholders. This collaborative process should involve sociologists, ethicists, policymakers, and representatives from potentially impacted communities. Such inclusive engagement ensures that interventions are grounded in a comprehensive understanding of societal values and potential consequences, leading to more balanced and effective safety measures.\n\n\n\n## 11. Toxicity, Bias, and Homogenization\n\nGenerative AI systems, while powerful, can produce toxic, biased, or homogenized content that propagates harmful stereotypes, hate speech, or ideological viewpoints. This presents significant risks, especially as large language models can generate human-sounding text or realistic content in other modalities like images or audio on virtually any topic at scale. Deploying these models globally is challenging due to distinct cultural values and norms around what constitutes sensitive or offensive content.\n\nToxicity refers to harmful content like hate speech, violent imagery, explicit adult content, or invasive comments that can cause psychological distress. Evaluating for toxicity is critical but complex, given the subjective and contextual nature of what qualifies as toxic across different cultures, languages, and communities. Solely relying on toxicity detection APIs has limitations, as these tools can exhibit biases, such as over-flagging identity terms or under-detecting coded expressions. Bias in generative AI manifests when models disproportionately represent or marginalize certain demographic groups, ideologies, or perspectives. This can occur through skewed object representations, imbalanced occupational and location biases, or the consistent depiction of harmful stereotypes. Bias often originates from training data that over-represents dominant groups and perspectives. Homogenization occurs when generative models produce mainstream, centrist outputs that conform to dominant cultural norms, failing to capture diverse viewpoints. Outlier perspectives from minority groups may be systematically underrepresented or distorted. This issue is significant even within single countries, where cultural diversity is often inadequately represented. As NIST also identifies, homogenization of viewpoints and performances is a looming threat via model collapse due to the rise of training models on synthetic training data.\n\nTo mitigate the risks associated with toxicity, bias, and homogenization in generative AI systems, a multi-pronged approach throughout the AI lifecycle is essential, combining better data practices, continuous model evaluations, and ongoing oversight and accountability (GV-2.1-004, GV-3.2-007, GV-3.2-008, GV-4.1-005, GV-4.2-", + "char_slice": [ + 28035, + 38778 + ] + }, + { + "headings": [ + "## Hugging Face Comments on NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile", + "## About Hugging Face", + "## Executive Summary", + "## Holistic Approach: Safety, Privacy, Consent, Sustainability by Design", + "## Community Feedback: Diverse Stakeholders", + "## Secure Disclosure and Governance", + "## GAI Risks and Actions", + "## 1. CBRN Information", + "## 2. Confabulation", + "## 3. Dangerous or Violent Recommendations", + "## 4. Data Privacy", + "## 5. Environmental Risk", + "## 6. Human-AI Configuration", + "## 7. Information Integrity", + "## content generated by their model.", + "## 8. Information Security", + "## 9. Intellectual Property", + "## 10. Obscene, Degrading, and/or Abusive Content", + "## 11. Toxicity, Bias, and Homogenization" + ], + "markdown": "models disproportionately represent or marginalize certain demographic groups, ideologies, or perspectives. This can occur through skewed object representations, imbalanced occupational and location biases, or the consistent depiction of harmful stereotypes. Bias often originates from training data that over-represents dominant groups and perspectives. Homogenization occurs when generative models produce mainstream, centrist outputs that conform to dominant cultural norms, failing to capture diverse viewpoints. Outlier perspectives from minority groups may be systematically underrepresented or distorted. This issue is significant even within single countries, where cultural diversity is often inadequately represented. As NIST also identifies, homogenization of viewpoints and performances is a looming threat via model collapse due to the rise of training models on synthetic training data.\n\nTo mitigate the risks associated with toxicity, bias, and homogenization in generative AI systems, a multi-pronged approach throughout the AI lifecycle is essential, combining better data practices, continuous model evaluations, and ongoing oversight and accountability (GV-2.1-004, GV-3.2-007, GV-3.2-008, GV-4.1-005, GV-4.2-001, GV-5.1-004, GV-6.2-014, MP-1.1-004). We support challenging assumptions at the design stage and examine the entire context in which a model is situated in a sociotechnical system before pursuing technical methods to counter bias. Implementing proactive data collection and curation practices increases the representation of underrepresented cultures, ideological diversity, and more inclusive perspectives during dataset creation while applying data filters to remove egregiously toxic data. Holistically and continually evaluating generations through participatory processes that engage impacted communities, examining outputs across a range of cultural contexts, languages, and sensitive topics using both automated and human evaluations while being respectful of the potential emotional harms of\n\n\n\nannotating such content, and disaggregating evaluation results by subpopulations can ensure comprehensive analysis. Establishing transparency through detailed documentation of training data sources and any processing applied and conducting regular third-party audits to test for biases, toxicity, and cultural normativity are all steps towards a robust oversight framework. While challenging, proactively mitigating risks around toxicity, bias, and homogenization from the data and modeling phases itself is crucial for developing responsible generative AI systems aligned with societal values. Solely relying on detection, filtering and debiasing techniques has limitations.\n\nHugging Face supports numerous leaderboards and benchmarks to evaluate these aspects and emphasizes community building. A participatory approach is essential because understanding the full scope of potential issues requires diverse perspectives. Hugging Face's commitment to open science and open models allows communities to adapt models to hyperlocal use cases, countering the risks of homogenization. Projects like BLOOM and Aya, which exemplify shared multilingual efforts and a global network of researchers with a shared goal, serve as blueprints for creating inclusive AI systems in the open. Collaborative efforts, such as the CIVICS dataset initiative at Hugging Face, highlight the importance of incorporating diverse voices and perspectives throughout the AI development process. By fostering open science and community engagement, Hugging Face aims to develop generative AI that respects and reflects the richness of global diversity.\n\n## 12. Value Chain and Component Integration\n\nWe agree that third-party components in a system can introduce risks if not vetted properly, and we applaud NIST's systemic considerations. As model developers are the best parties to document models, we support developers of third-party components investing in relevant documentation. The methods by which third-party components are procured and applied should be better documented. While approaches such as leaderboards can help compare models, more research is needed to provide trusted mechanisms for analyzing and evaluating components such as benchmark datasets. Documentation should include the developer parties and methodologies for procurement and process documentation (GV-1.5-002). NIST can also provide guidance on the appropriate methods for determining component reliability and tools for comparing components.\n\nWe thank NIST for work on this profile and look forward to continuing to provide support.\n\nSubmitted by: Avijit Ghosh, Applied Policy Researcher, Hugging Face Yacine Jernite, ML and Society Lead, Hugging Face Irene Solaiman, Head of Global Policy, Hugging Face", + "char_slice": [ + 37550, + 42357 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2024_NTIA_Response.pdf", + "md_content": "Hugging Face, Inc. 20 Jay Street, Suite 620,\n\nBrooklyn, NY 11201\n\n## Hugging Face Response to the National Telecommunications and Information Administration Request for Comments on Dual Use Foundation Artificial Intelligence Models with Widely Available Model Weights\n\nHugging Face applauds the ongoing work of the National Telecommunications and Information Administration (NTIA) in examining dual use foundation models (FMs). The following comments are informed by our experiences as an open platform for AI systems, working to make AI accessible and broadly available to researchers for responsible development.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## Questions\n\nIn order to best address the risks and benefits of widely available model weights, we explore both the dimension of wide availability of a model and access to model weights in our responses to avoid conflating risks inherent in the technology with risks associated with release methods. We refer to models with widely available model weights as 'open-weight models'.\n\n- 1. How should NTIA define 'open' or 'widely available' when thinking about foundation models and model weights?\n\nBoth 'open' and 'widely available' as terms are distinct, and require appropriate definitions when discussing risks and benefits.\n\n'Open' should consider components that shape model impact. The risks and benefits of foundation models are shaped by decision along their development and deployment chain (e.g. when considering risks of biases and discriminatory outcomes 1 ). The contribution of openness\n\n1 Let's talk about biases in machine learning! Ethics and Society Newsletter #2\n\n\n\n\n\nto supporting beneficial development of the technology, 2 as well as the governance processes it requires to help mitigate risks of the technology, will depend on access and transparency into not just the model weights but also its development datasets , development conditions, 3 4 and deployment infrastructure.\n\nOpen weights for a foundation model are notable on two main accounts. First, given how resource-intensive the data curation and training of the models has become, sharing this particular artifact in the AI value chain is necessary to enable downstream research that only requires model access from many stakeholders outside of well-resourced companies. 5 Second, the release of a model's weights has historically served as a catalyst for development projects that prioritize overall transparency 6 and proactive consideration of regulatory processes 7 that enable more comprehensive research on risks and benefits along the development chain.\n\nTo account for all those dynamics, we strongly recommend considering at least three main components in FM openness: its datasets , the trained model , and the model's training and deployment mechanisms .\n\n'Widely available' should consider breadth (not just scale) of access. In general, providing model access to a more diverse set of stakeholders can have a net positive impact on the safety of the technology; access should be available to stakeholders who are best positioned to identify risks and limitations 8 - and particularly to third-party organizations that do not have a direct relationship (or conflicts of interest) with the developer and direct beneficiaries. 9 Breadth of access can be achieved independently of scale through mechanisms such as gating, where access is granted by an (independent) organization for specific well-motivated research and development projects. 10 Simple availability is also distinct from accessibility; model weights being available may not be accessible to people without the necessary compute infrastructure or skill set for model hosting. For uses of a model that do not require specific customization, a managed API supported by extensive computational resources greatly increases the scale of availability independently of whether model weights are directly accessible.\n\n- a. Is there evidence or historical examples suggesting that weights of models similar to currently-closed AI systems will, or will not, likely become widely available? If so, what are they?\n\nPast examples include staged releases of model weights where closed weights are released over time, as seen with OpenAI's GPT-2 11 , Meta's LLaMA 12 , and Stability AI's Stable Diffusion 13 . All recent prominent closed models have been followed by an open comparable alternative.\n\n2 AI Policy @ : Open ML Considerations in the EU AI Act\n\n\n\n\ud83d\udcda\n\n6 Introducing The Foundation Model Transparency Index\n\n7 Do Foundation Model Providers Comply with the Draft EU AI Act?\n\n8 A Safe Harbor for Independent AI Evaluation\n\n9 Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance\n\n10 Access form for the BigScience training corpus\n\n11 Release Strategies and the Social Impacts of Language Models\n\n12 Introducing LLaMA: A foundational, 65-billion-parameter large language model\n\n13 Stable Diffusion Public Release - Stability AI\n\n\n\nExamples include OpenAI's GPT-3 deployed via API in 2021 and model weights for BigScience's BLOOM and Meta's OPT both released in 2022. Recently, DataBricks was reported to have trained a competitive large language model for 10M$, significantly lower than the initial development cost of closed products; 14 indicating that relying on cost or trade secrets is unlikely to limit availability of top-performing models to their current set of developers.\n\nBeing 'first to market' presents a strong economic competitive advantage for closed systems, and reproduction efforts of the model itself are motivated by a range of values, including scientific understanding, free and open source software (FOSS) ideals of accessibility, and customization, including fitness for purpose and harm mitigation strategies. 15\n\nb. Is it possible to generally estimate the timeframe between the deployment of a closed model and the deployment of an open foundation model of similar performance on relevant tasks? How do you expect that timeframe to change? Based on what variables? How do you expect those variables to change in the coming months and years?\n\nTimelines vary and estimates will change by utility of the model and costs. Factors include computational cost, training data accessibility, and potential anti-competitive practices from original developers. Research focus and more compute and data-efficient algorithms are advantages for open models, whereas product focus, data concentration, high resources, and legally untested or contested practices are advantages for close development. Overall, socially responsible research in open reproduction of closed models will necessarily take more time for release 16 as legal and ethical concerns are addressed proactively rather than retroactively after issues have been observed at sufficient scale.\n\n- c. Should 'wide availability' of model weights be defined by level of distribution? If so, at what level of distribution (e.g., 10,000 entities; 1 million entities; open publication; etc.) should model weights be presumed to be 'widely available'? If not, how should NTIA define 'wide availability?'?\n\nModel weights can be shared individually between parties, on platforms with or without documentation and with or without access management, and via p2p/torrent. Usability depends on resources and infrastructure; subsidized compute from cloud compute providers can increase access drastically, including for uses that go against the spirit of ToU but are hard to detect. APIs can broaden model availability, even to malicious actors, especially already relying on developer infrastructure. APIs also can be attacked to provide model information 17 . Even controlled risks can outweigh open models in magnitude through deployment scale. 18\n\n14 Inside the Creation of DBRX, the World's Most Powerful Open Source AI Model | WIRED\n\n15 Policy Readout - Columbia Convening on Openness and AI\n\n16 BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model\n\n17 Stealing Part of a Production Language Model\n\n18 Google's Bard AI chatbot is vulnerable to use by hackers. So is ChatGPT.\n\n\n\nd. Do certain forms of access to an open foundation model (web applications, Application Programming Interfaces (API), local hosting, edge deployment) provide more or less benefit or more or less risk than others? Are these risks dependent on other details of the system or application enabling access? i. Are there promising prospective forms or modes of access that could strike a more favorable benefit-risk balance? If so, what are they?\n\nThere is no standard for weighing risks and benefits, nor are there standards or clear methods for definitive access decisions. Distributed methods for sharing via torrent are arising such as tweeting magnet links as seen with xAI's Grok 19 . These are broadly accessible but do not allow for updates in information and rely on trust and robustness of the torrent sharer.\n\nWeb applications with user interfaces lower the barrier for users with little to no AI experience to engage with the model. Functionality depends on application and interface, such as the ability to toggle temperature. This is often surface-level access, and cannot be built upon.\n\nAPI deployment can offer varying functionality, allowing only querying or allowing fine-tuning. APIs ease use for prescribed use cases as they only require internet connection and possibly a user account. APIs are often commercialized. Deployment benefits include monitoring, last-layer interventions, rate-limiting. Limitations include research and auditing inaccessibility, privacy concerns, and infrastructural reliability issues. API terms of use can be monitored and detected, and users can be blocked or revoked access. However content moderation proves difficult; blocks can easily be bypassed; false positives can block good faith researchers. 20 Last-layer interventions include watermarks after output generation. Limited API access makes research less reliable or presents additional barriers when limiting access to logits, forcing version changes, or obfuscating input and output processing that confound research findings. API deployment also introduces additional privacy challenges when user inputs are collected. APIs also entirely rely on a model hoster and their infrastructure, which may have significant demand and varying reliability.\n\nPlatform sharing of model weights enables broad access to models for local use and modification, controlled by access management (e.g. gating) and visibility (privacy, sensitive content tags) features. Content guidelines and ToU on model sharing platforms are defined at the model rather than at the use level. Releases on platforms can range from fully permissive to research-only (e.g. Cohere Command-R 21 ).\n\nLocal hosting provides full control of the model and inputs to the model, satiates any privacy and sensitive information concerns, and opens many possible research directions. However, this is less monitorable for the deployer, should the user break terms of use. Edge deployment is a type of local hosting that has environment, portability, and privacy benefits.\n\n19 Grok model release via torrent on the X platform\n\n20 A Safe Harbor for AI Evaluation and Red Teaming\n\n21 Command R\n\n\n\n- 2. How do the risks associated with making model weights widely available compare to the risks associated with non-public model weights?\n- a. What, if any, are the risks associated with widely available model weights? How do these risks change, if at all, when the training data or source code associated with fine tuning, pretraining, or deploying a model is simultaneously widely available?\n\nIn most cases, the risks associated with open-weight models are broadly similar to any other part of a software system (with or without AI components), and are similarly context-dependent. 2223 Risks should also be weighed against non-AI systems. 24 Prominent considerations include new capabilities and systems designed specifically for harm.\n\nNew AI use cases arise fast, as seen with the easy creation of photo-realistic images enabled by diffusion models, and more recently video and audio generations. Accelerated development timelines can outpace societal and technical interventions. Both closed development and large-scale release with rapid distribution contribute to risks arising and timeline considerations. Open development often provides additional time and opportunities to question choices, allowing many perspectives to shape systems earlier design decisions 25 . Conversely, closed product development followed by rapid distribution can lead to significant disruptions in important sectors for sometimes uncertain benefits. 2627 Rapid distribution depends on a combination of the availability of model weights, cost and technical difficulty of running a model, availability of a product or API version of an AI system, and broad advertisement and awareness of the AI system. Ensuring that scientific understanding of foundation models and development of new applications of the technology is more gradual helps identify issues and understand technological impacts earlier.\n\nAI systems that are designed specifically for harmful uses, such as harassment or NCII, present distinct challenges from AI systems with dual use risks. Such models should most aptly be compared to malware in the context of traditional software. 28 On model sharing platforms, such models are subject to moderation following content policies. 29 Providing explicit guidance on categories of misuse and learning from existing mechanisms of platform governance to promote cybersecurity will be essential to managing those risks. In some cases, broadly useful models can be fine-tuned or adapted to specifically re-purpose them for such harmful uses. Safety by design practices can address those risks, for example limiting the prevalence of data that would make the models easier to misuse in general pre-training 30 or researching new methods for\n\n22 Open Source Software Security | CISA\n\n23 Artificial Intelligence | CISA\n\n24 On the Societal Impact of Open Foundation Models\n\n25 Deconstructing Design Decisions: Why Courts Must Interrogate Machine Learning and Other Technologies\n\n26 College professors are in 'full-on crisis mode' as they catch one 'ChatGPT plagiarist' after another\n\n27 The impact of ChatGPT on higher education\n\n28 Malware, Phishing, and Ransomware | Cybersecurity and Infrastructure Security Agency CISA\n\n29 Content Policy - Hugging Face\n\n30 StarCoder 2 and The Stack v2: The Next Generation; developers opted to remove known malware from pre-training data to make\n\nit harder to fine-tune for misuse\n\n\n\nsecuring models at the weights level 31 . Additionally, fine-tuning models still requires collating datasets of harmful uses; limiting access to those represents another important layer of risk mitigation.\n\nTaking action after a model release when new risks are identified presents distinct challenges and opportunities for closed and open weights models . API access models can be deleted either voluntarily by the deployer or by court request when found to be in breach of existing regulation, 32 but the cost of doing so when they are tightly integrated in a widely used commercial product may disincentivize organizations from taking actions, and internal scrutiny may not always be sufficient. 33 Conversely, good platform governance can more flexibly provide warnings and documentation, switch adoption to safer models, add technical access management such as gating, and remove models that violate platform policy; drastically reducing availability as a risk mitigation strategy. 34\n\nFor models shared through peer-to-peer systems such as torrents, updates or removals will often rely on alternative communication channels and interventions in deployed contexts.\n\nReleasing training and especially evaluation and testing datasets is usually a net positive from a risk perspective. Information about how a model was trained, where its data may present risks, what mitigation strategies were adopted, and how it was tested, can help to identify model vulnerabilities or limitations and conduct risk-benefit analyses. Challenges include intellectual property and privacy of data subjects. We note several approaches available to effectively manage these tensions 35 .\n\n- b. Could open foundation models reduce equity in rights and safety-impacting AI systems (e.g. healthcare, education, criminal justice, housing, online platforms, etc.)?\n\nMaximally open systems, including training data, weights, and evaluation protocol, can aid in identifying flaws and biases. Insufficient documentation can reduce effectiveness. 36 To maximize benefits, we recommend developers maintain up-to-date information about systems and publicly disclose biases and vulnerabilities as they are identified. The Hugging Face Hub platform supports a collaborative approach to model documentation. 37 We encourage adopters to maintain an inventory of which open systems they are using to help understand relevant risks to study and document. An example could be an inventory of systems used in the federal government. 38\n\n31 Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models\n\n32 FTC settlement includes removal of trained facial recognition model\n\n33 Microsoft engineer sounds alarm on company's AI image generator in letter to FTC\n\n34 Content Policy - Hugging Face\n\n35 Training Data Transparency in AI: Tools, Trends, and Policy Recommendations\n\n37 Model Cards - Hugging Face\n\n36 Black-Box Access is Insufficient for Rigorous AI Audits \ud83d\udcda\n\n38 Hugging Face Response to OMB RFC on federal use of AI\n\n\ud83d\uddf3\n\n\n\n- c. What, if any, risks related to privacy could result from the wide availability of model weights?\n\nResearch has shown trained models can regurgitate private information from their pre-training. This is a sign of improperly trained models, and the behavior is difficult to predict. Weight availability can marginally exacerbate attacks, but is rarely the most efficient method. Model memorization is highly dependent on the model architecture, size, and training procedure. 394041 Mitigations include proper data governance and management practices during training. 42 Open development has exemplified these practices, including via privacy-conscious data sourcing (e.g. BigScience 43 ), and developing pseudonymization and personal data redaction methods for the training data domain (e.g. the StarCoder 44 ). Privacy risks tied to the use of foundation models chiefly arise from API deployment settings that store user data, 45 however, which open-weights models can help mitigate (see our answer to 3b below).\n\nd. Are there novel ways that state or non-state actors could use widely available model weights to create or exacerbate security risks, including but not limited to threats to infrastructure, public health, human and civil rights, democracy, defense, and the economy?\n\nOpen-weight models are unnecessary for state actor threats; should state actors prefer to use FMs, they likely would prefer to build their own capabilities. 46 Narrow AI systems are more tailorable to a given threat. Models with realistic outputs are expensive to use, often more so than human alternatives or narrower systems. For example, the costs of human and AI generated disinformation do not always favor large language model (LLM) use. 47\n\nAll existing models exacerbate risks given the following conditions: models introduce a new attack vector 48 ; models reduce the overall cost of attack through generation or distribution 49 ; malicious actors are unable to build a tailored model. When closed-weight models allow higher scale of use via free or compute-subsidized access, relative risks should be reevaluated.\n\nOpen-weight model risks can outweigh those of APIs when the following additional conditions are met: models need fine-tuning in ways unavailable via API; malicious content can easily be automatically identified as malicious (e.g. generated outputs for malware creation or legitimate software 50 ); safeguards are robust to basic evasion (e.g. 'jailbreaking' 51 or traditional content moderation evasion techniques 52 ).\n\n39 Quantifying Memorization Across Neural Language Models\n\n40 Extracting Training Data from Diffusion Models\n\n41 Understanding and Mitigating Copying in Diffusion Models\n\n42 https://github.com/allenai/fm-cheatsheet/blob/main/app/resources/paper.pdf\n\n43 Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources\n\n44 StarCoder: may the source be with you!\n\n45 AIDB: ChatGPT Banned by Italian Authority Due to OpenAI's Lack of Legal Basis for Data Collection and Age Verification\n\n- 46 Yi-34B, Llama 2, and common practices in LLM training: a fact check of the New York Times | EleutherAI Blog\n\n47 How Much Money Could Large Language Models Save Propagandists? | Center for Security and Emerging Technology\n\n48 Google's Bard AI chatbot is vulnerable to use by hackers. So is ChatGPT.\n\n49 The criminal use of ChatGPT - a cautionary tale about large language models | Europol\n\n50 Incident 443: ChatGPT Abused to Develop Malicious Softwares\n\n51 [2307.15043] Universal and Transferable Adversarial Attacks on Aligned Language Models\n\n52 Microsoft CEO responds to AI-generated Taylor Swift fake nude images\n\n\n\n- e. What, if any, risks could result from differences in access to widely available models across different jurisdictions?\n\nCross-jurisdictional legal uncertainty can be detrimental to research collaborations that span sectors, organizations, and geographies. This includes liability per contributor, IP and copyright law, and foundational laws around digital privacy and regulation of misuses such as CSAM.\n\n- f. Which are the most severe, and which the most likely risks described in answering the questions above? How do these set of risks relate to each other, if at all?\n\nIn terms of novelty and scalability of the application, disparate impact of openness, and severity of the harms, non-consensual intimate imagery (NCII) in static images and videos remains the most serious and pressing risk factor of new FMs. Openness, including access to training datasets and training code, aids mitigation by enabling external scrutiny through investigating the highest contributor factors (especially at the dataset level 53 ) and enabling safety by design.\n\nMarginal risk for open-weight models is less certain for pressing risks such as nonconsensual voice cloning; hosted models can be equally harmful 54 with little recourse. The most concerning models fall well below proposed thresholds for dual use, and can easily be independently trained from scratch by medium-resourced actors. Effective policy action should target use cases and provide guidelines for all models across availability.\n\n- 3. What are the benefits of foundation models with model weights that are widely available as compared to fully closed models?\n- a. What benefits do open model weights offer for competition and innovation, both in the AI marketplace and in other areas of the economy? In what ways can open dual-use foundation models enable or enhance scientific research, as well as education/training in computer science and related fields?\n\nOpen-weight models contribute to competition, innovation, and broad understanding of AI systems to support effective and reliable development. In terms of value creation of open technology , the historical and economic impact of Open Source Software (OSS) provides important context for the expected impact of open-weight models. Two main phenomena bear strong relevance: First, the estimated demand-side value of OSS is several orders of magnitude larger ($8.8 trillion) than its supply-side value (>$4 billion). 55 Investment in open technology pays off over 1000x in value in other areas of the economy. Second, adoption enables users to reallocate resources to internal development that supports market diversification and sustainable self-driven growth 56 , which is critical for start-ups and SMEs. OSS trades off capital for human expertise. Recent work has shown that these dynamics are extending to AI;\n\n53 Multimodal datasets: misogyny, pornography, and malignant stereotypes\n\n54 AI Startup ElevenLabs Bans Account Blamed for Biden Audio Deepfake - Bloomberg\n\n55 The Value of Open Source Software\n\n56 Open Source Software and Global Entrepreneurship\n\n\n\ncost efficiency and customizability increasingly outweigh managed solutions, given sufficient company expertise. 57\n\nOpen-weight models are particularly relevant to mitigating market concentration at the data level . High-quality data, both in the form of licensed content, 5859 interaction data, and data uploaded by customers of managed systems, is shaping up to be the next most likely point of monopoly behaviors as training compute costs decrease (see our answer to Q1.a). Open weights models increase the diversity of product offerings from providers and even enable adopters to manage their own on-premise solutions. Companies using open-weights models or systems built on them are in a position to keep their own data value, 60 preserve privacy, and build consortia on their own terms as needed to support better technology development for their applications without having to give up their intellectual property. 61 This is needed to sustain high-quality data creation and fair valuation of the data contribution to AI systems. Additionally, open-weight models have been customized and adapted to run on a greater variety of infrastructure, including individual GPUs and even CPUs, reducing points of market concentration with cloud providers and reducing costs for procurement. 6263\n\nAI foundation models combine innovative ways of building AI systems-and most innovation supporting recent AI product releases has come from open and academic research. For example, the video generation system SORA builds on many public research contributions. 64 Software behind development is largely open source (Pytorch, Tensorflow, HF libraries) and development often requires access to the weights. More convenient 65 and more secure 66 weight storage formats are developed on OSS settings, among many framework improvements. This includes fine-tuning major LLMs and research breakthroughs such as model merging. 67 Open-weight models' role in enhancing existing scientific research is evident in works on topics ranging from knowledge distillation 68 to watermarking 69 . Robust innovation on both performance and safety questions requires scientific rigor and scrutiny, which is enabled by openness and external reproducibility 70 . Supporting that research requires sharing models to validate findings and lower the barrier to entry for participation given the growing resource gap between researchers in different institutions. 71 Recent open-weights releases of foundation models from AI system providers such as Cohere's Command+R, 72 Google's Gemma, 73 and OpenAI's Whisper 3 74 , among others, represent a welcome contribution to this research.\n\n57 16 Changes to the Way Enterprises Are Building and Buying Generative AI | Andreessen Horowitz\n\n58 Exclusive: Reddit in AI content licensing deal with Google | Reuters\n\n- 59 OpenAI In Talks With Dozens of Publishers to License Content - Bloomberg\n- 60 Will StarCoder 2 Win Over Enterprises?\n- 61 SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore\n- 62 GitHub - ggerganov/llama.cpp: LLM inference in C/C++\n- 63 llamafile: bringing LLMs to the people, and to your own computer - Mozilla Innovations\n- 64 Sora Reference Papers - a fffiloni Collection\n- 65 GGUF model saving format\n- 66 GitHub - huggingface/safetensors: Simple, safe way to store and distribute tensors\n- 67 Evolving New Foundation Models: Unleashing the Power of Automating Model Development\n- 68 Knowledge Distillation of Large Language Models\n- 69 A Watermark for Large Language Models\n- 70 EleutherAI: Going Beyond \"Open Science\" to \"Science in the Open\"\n- 71 The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?\n- 72 Command R\n- 73 Gemma: Google introduces new state-of-the-art open models\n- 74 Introducing Whisper, openai/whisper-large-v3 \u00b7 Hugging Face\n\n\n\nOpen Weights Models and Evaluation\n\nThe availability of open-weights foundation models is of particular importance to the development of the robust and reliable evaluation ecosystem required to properly govern this category of technology. Access to open-weights models serves several distinct purposes for evaluation. First, some categories of evaluation require direct access to model weights . Evaluation of a model's environmental impact and energy consumption, for example, involves running models on controlled hardware. 75 Evaluating model memorization behaviors 76 (related to privacy and intellectual property concerns), covert biases, 77 and data contamination 78 all require varying levels of access from output logits to model activations without additional input or output processing that are not consistently available through API deployment.\n\nSecond, access to model weights makes research and development of new evaluations significantly cheaper . Developing a new evaluation technique typically requires extensive iteration on variations of a system. Being able to run a model locally in a controlled environment makes this process significantly more efficient and cheaper, and has been instrumental for example in our own work on developing bias evaluations in image generation settings before applying them to commercial systems. 79\n\nThird, while model developers should be incentivized to thoroughly evaluate their models, their incentives and priorities may be mis-aligned with those of external stakeholders , especially when models are part of commercial product development. Developer-run evaluations may underplay limitations and harms of a model - often without explicitly setting out to do so; to address this, recent work has called for safe harbor regimes to enable external research 80 , and outlined limitations of 'black-box' access 81 and auditing without sufficient agency. 82 Similarly, self-reported performance numbers on task can often be misleading or over-represent a model's suitability for use. Common issues that contribute to this phenomenon include misleading evaluation framing 83 , choice of metrics (leading to different perceptions of 'emergen capabilities' 84 ), rampant issues of inappropriate benchmark decontamination, 8586 and reporting results on different model versions than the ones deployed in products. 87 Finally, evaluations of foundation model applications in specific domains should be led by experts in these domains rather than by model developers to best reflect the risks and benefits of AI adoption. 88\n\n75 Power Hungry Processing: Watts Driving the Cost of AI Deployment?\n\n76 Quantifying Memorization Across Neural Language Models\n\n77 Dialect prejudice predicts AI decisions about people's character, employability, and criminality\n\n78 Membership Inference Attacks against Language Models via Neighbourhood Comparison - ACL Anthology\n\n79 Stable Bias: Analyzing Societal Representations in Diffusion Models\n\n80 A Safe Harbor for AI Evaluation and Red Teaming\n\n81 Black-Box Access is Insufficient for Rigorous AI Audits\n\n82 AI auditing: The Broken Bus on the Road to AI Accountability\n\n83 GPT-4 and professional benchmarks: the wrong answer to the wrong question\n\n84 Are Emergent Abilities of Large Language Models a Mirage?\n\n85 Investigating Data Contamination for Pre-training Language Models\n\n86 Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs\n\n87 An In-depth Look at Gemini's Language Abilities\n\n88 The Shaky Foundations of Foundation Models in Healthcare\n\n\n\nFourth, results obtained through third-party use of a model's managed API are unreliable without extensive documentation of the model versions, processing steps, or risk mitigation strategies. Model providers can change the underlying version of a model at any time, 89 and do not always disclose sufficient details about post-processing steps that might have an impact on the results, such as automatic edition of prompts. 90 This is particularly relevant since, regardless of any intentional misrepresentation, commonly used evaluations have been shown to be sensitive to specific implementation details that often require extensive evaluation to characterize. 9192\n\nAccess to open-weights models that are substantially similar to those used in commercial products is thus necessary to support a robust evaluation ecosystem , both to guide further development of the technology and to support the development of appropriate standards.\n\n- b. How can making model weights widely available improve the safety, security, and trustworthiness of AI and the robustness of public preparedness against potential AI risks?\n\nOpen-weight models can help in identifying attacks before they can be leveraged at scale; for example, recent work on 'stealing' the last layer was developed with LLaMA 2 93 . Research has also found some adversarial attacks apply to open and closed models, 94 flagging the need to understand limitations of safety techniques. As discussed in 3a, reproducibility increases trust in AI systems by ensuring scientific rigor. Replicable evaluations with full transparency of replicable pieces can resolve misunderstandings or misleading comparisons. 95 Deployers canalso update models as needed, without being locked into hosted model contracts and agreements - which is particularly relevant when model vulnerabilities are identified or when a plausibly more secure model becomes available.\n\nProminent privacy concerns arise from API deployment and user data storage 9697 and training on user data that contains private information. 98 This tendency is particularly concerning as access to private information has been identified as a significantly stronger risk factor than model capacity in some misinformation settings. 99 Open-weight models can mitigate this risk by enabling a more controlled deployment environment and obviating the need to send data to third party organizations. Open practices can enable broader red-teaming and evaluation practices focused on privacy that can help secure all models. 100\n\n89 GPT-4 API general availability and deprecation of older models in the Completions API\n\n90 DALL\u00b7E 3\n\n91 What's going on with the Open LLM Leaderboard?\n\n92 Open LLM Leaderboard: DROP deep dive\n\n93 Stealing Part of a Production Language Model\n\n94 Universal and Transferable Adversarial Attacks on Aligned Language Models\n\n95 What's going on with the Open LLM Leaderboard?\n\n96 March 20 ChatGPT outage: Here's what happened\n\n97 AI Incident database: Incident 513: ChatGPT Banned by Italian Authority Due to OpenAI's Lack of Legal Basis for Data Collection and Age Verification\n\n98 \"It's a Fair Game'', or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents\n\nOn the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial\n\n100 Privacy risks in deployment: Introducing the Chatbot Guardrails Arena 99\n\n\n\n- c. Could open model weights, and in particular the ability to retrain models, help advance equity in rights and safety-impacting AI systems (e.g. healthcare, education, criminal justice, housing, online platforms etc.)?\n\nYes; in addition to examples in 3a, LLaMA's adaptation into Alpaca 101 significantly eased accessibility while maintaining high performance. Some API restrictions prevent equity-advancing research. 102 Open practices, such as open data work, can help address historical biases and prevent 'hate-scaling'. 103\n\n- e. How do these benefits change, if at all, when the training data or the associated source code of the model is simultaneously widely available?\n\nAccess to the training data of an open-weights foundation model provides substantial additional benefits from the perspective of research, rights and governance, and safety. 104 Direct access top the training dataset can support research on model privacy and memorization, 105 risks tied to hate content 106 or NCII generation 107 , intentional and unintentional impacts of data filtering approaches 108 , among others. Interactive or managed access to a dataset can similarly help answer questions about data provenance and privacy 109 and support data governance centered on the rights of data subjects. 110 Support for projects that develop foundation models in fully open settings by making both trained weights and access or extensive documentation of their dataset available can enable more socially responsible and inclusive technology development. 111\n\n- 4. Are there other relevant components of open foundation models that, if simultaneously widely available, would change the risks or benefits presented by widely available model weights? If so, please list them and explain their impact.\n\nIn addition to examining artifacts listed in 1, often-overlooked artifacts include process and documentation. Project governance shapes technical artifacts, including goals, trade-offs, funding, and mechanisms for internal feedback. In fully open development, processes include goal setting, value alignment, and enabling various stakeholders to question upstream choices. The BigScience project provides this level of openness for a multilingual LLM, 112 and the BigCode project 113 took a similar approach to develop a code LLM, including a new form of Governance Card to report relevant information including funding, main development choices and reasoning, handling of cybersecurity risks and personal data, and annotator pay for data augmentation work. 114\n\n101 Alpaca: A Strong, Replicable Instruction-Following Model\n\n102 Dialect prejudice predicts AI decisions about people's character, employability, and criminality\n\n103 On Hate Scaling Laws For Data-Swamps\n\n104 Training Data Transparency in AI: Tools, Trends, and Policy Recommendations\n\nOn Hate Scaling Laws For Data-Swamps\n\n106 105 How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN \ud83d\udcda \ud83d\uddf3\n\n107 Multimodal datasets: misogyny, pornography, and malignant stereotypes\n\n108 A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity\n\n109 The ROOTS Search Tool: Data Transparency for LLMs\n\n110 Data Governance in the Age of Large-Scale Data-Driven Language Technology\n\n111 Social Context of LLMs - the BigScience Approach, Part 3: Data Governance and Representation | Montreal AI Ethics Institute\n\n112 BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model\n\n113 BigCode Project\n\n114 The BigCode Project Governance Card\n\n\n\nDocumentation approaches include model cards 115 and datasheets 116 that can be shared broadly. Accompanying weights with information necessary to assess their usability for a given use case helps ensure they are used in safer contexts. Data quality is broadly recognized to directly affect ML system performance, 117 and dataset limitations, such as toxicity and harmful social biases, are typically reflected in a trained model. 118 Datasets are often the most practical artifact of study to understand the capabilities, risks, and limitations of a model. Sufficient access to a training dataset 119 can help answer relevant questions about a model's characteristics 120 .\n\n- 5. What are the safety-related or broader technical issues involved in managing risks and amplifying benefits of dual-use foundation models with widely available model weights?\n\nWe need significantly more investment in evaluation to foster practices of safety, privacy, and fairness by design. Results from reliable evaluation methodologies and design decisions such as data selection can determine whether a model should be used, and in what conditions.\n\n- a. What model evaluations, if any, can help determine the risks or benefits associated with making weights of a foundation model widely available?\n\nCritical variables for assessing risks and benefits include type of model and design process and data selection. Model evaluations are important in determining usability and use case, and in shaping moderation decisions, but should be complemented with sufficient documentation of both the development process and governance mechanisms (see responses to 3e and 4) and the specific evaluation methodology (see response to 3a). Past moderation decisions on Hugging Face about whether sharing specific models was consistent with our terms of service has depended on all of those factors, as model evaluations by themselves can fail to sufficiently address social dynamics and the context of use and development.\n\nModels trained specifically for sensitive applications, such as identifying personal data (e.g. the StarPII model 121 used for private information redaction in the BigCode project training data) or producing hate speech, 122 require different governance and access to more specific stakeholders. Processes for determining release method will differ by deployer organization and calls for collective decision making boards are unanswered, although discourse is ongoing. 123124\n\n115 Model Cards for Model Reporting\n\n116 Datasheets for Datasets\n\n117 The Effects of Data Quality on Machine Learning Performance\n\n120 The ROOTS Search Tool: Data Transparency for LLMs 119 \ud83d\udcda Training Data Transparency in AI: Tools, Trends, and Policy Recommendations \ud83d\uddf3 118 A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity\n\n121 StarPII release decision\n\n122 Handling and Presenting Harmful Text in NLP Research\n\n123 Publication Norms for Responsible AI - Partnership on AI\n\n124 The Time Is Now to Develop Community Norms for the Release of Foundation Models\n\n\n\n- b. Are there effective ways to create safeguards around FMs, either to ensure that model weights do not become available, or to protect system integrity or human well-being (including privacy) and reduce security risks in those cases where weights are widely available?\n\nPrivacy by design includes steps throughout development including removing unnecessary personal data from a training dataset (see BigScience and BigCode examples in 2c) and following data minimization principles 125 helps reduce risks described in 2c. However, privacy risks also come from the extent to which current deployed systems encourage users to share personal data during their interactions. 126 Ensuring that the product design minimizes instances where this information is stored or transmitted helps prevent damaging breaches.\n\nInterventions at the fine-tuning level, such as in Constitutional AI or reinforcement learning with human feedback to aid instruction following, can help steer a model towards more desirable behaviors. While these interventions have an important role to play in adding friction to misuses of models, they are also limited in their robustness and effectiveness 127 and insufficient to address fundamental issues with models. 128129 Input and output monitoring can also be used to automatically block queries that are incompatible with a model's term of services. However,this approach presents all of the familiar limitations and potential discriminate impacts of automatic content moderation. In particular, the social cost of developing these safeguard datasets 130 and of live content moderation 131 should be considered and minimized.\n\nModel weight encryption is an ongoing research area, with proposals 132 but no wide deployment.\n\n- c. What are the prospects for developing effective safeguards in the future?\n\nIn addition to building on 5b, safety by design, tight scoping of objectives, and intentional and robust data curation all are paths forward. Using AI as safeguards on AI systems at the deployment level may mitigate some risks, can easily compound biases and have disproportionate impact on minority groups using the systems. 133\n\n- d. Are there ways to regain control over and/or restrict access to and/or limit use of weights of an open foundation model that, either inadvertently or purposely, have already become widely available? What are the approximate costs of these methods today? How reliable are they?\n\nClearly defined platform governance frameworks and disclosure mechanisms can most effectively limit availability for systems that require this guardrail due to vulnerabilities or incompatibility with existing regulations. For models found to present critical vulnerabilities, AI can draw on cybersecurity practices to notify users of updates. More research is needed to\n\n125 Artificial intelligence: the CNIL opens a consultation on the creation of datasets for AI\n\n127 Jailbreaking Black Box Large Language Models in Twenty Queries 126 \"It's a Fair Game\", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conv\n\n128 Dialect prejudice predicts AI decisions about people's character, employability, and criminality\n\n130 129 The Male CEO and the Female Assistant: Probing Gender Biases in Text-To-Image Models Through Paired Stereotype Test\n\nInmates in Finland are training AI as part of prison labor - The Verge\n\n131 Inside Facebook's African Sweatshop | TIME\n\n132 NN-Lock: A Lightweight Authorization to Prevent IP Threats of Deep Learning Models | ACM Journal on Emerging Technologies in Computing Systems\n\n133 Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection\n\n\n\nscope what constitutes a critical vulnerability and to standardize evaluations. For models that violate existing regulations or platform content policies, removal from the platform can help limit their spread and mitigate their negative uses. On the Hugging Face platform, we have taken actions to limit the use of models trained in a way that leads them to disproportionately produce harassing text (e.g. GPT4chan 134 ) or models that were trained to reproduce a person's voice or likeness without their consent.\n\n- e. What if any secure storage techniques or practices could be considered necessary to prevent unintentional distribution of model weights?\n\nApproaches can include a trusted research environment that provides researchers full access to model components through an isolated virtual machine (e.g. HathiTrust Data Capsules 135 ) to ensure security, but are resource-intensive in terms of both computational and human support to adapt the environment to specific research needs. FMs and LLMs are notably computationally intensive. Public repositories with access management (such as Hugging Face gated repositories 136 ) can conveniently balance easy access for legitimate researchers and stakeholders and limited release of model weights. In staged or gated releases, researchers may enter legal agreements with terms of use including preventing model weight distribution.\n\n- f. Which components of a foundation model need to be available, and to whom, in order to analyze, evaluate, certify, or red-team the model? To the extent possible, 14 please identify specific evaluations or types of evaluations and the component(s) that need to be available for each.\n\nDifferent tasks and level of access needed per task may not be obvious just as risk landscapes for dual-use FMs cannot be fully taxonomized at a given time. While this mapping exercise has not and likely cannot be exhaustively conducted, some examples include bias evaluations listed in 3a and Table 1 of the paper Black-Box Access is Insufficient for Rigorous AI Audits details audit tasks and levels of access needed. 137\n\n- g. Are there means by which to test or verify model weights? What methodology or methodologies exist to audit model weights and/or foundation models?\n\nHugging Face has used a simple model hash to verify whether two models are the same item. For open-weight models, verification is as simple as for any other large files. For APIs, verification is significantly more complex given the stochastic nature of the outputs and complex stack. As outlined above, this can be a source of security vulnerabilities for critical infrastructure that relies on externally managed APIs.\n\n134 ykilcher/gpt-4chan \u00b7 Hugging Face\n\n135 Data Capsules\n\n136 Gated models\n\n137 [2401.14446] Black-Box Access is Insufficient for Rigorous AI Audits\n\n\n\n- 7. What are current or potential voluntary, domestic regulatory, and international mechanisms to manage the risks and maximize the benefits of foundation models with widely available weights? What kind of entities should take a leadership role across which features of governance?\n- a. What security, legal, or other measures can reasonably be employed to reliably prevent wide availability of access to a foundation model's weights, or limit their end use?\n\nPopular measures include terms of licenses with use clauses 138 and gating mechanisms in addition to safeguards enumerated in recent work. 139\n\n- b. How might the wide availability of open foundation model weights facilitate, or else frustrate, government action in AI regulation?\n\nAs noted in 2, 3, and 5, wide availability of open foundation model weights facilitates government action in AI regulation by supporting research on evaluation and standards, fostering more transparent and accountable development, and promoting fair competition. As noted in 2, governance of widely available open-weights models can also require different interventions than API-only models, with different benefits and challenges.\n\n- c. When, if ever, should entities deploying AI disclose to users or the general public that they are using open foundation models either with or without widely available weights?\n\nEntities deploying AI systems should typically disclose that fact, as noted in the AI Bill of Rights. 140 In general, extending the concept of a Software Bill of Materials (SBOM) 141 to AI systems, including identifying which versions of foundation models (open or closed) are being used would have beneficial implications for cybersecurity and reliability.\n\n- d. What role, if any, should the U.S. government take in setting metrics for risk, creating standards for best practices, and/or supporting or restricting the availability of foundation model weights? i. Should other government or non-government bodies, currently existing or not, support the government in this role? Should this vary by sector?\n\nEvaluation presents two main challenges. First, specific risk evaluations is an open research area, and risk priorities vary by organization. Evaluations as a field require significant investment; widely used benchmarks for both performance and risks are criticized for often providing insufficient or even misleading results. 142 Second, model evaluation often depends on the type of models evaluated and type of risk 143 , especially for topics like privacy risks. Continual investment is needed to ensure evaluations reflect risks as models evolve. Sectoral contexts improve metric robustness.\n\n138 OpenRAIL: Towards open and responsible AI licensing frameworks\n\n139 The Gradient of Generative AI Release: Methods and Considerations\n\n140 Blueprint for an AI Bill of Rights | OSTP | The White House\n\n141 Software Bill of Materials (SBOM) | CISA\n\n142 Dialect prejudice predicts AI decisions about people's character, employability, and criminality\n\n143 Evaluating the Social Impact of Generative AI Systems in Systems and Society\n\n\n\nThe U.S. government action can include establishing standards for best practices building on existing work 144 and prioritize requirements of safety by design across both the AI development chain and its deployment environments. 145 General conditions should treat models that are broadly commercialized and broadly shared similarly. Actions should foster a research ecosystem that has sufficient access to the artifacts and infrastructure to conduct research, incentives to share lessons and artifacts for reproducibility, access to broad subject matter experts including social scientists.\n\n- e. What should the role of model hosting services (e.g. HuggingFace, GitHub, etc.) be in making dual-use models with open weights more or less available? Should hosting services host models that do not meet certain safety standards? By whom should those standards be prescribed? 16\n\nAt Hugging Face, use and utility are contextual. Our platform provides features for access management to enable developers to tailor model availability and breadth of access to strike the best balance between benefits and risks. We universally encourage extensive and transparent documentation, mandating documentation for artifacts over 10,000 downloads and providing guides and resources. Content guidelines 146 address specific harms, including breach of consent 147 and use of models for intentional harm. Sharing platforms enable research on systems that may serve different purposes, and in many cases purely as research artifacts.\n\n- f. Should there be different standards for government as opposed to private industry when it comes to sharing model weights of open foundation models or contracting with companies who use them?\n\nGiven its responsibility to stakeholders, the government should meet sufficient procurement requirements to ensure accountability, rights-respecting deployment of technology, and to ensure that AI adoption does provide a net benefit for considered use cases. We appreciate the EO directive to do so and the work of the OMB in this direction.\n\n- g. What should the U.S. prioritize in working with other countries on this topic, and which countries are most important to work with?\n\nInternational collaboration is key, especially among U.S. allies such as the UK, Singapore, and France. Vulnerability and security incident sharing is best raised among allies, especially for those with dedicated AI bodies such as the U.S. and UK AI Safety Institutes.\n\nInternational standards, including among countries with differing values and regulatory approaches, are needed. An urgent need is disclosure standards, outlining what should be disclosed broadly, which can be fine-tuned per jurisdiction to meet requirements for example in\n\n144 NIST Risk Management Framework (RMF)\n\n145 Hugging Face Response to OMB RFC on federal use of AI\n\n146 Announcing our new Content Guidelines and Policy\n\n147 Announcing our new Content Guidelines and Policy\n\n\n\nadhering to local laws. Evaluation standards can help build cross-cultural and multilingual approaches to model safety.\n\nh. What insights from other countries or other societal systems are most useful to consider?\n\nLessons can be drawn from the EU's established AI Office, which maintains standards and operational definitions as AI evolves, and is empowered to make case-by-case definitions about which models might present a systemic risk to reflect the highly contextual nature of risk assessments. The U.S. should designate an accountable organization for metrics, standards, and requirements, and sufficiently resourcing them to keep pace with AI development. The National Institute of Standards and Technology is well positioned and should be well resourced.\n\ni. Are there effective mechanisms or procedures that can be used by the government or companies to make decisions regarding an appropriate degree of availability of model weights in a dual-use foundation model or the dual-use foundation model ecosystem? Are there methods for making effective decisions about open AI deployment that balance both benefits and risks? This may include responsible capability scaling policies, preparedness frameworks, et cetera.\n\nAs discussed in 5a, no definitive processes or standards exist. In addition to discourse referenced in 5a, ongoing cross-sectoral collaborations are sharing best practices. 148 Strict guidelines about safety by design processes can be helpful for developers as they have control over relevant properties, including what data selection, whether the data covers high risk topics such as nuclear devices, and whether the model is trained to produce malware. Limitations in the state of evaluation development makes quantifying model behavior technically difficult.\n\nAs discussed in 5d, models that are trained specifically on PII/malware/hate speech may have different release conditions. Non-regulatory practices such as vulnerability sharing can draw from cybersecurity. In high risk settings, existing controls on information flow should apply. 149 Narrow models for specific domains or applications should use practices from that domain.\n\nj. Are there particular individuals/entities who should or should not have access to open-weight foundation models? If so, why and under what circumstances?\n\nResearch institutions, especially those operating in public interest and outside of commercial product development, should not only have access but also infrastructural support. Government resourcing 150 can not only provide financial and technical support, but also ensure that researchers with sufficient breadth of expertise and external stakeholders have access to technical artifacts that may be used in rights-impacting settings. As noted in 3a, third party auditing is most effective when entities who can conditions without developer engagement. 151 Open-weight models can help create these conditions, along with varying\n\n148 PAI's Deployment Guidance for Foundation Model Safety\n\n149 How AI Can Be Regulated Like Nuclear Energy\n\n150 National Artificial Intelligence Research Resource Pilot | NSF\n\n151 Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance\n\n\n\naccess to components such as datasets, including e.g. extensive documentation and query access. 152\n\n- 8. In the face of continually changing technology, and given unforeseen risks and benefits, how can governments, companies, and individuals make decisions or plans today about open foundation models that will be useful in the future?\n- a. How should these potentially competing interests of innovation, competition, and security be addressed or balanced?\n\nPolicy decisions can jointly advance interests of innovation, competition, and security. Work across technological settings (e.g. in the contexts of election systems 153 or biosecurity in protein design 154 ) has shown that a priori antagonistic goals can often both benefit from openness given appropriately tailored approaches. In specific settings where the tensions are harder to resolve, specificity in how these tensions are managed and narrowly tailored policies are particularly important. 155 Across both settings of widely available weights and API-only development, extensive documentation, replicable evaluation, and transparency into design choices of FMs all contribute positively to all of these interests.\n\n- b. Noting that E.O. 14110 grants the Secretary of Commerce the capacity to adapt the threshold, is the amount of computational resources required to build a model, such as the cutoff of 1026 integer or floating-point operations used in the 17 Executive Order, a useful metric for thresholds to mitigate risk in the long-term, particularly for risks associated with wide availability of model weights?\n\nConcerns with thresholds include their robustness, purpose/trigger, enforcement, and verifiability. Methods for determining and proxies for model capability have differing effectiveness. Thresholds along any singular variable, such as training compute, will not give robust insight to model capability and will adapt with time and incentives. Additionally, while model evaluations should eventually play a major role in helping assess proper use and governance of foundation models, our scientific understanding of these artifacts and evaluation systems are not currently mature or robust enough to serve this purpose; and will require significant additional investment and access to open models as outlined in our response to 3a.\n\nFor determining what potential thresholds should trigger, we recommend starting with a voluntary reporting framework. Rather than red-teaming results, disclosure should include what replicable evaluations have been run, including on a spectrum of risks. This will build regulatory capacity and foster a stronger evaluation ecosystem. Thresholds should not be singular hardlines. 156 Levels of mandates should apply by contextual use case and mode of\n\n152\n\nTraining Data Transparency in AI: Tools, Trends, and Policy Recommendations\n\n154 Protein design meets biosecurity | Science\n\n153 Transparency versus security: early analysis of antagonistic requirements \ud83d\udcda\n\n\ud83d\uddf3\n\n155 Openness versus Secrecy in Scientific Research Abstract - PMC\n\n156\n\nDrawing Lines: Tiers for Foundation Models\n\n\n\ndeployment , whether in research or commercial applications. Robust transparency requirements should apply to all commercial products.\n\nDisclosure is critical and lessons can be drawn from cybersecurity to determining levels of disclosure and what can be made public. Meeting a base level requirement should mandate some level of disclosure. Models whose deployment is supported by extensive cloud compute capacity, due to model size or volume of users, should warrant additional scrutiny including cybersecurity audits given the scale of their impact. Robust documentation can also be beneficial for commercial usability and trust.\n\n## Conclusion\n\nAI innovation would not be possible without open access and open science. Openness broadly continues to benefit the field. Since the AI field is a fast-evolving space with many arising considerations, expanding scope to many system artifacts can help to better weigh risks and benefits. Many risks apply across model weight availability and tailored threat models can narrow intervention options. Hugging Face greatly appreciates the opportunity to provide insights and we remain eager to support NTIA and provide our expertise moving forward.", + "chunks": [ + { + "headings": [ + "Hugging Face, Inc. 20 Jay Street, Suite 620," + ], + "markdown": "Hugging Face, Inc. 20 Jay Street, Suite 620,\n\nBrooklyn, NY 11201\n\n## Hugging Face Response to the National Telecommunications and Information Administration Request for Comments on Dual Use Foundation Artificial Intelligence Models with Widely Available Model Weights\n\nHugging Face applauds the ongoing work of the National Telecommunications and Information Administration (NTIA) in examining dual use foundation models (FMs). The following comments are informed by our experiences as an open platform for AI systems, working to make AI accessible and broadly available to researchers for responsible development.\n\n## About Hugging Face\n\nHugging Face is a community-oriented company based in the U.S. and France working to democratize good Machine Learning (ML), and has become the most widely used platform for sharing and collaborating on ML systems. We are an open-source and open-science platform hosting machine learning models and datasets within an infrastructure that supports easily processing and analyzing them; conducting novel AI research; and providing educational resources, courses, and tooling to lower the barrier for all backgrounds to contribute to AI.\n\n## Questions\n\nIn order to best address the risks and benefits of widely available model weights, we explore both the dimension of wide availability of a model and access to model weights in our responses to avoid conflating risks inherent in the technology with risks associated with release methods. We refer to models with widely available model weights as 'open-weight models'.\n\n- 1. How should NTIA define 'open' or 'widely available' when thinking about foundation models and model weights?\n\nBoth 'open' and 'widely available' as terms are distinct, and require appropriate definitions when discussing risks and benefits.\n\n'Open' should consider components that shape model impact. The risks and benefits of foundation models are shaped by decision along their development and deployment chain (e.g. when considering risks of biases and discriminatory outcomes 1 ). The contribution of openness\n\n1 Let's talk about biases in machine learning! Ethics and Society Newsletter #2\n\n\n\n\n\nto supporting beneficial development of the technology, 2 as well as the governance processes it requires to help mitigate risks of the technology, will depend on access and transparency into not just the model weights but also its development datasets , development conditions, 3 4 and deployment infrastructure.\n\nOpen weights for a foundation model are notable on two main accounts. First, given how resource-intensive the data curation and training of the models has become, sharing this particular artifact in the AI value chain is necessary to enable downstream research that only requires model access from many stakeholders outside of well-resourced companies. 5 Second, the release of a model's weights has historically served as a catalyst for development projects that prioritize overall transparency 6 and proactive consideration of regulatory processes 7 that enable more comprehensive research on risks and benefits along the development chain.\n\nTo account for all those dynamics, we strongly recommend considering at least three main components in FM openness: its datasets , the trained model , and the model's training and deployment mechanisms .\n\n'Widely available' should consider breadth (not just scale) of access. In general, providing model access to a more diverse set of stakeholders can have a net positive impact on the safety of the technology; access should be available to stakeholders who are best positioned to identify risks and limitations 8 - and particularly to third-party organizations that do not have a direct relationship (or conflicts of interest) with the developer and direct beneficiaries. 9 Breadth of access can be achieved independently of scale through mechanisms such as gating, where access is granted by an (independent) organization for specific well-motivated research and development projects. 10 Simple availability is also distinct from accessibility; model weights being available may not be accessible to people without the necessary compute infrastructure or skill set for model hosting. For uses of a model that do not require specific customization, a managed API supported by extensive computational resources greatly increases the scale of availability independently of whether model weights are directly accessible.\n\n- a. Is there evidence or historical examples suggesting that weights of models similar to currently-closed AI systems will, or will not, likely become widely available? If so, what are they?\n\nPast examples include staged releases of model weights where closed weights are released over time, as seen with OpenAI's GPT-2 11 , Meta's LLaMA 12 , and Stability AI's Stable Diffusion 13 . All recent prominent closed models have been followed by an open comparable alternative.\n\n2 AI Policy @ : Open ML Considerations in the EU AI Act\n\n\n\n\ud83d\udcda\n\n6 Introducing The Foundation Model Transparency Index\n\n7 Do Foundation Model Providers Comply with the Draft EU AI Act?\n\n8 A Safe Harbor for Independent AI Evaluation\n\n9 Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance\n\n10 Access form for the BigScience training corpus\n\n11 Release Strategies and the Social Impacts of Language Models\n\n12 Introducing LLaMA: A foundational, 65-billion-parameter large language model\n\n13 Stable Diffusion Public Release - Stability AI\n\n\n\nExamples include OpenAI's GPT-3 deployed via API in 2021 and model weights for BigScience's BLOOM and Meta's OPT both released in 2022. Recently, DataBricks was reported to have trained a competitive large language model for 10M$, significantly lower than the initial development cost of closed products; 14 indicating that relying on cost or trade secrets is unlikely to limit availability of top-performing models to their current set of developers.\n\nBeing 'first to market' presents a strong economic competitive advantage for closed systems, and reproduction efforts of the model itself are motivated by a range of values, including scientific understanding, free and open source software (FOSS) ideals of accessibility, and customization, including fitness for purpose and harm mitigation strategies. 15\n\nb. Is it possible to generally estimate the timeframe between the deployment of a closed model and the deployment of an open foundation model of similar performance on relevant tasks? How do you expect that timeframe to change? Based on what variables? How do you expect those variables to change in the coming months and years?\n\nTimelines vary and estimates will change by utility of the model and costs. Factors include computational cost, training data accessibility, and potential anti-competitive practices from original developers. Research focus and more compute and data-efficient algorithms are advantages for open models, whereas product focus, data concentration, high resources, and legally untested or contested practices are advantages for close development. Overall, socially responsible research in open reproduction of closed models will necessarily take more time for release 16 as legal and ethical concerns are addressed proactively rather than retroactively after issues have been observed at sufficient scale.\n\n- c. Should 'wide availability' of model weights be defined by level of distribution? If so, at what level of distribution (e.g., 10,000 entities; 1 million entities; open publication; etc.) should model weights be presumed to be 'widely available'? If not, how should NTIA define 'wide availability?'?\n\nModel weights can be shared individually between parties, on platforms with or without documentation and with or without access management, and via p2p/torrent. Usability depends on resources and infrastructure; subsidized compute from cloud compute providers can increase access drastically, including for uses that go against the spirit of ToU but are hard to detect. APIs can broaden model availability, even to malicious actors, especially already relying on developer infrastructure. APIs also can be attacked to provide model information 17 . Even controlled risks can outweigh open models in magnitude through deployment scale. 18\n\n14 Inside the Creation of DBRX, the World's Most Powerful Open Source AI Model | WIRED\n\n15 Policy Readout - Columbia Convening on Openness and AI\n\n16 BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model\n\n17 Stealing Part of a Production Language Model\n\n18 Google's Bard AI chatbot is vulnerable to use by hackers. So is ChatGPT.\n\n\n\nd. Do certain forms of access to an open foundation model (web applications, Application Programming Interfaces (API), local hosting, edge deployment) provide more or less benefit or more or less risk than others? Are these risks dependent on other details of the system or application enabling access? i. Are there promising prospective forms or modes of access that could strike a more favorable benefit-risk balance? If so, what are they?\n\nThere is no standard for weighing risks and benefits, nor are there standards or clear methods for definitive access decisions. Distributed methods for sharing via torrent are arising such as tweeting magnet links as seen with xAI's Grok 19 . These are broadly accessible but do not allow for updates in information and rely on trust and robustness of the torrent sharer.\n\nWeb applications with user interfaces lower the barrier for users with little to no AI experience to engage with the model. Functionality depends on application and interface, such as the ability to toggle temperature. This is often surface-level access, and cannot be built upon.\n\nAPI deployment can offer varying functionality, allowing only querying or allowing fine-tuning. APIs ease use for prescribed use cases as they only require internet connection and possibly a user account. APIs are often commercialized. Deployment benefits include monitoring, last-layer interventions, rate-limiting. Limitations include research and auditing inaccessibility, privacy concerns, and infrastructural reliability issues. API terms of use can be monitored and detected, and users can be blocked or revoked access. However content moderation proves difficult; blocks can easily be bypassed; false positives can block good faith researchers. 20 Last-layer interventions include watermarks after output generation. Limited API access makes research less reliable or presents additional barriers when limiting access to logits, forcing version changes, or obfuscating input and output processing that confound research findings. API deployment also introduces additional privacy challenges when user inputs are collected. API", + "char_slice": [ + 0, + 10835 + ] + }, + { + "headings": [ + "## Hugging Face Response to the National Telecommunications and Information Administration Request for Comments on Dual Use Foundation Artificial Intelligence Models with Widely Available Model Weights", + "## About Hugging Face", + "## Questions" + ], + "markdown": "allow for updates in information and rely on trust and robustness of the torrent sharer.\n\nWeb applications with user interfaces lower the barrier for users with little to no AI experience to engage with the model. Functionality depends on application and interface, such as the ability to toggle temperature. This is often surface-level access, and cannot be built upon.\n\nAPI deployment can offer varying functionality, allowing only querying or allowing fine-tuning. APIs ease use for prescribed use cases as they only require internet connection and possibly a user account. APIs are often commercialized. Deployment benefits include monitoring, last-layer interventions, rate-limiting. Limitations include research and auditing inaccessibility, privacy concerns, and infrastructural reliability issues. API terms of use can be monitored and detected, and users can be blocked or revoked access. However content moderation proves difficult; blocks can easily be bypassed; false positives can block good faith researchers. 20 Last-layer interventions include watermarks after output generation. Limited API access makes research less reliable or presents additional barriers when limiting access to logits, forcing version changes, or obfuscating input and output processing that confound research findings. API deployment also introduces additional privacy challenges when user inputs are collected. APIs also entirely rely on a model hoster and their infrastructure, which may have significant demand and varying reliability.\n\nPlatform sharing of model weights enables broad access to models for local use and modification, controlled by access management (e.g. gating) and visibility (privacy, sensitive content tags) features. Content guidelines and ToU on model sharing platforms are defined at the model rather than at the use level. Releases on platforms can range from fully permissive to research-only (e.g. Cohere Command-R 21 ).\n\nLocal hosting provides full control of the model and inputs to the model, satiates any privacy and sensitive information concerns, and opens many possible research directions. However, this is less monitorable for the deployer, should the user break terms of use. Edge deployment is a type of local hosting that has environment, portability, and privacy benefits.\n\n19 Grok model release via torrent on the X platform\n\n20 A Safe Harbor for AI Evaluation and Red Teaming\n\n21 Command R\n\n\n\n- 2. How do the risks associated with making model weights widely available compare to the risks associated with non-public model weights?\n- a. What, if any, are the risks associated with widely available model weights? How do these risks change, if at all, when the training data or source code associated with fine tuning, pretraining, or deploying a model is simultaneously widely available?\n\nIn most cases, the risks associated with open-weight models are broadly similar to any other part of a software system (with or without AI components), and are similarly context-dependent. 2223 Risks should also be weighed against non-AI systems. 24 Prominent considerations include new capabilities and systems designed specifically for harm.\n\nNew AI use cases arise fast, as seen with the easy creation of photo-realistic images enabled by diffusion models, and more recently video and audio generations. Accelerated development timelines can outpace societal and technical interventions. Both closed development and large-scale release with rapid distribution contribute to risks arising and timeline considerations. Open development often provides additional time and opportunities to question choices, allowing many perspectives to shape systems earlier design decisions 25 . Conversely, closed product development followed by rapid distribution can lead to significant disruptions in important sectors for sometimes uncertain benefits. 2627 Rapid distribution depends on a combination of the availability of model weights, cost and technical difficulty of running a model, availability of a product or API version of an AI system, and broad advertisement and awareness of the AI system. Ensuring that scientific understanding of foundation models and development of new applications of the technology is more gradual helps identify issues and understand technological impacts earlier.\n\nAI systems that are designed specifically for harmful uses, such as harassment or NCII, present distinct challenges from AI systems with dual use risks. Such models should most aptly be compared to malware in the context of traditional software. 28 On model sharing platforms, such models are subject to moderation following content policies. 29 Providing explicit guidance on categories of misuse and learning from existing mechanisms of platform governance to promote cybersecurity will be essential to managing those risks. In some cases, broadly useful models can be fine-tuned or adapted to specifically re-purpose them for such harmful uses. Safety by design practices can address those risks, for example limiting the prevalence of data that would make the models easier to misuse in general pre-training 30 or researching new methods for\n\n22 Open Source Software Security | CISA\n\n23 Artificial Intelligence | CISA\n\n24 On the Societal Impact of Open Foundation Models\n\n25 Deconstructing Design Decisions: Why Courts Must Interrogate Machine Learning and Other Technologies\n\n26 College professors are in 'full-on crisis mode' as they catch one 'ChatGPT plagiarist' after another\n\n27 The impact of ChatGPT on higher education\n\n28 Malware, Phishing, and Ransomware | Cybersecurity and Infrastructure Security Agency CISA\n\n29 Content Policy - Hugging Face\n\n30 StarCoder 2 and The Stack v2: The Next Generation; developers opted to remove known malware from pre-training data to make\n\nit harder to fine-tune for misuse\n\n\n\nsecuring models at the weights level 31 . Additionally, fine-tuning models still requires collating datasets of harmful uses; limiting access to those represents another important layer of risk mitigation.\n\nTaking action after a model release when new risks are identified presents distinct challenges and opportunities for closed and open weights models . API access models can be deleted either voluntarily by the deployer or by court request when found to be in breach of existing regulation, 32 but the cost of doing so when they are tightly integrated in a widely used commercial product may disincentivize organizations from taking actions, and internal scrutiny may not always be sufficient. 33 Conversely, good platform governance can more flexibly provide warnings and documentation, switch adoption to safer models, add technical access management such as gating, and remove models that violate platform policy; drastically reducing availability as a risk mitigation strategy. 34\n\nFor models shared through peer-to-peer systems such as torrents, updates or removals will often rely on alternative communication channels and interventions in deployed contexts.\n\nReleasing training and especially evaluation and testing datasets is usually a net positive from a risk perspective. Information about how a model was trained, where its data may present risks, what mitigation strategies were adopted, and how it was tested, can help to identify model vulnerabilities or limitations and conduct risk-benefit analyses. Challenges include intellectual property and privacy of data subjects. We note several approaches available to effectively manage these tensions 35 .\n\n- b. Could open foundation models reduce equity in rights and safety-impacting AI systems (e.g. healthcare, education, criminal justice, housing, online platforms, etc.)?\n\nMaximally open systems, including training data, weights, and evaluation protocol, can aid in identifying flaws and biases. Insufficient documentation can reduce effectiveness. 36 To maximize benefits, we recommend developers maintain up-to-date information about systems and publicly disclose biases and vulnerabilities as they are identified. The Hugging Face Hub platform supports a collaborative approach to model documentation. 37 We encourage adopters to maintain an inventory of which open systems they are using to help understand relevant risks to study and document. An example could be an inventory of systems used in the federal government. 38\n\n31 Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models\n\n32 FTC settlement includes removal of trained facial recognition model\n\n33 Microsoft engineer sounds alarm on company's AI image generator in letter to FTC\n\n34 Content Policy - Hugging Face\n\n35 Training Data Transparency in AI: Tools, Trends, and Policy Recommendations\n\n37 Model Cards - Hugging Face\n\n36 Black-Box Access is Insufficient for Rigorous AI Audits \ud83d\udcda\n\n38 Hugging Face Response to OMB RFC on federal use of AI\n\n\ud83d\uddf3\n\n\n\n- c. What, if any, risks related to privacy could result from the wide availability of model weights?\n\nResearch has shown trained models can regurgitate private information from their pre-training. This is a sign of improperly trained models, and the behavior is difficult to predict. Weight availability can marginally exacerbate attacks, but is rarely the most efficient method. Model memorization is highly dependent on the model architecture, size, and training procedure. 394041 Mitigations include proper data governance and management practices during training. 42 Open development has exemplified these practices, including via privacy-conscious data sourcing (e.g. BigScience 43 ), and developing pseudonymization and personal data redaction methods for the training data domain (e.g. the StarCoder 44 ). Privacy risks tied to the use of foundation models chiefly arise from API deployment settings that store user data, 45 however, which open-weights models can help mitigate (see our answer to 3b below).\n\nd. Are there novel ways that state or non-state actors could use widely available model weights to create or exacerbate security risks, including but not limited to threats to infrastructure, public health, human and civil rights, democracy, defense, and the economy?\n\nOpen-weight models are unnecessary for state actor threats; should state actors prefer to use FMs, they likely would prefer to build their own capabilities. 46 Narrow AI systems are more tailorable to a given threat. Models with realistic outputs are expensive to use, often more so than human alternatives or narrower systems. For example, the costs of human and AI generated disinformation do not always favor large language model (LLM) use. 47\n\nAll existing models exacerbate risks given the following conditions: models introduce a new attack vector 48 ; models reduce the overall cost of attack through generation or distribution 49 ; malicious actors are unable to build a", + "char_slice": [ + 9430, + 20297 + ] + }, + { + "headings": [ + "## Hugging Face Response to the National Telecommunications and Information Administration Request for Comments on Dual Use Foundation Artificial Intelligence Models with Widely Available Model Weights", + "## About Hugging Face", + "## Questions" + ], + "markdown": "), and developing pseudonymization and personal data redaction methods for the training data domain (e.g. the StarCoder 44 ). Privacy risks tied to the use of foundation models chiefly arise from API deployment settings that store user data, 45 however, which open-weights models can help mitigate (see our answer to 3b below).\n\nd. Are there novel ways that state or non-state actors could use widely available model weights to create or exacerbate security risks, including but not limited to threats to infrastructure, public health, human and civil rights, democracy, defense, and the economy?\n\nOpen-weight models are unnecessary for state actor threats; should state actors prefer to use FMs, they likely would prefer to build their own capabilities. 46 Narrow AI systems are more tailorable to a given threat. Models with realistic outputs are expensive to use, often more so than human alternatives or narrower systems. For example, the costs of human and AI generated disinformation do not always favor large language model (LLM) use. 47\n\nAll existing models exacerbate risks given the following conditions: models introduce a new attack vector 48 ; models reduce the overall cost of attack through generation or distribution 49 ; malicious actors are unable to build a tailored model. When closed-weight models allow higher scale of use via free or compute-subsidized access, relative risks should be reevaluated.\n\nOpen-weight model risks can outweigh those of APIs when the following additional conditions are met: models need fine-tuning in ways unavailable via API; malicious content can easily be automatically identified as malicious (e.g. generated outputs for malware creation or legitimate software 50 ); safeguards are robust to basic evasion (e.g. 'jailbreaking' 51 or traditional content moderation evasion techniques 52 ).\n\n39 Quantifying Memorization Across Neural Language Models\n\n40 Extracting Training Data from Diffusion Models\n\n41 Understanding and Mitigating Copying in Diffusion Models\n\n42 https://github.com/allenai/fm-cheatsheet/blob/main/app/resources/paper.pdf\n\n43 Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources\n\n44 StarCoder: may the source be with you!\n\n45 AIDB: ChatGPT Banned by Italian Authority Due to OpenAI's Lack of Legal Basis for Data Collection and Age Verification\n\n- 46 Yi-34B, Llama 2, and common practices in LLM training: a fact check of the New York Times | EleutherAI Blog\n\n47 How Much Money Could Large Language Models Save Propagandists? | Center for Security and Emerging Technology\n\n48 Google's Bard AI chatbot is vulnerable to use by hackers. So is ChatGPT.\n\n49 The criminal use of ChatGPT - a cautionary tale about large language models | Europol\n\n50 Incident 443: ChatGPT Abused to Develop Malicious Softwares\n\n51 [2307.15043] Universal and Transferable Adversarial Attacks on Aligned Language Models\n\n52 Microsoft CEO responds to AI-generated Taylor Swift fake nude images\n\n\n\n- e. What, if any, risks could result from differences in access to widely available models across different jurisdictions?\n\nCross-jurisdictional legal uncertainty can be detrimental to research collaborations that span sectors, organizations, and geographies. This includes liability per contributor, IP and copyright law, and foundational laws around digital privacy and regulation of misuses such as CSAM.\n\n- f. Which are the most severe, and which the most likely risks described in answering the questions above? How do these set of risks relate to each other, if at all?\n\nIn terms of novelty and scalability of the application, disparate impact of openness, and severity of the harms, non-consensual intimate imagery (NCII) in static images and videos remains the most serious and pressing risk factor of new FMs. Openness, including access to training datasets and training code, aids mitigation by enabling external scrutiny through investigating the highest contributor factors (especially at the dataset level 53 ) and enabling safety by design.\n\nMarginal risk for open-weight models is less certain for pressing risks such as nonconsensual voice cloning; hosted models can be equally harmful 54 with little recourse. The most concerning models fall well below proposed thresholds for dual use, and can easily be independently trained from scratch by medium-resourced actors. Effective policy action should target use cases and provide guidelines for all models across availability.\n\n- 3. What are the benefits of foundation models with model weights that are widely available as compared to fully closed models?\n- a. What benefits do open model weights offer for competition and innovation, both in the AI marketplace and in other areas of the economy? In what ways can open dual-use foundation models enable or enhance scientific research, as well as education/training in computer science and related fields?\n\nOpen-weight models contribute to competition, innovation, and broad understanding of AI systems to support effective and reliable development. In terms of value creation of open technology , the historical and economic impact of Open Source Software (OSS) provides important context for the expected impact of open-weight models. Two main phenomena bear strong relevance: First, the estimated demand-side value of OSS is several orders of magnitude larger ($8.8 trillion) than its supply-side value (>$4 billion). 55 Investment in open technology pays off over 1000x in value in other areas of the economy. Second, adoption enables users to reallocate resources to internal development that supports market diversification and sustainable self-driven growth 56 , which is critical for start-ups and SMEs. OSS trades off capital for human expertise. Recent work has shown that these dynamics are extending to AI;\n\n53 Multimodal datasets: misogyny, pornography, and malignant stereotypes\n\n54 AI Startup ElevenLabs Bans Account Blamed for Biden Audio Deepfake - Bloomberg\n\n55 The Value of Open Source Software\n\n56 Open Source Software and Global Entrepreneurship\n\n\n\ncost efficiency and customizability increasingly outweigh managed solutions, given sufficient company expertise. 57\n\nOpen-weight models are particularly relevant to mitigating market concentration at the data level . High-quality data, both in the form of licensed content, 5859 interaction data, and data uploaded by customers of managed systems, is shaping up to be the next most likely point of monopoly behaviors as training compute costs decrease (see our answer to Q1.a). Open weights models increase the diversity of product offerings from providers and even enable adopters to manage their own on-premise solutions. Companies using open-weights models or systems built on them are in a position to keep their own data value, 60 preserve privacy, and build consortia on their own terms as needed to support better technology development for their applications without having to give up their intellectual property. 61 This is needed to sustain high-quality data creation and fair valuation of the data contribution to AI systems. Additionally, open-weight models have been customized and adapted to run on a greater variety of infrastructure, including individual GPUs and even CPUs, reducing points of market concentration with cloud providers and reducing costs for procurement. 6263\n\nAI foundation models combine innovative ways of building AI systems-and most innovation supporting recent AI product releases has come from open and academic research. For example, the video generation system SORA builds on many public research contributions. 64 Software behind development is largely open source (Pytorch, Tensorflow, HF libraries) and development often requires access to the weights. More convenient 65 and more secure 66 weight storage formats are developed on OSS settings, among many framework improvements. This includes fine-tuning major LLMs and research breakthroughs such as model merging. 67 Open-weight models' role in enhancing existing scientific research is evident in works on topics ranging from knowledge distillation 68 to watermarking 69 . Robust innovation on both performance and safety questions requires scientific rigor and scrutiny, which is enabled by openness and external reproducibility 70 . Supporting that research requires sharing models to validate findings and lower the barrier to entry for participation given the growing resource gap between researchers in different institutions. 71 Recent open-weights releases of foundation models from AI system providers such as Cohere's Command+R, 72 Google's Gemma, 73 and OpenAI's Whisper 3 74 , among others, represent a welcome contribution to this research.\n\n57 16 Changes to the Way Enterprises Are Building and Buying Generative AI | Andreessen Horowitz\n\n58 Exclusive: Reddit in AI content licensing deal with Google | Reuters\n\n- 59 OpenAI In Talks With Dozens of Publishers to License Content - Bloomberg\n- 60 Will StarCoder 2 Win Over Enterprises?\n- 61 SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore\n- 62 GitHub - ggerganov/llama.cpp: LLM inference in C/C++\n- 63 llamafile: bringing LLMs to the people, and to your own computer - Mozilla Innovations\n- 64 Sora Reference Papers - a fffiloni Collection\n- 65 GGUF model saving format\n- 66 GitHub - huggingface/safetensors: Simple, safe way to store and distribute tensors\n- 67 Evolving New Foundation Models: Unleashing the Power of Automating Model Development\n- 68 Knowledge Distillation of Large Language Models\n- 69 A Watermark for Large Language Models\n- 70 EleutherAI: Going Beyond \"Open Science\" to \"Science in the Open\"\n- 71 The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?\n- 72 Command R\n- 73 Gemma: Google introduces new state-of-the-art open models\n- 74 Introducing Whisper, openai/whisper-large-v3 \u00b7 Hugging Face\n\n\n\nOpen Weights", + "char_slice": [ + 19021, + 29011 + ] + }, + { + "headings": [ + "## Hugging Face Response to the National Telecommunications and Information Administration Request for Comments on Dual Use Foundation Artificial Intelligence Models with Widely Available Model Weights", + "## About Hugging Face", + "## Questions" + ], + "markdown": "AI In Talks With Dozens of Publishers to License Content - Bloomberg\n- 60 Will StarCoder 2 Win Over Enterprises?\n- 61 SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore\n- 62 GitHub - ggerganov/llama.cpp: LLM inference in C/C++\n- 63 llamafile: bringing LLMs to the people, and to your own computer - Mozilla Innovations\n- 64 Sora Reference Papers - a fffiloni Collection\n- 65 GGUF model saving format\n- 66 GitHub - huggingface/safetensors: Simple, safe way to store and distribute tensors\n- 67 Evolving New Foundation Models: Unleashing the Power of Automating Model Development\n- 68 Knowledge Distillation of Large Language Models\n- 69 A Watermark for Large Language Models\n- 70 EleutherAI: Going Beyond \"Open Science\" to \"Science in the Open\"\n- 71 The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?\n- 72 Command R\n- 73 Gemma: Google introduces new state-of-the-art open models\n- 74 Introducing Whisper, openai/whisper-large-v3 \u00b7 Hugging Face\n\n\n\nOpen Weights Models and Evaluation\n\nThe availability of open-weights foundation models is of particular importance to the development of the robust and reliable evaluation ecosystem required to properly govern this category of technology. Access to open-weights models serves several distinct purposes for evaluation. First, some categories of evaluation require direct access to model weights . Evaluation of a model's environmental impact and energy consumption, for example, involves running models on controlled hardware. 75 Evaluating model memorization behaviors 76 (related to privacy and intellectual property concerns), covert biases, 77 and data contamination 78 all require varying levels of access from output logits to model activations without additional input or output processing that are not consistently available through API deployment.\n\nSecond, access to model weights makes research and development of new evaluations significantly cheaper . Developing a new evaluation technique typically requires extensive iteration on variations of a system. Being able to run a model locally in a controlled environment makes this process significantly more efficient and cheaper, and has been instrumental for example in our own work on developing bias evaluations in image generation settings before applying them to commercial systems. 79\n\nThird, while model developers should be incentivized to thoroughly evaluate their models, their incentives and priorities may be mis-aligned with those of external stakeholders , especially when models are part of commercial product development. Developer-run evaluations may underplay limitations and harms of a model - often without explicitly setting out to do so; to address this, recent work has called for safe harbor regimes to enable external research 80 , and outlined limitations of 'black-box' access 81 and auditing without sufficient agency. 82 Similarly, self-reported performance numbers on task can often be misleading or over-represent a model's suitability for use. Common issues that contribute to this phenomenon include misleading evaluation framing 83 , choice of metrics (leading to different perceptions of 'emergen capabilities' 84 ), rampant issues of inappropriate benchmark decontamination, 8586 and reporting results on different model versions than the ones deployed in products. 87 Finally, evaluations of foundation model applications in specific domains should be led by experts in these domains rather than by model developers to best reflect the risks and benefits of AI adoption. 88\n\n75 Power Hungry Processing: Watts Driving the Cost of AI Deployment?\n\n76 Quantifying Memorization Across Neural Language Models\n\n77 Dialect prejudice predicts AI decisions about people's character, employability, and criminality\n\n78 Membership Inference Attacks against Language Models via Neighbourhood Comparison - ACL Anthology\n\n79 Stable Bias: Analyzing Societal Representations in Diffusion Models\n\n80 A Safe Harbor for AI Evaluation and Red Teaming\n\n81 Black-Box Access is Insufficient for Rigorous AI Audits\n\n82 AI auditing: The Broken Bus on the Road to AI Accountability\n\n83 GPT-4 and professional benchmarks: the wrong answer to the wrong question\n\n84 Are Emergent Abilities of Large Language Models a Mirage?\n\n85 Investigating Data Contamination for Pre-training Language Models\n\n86 Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs\n\n87 An In-depth Look at Gemini's Language Abilities\n\n88 The Shaky Foundations of Foundation Models in Healthcare\n\n\n\nFourth, results obtained through third-party use of a model's managed API are unreliable without extensive documentation of the model versions, processing steps, or risk mitigation strategies. Model providers can change the underlying version of a model at any time, 89 and do not always disclose sufficient details about post-processing steps that might have an impact on the results, such as automatic edition of prompts. 90 This is particularly relevant since, regardless of any intentional misrepresentation, commonly used evaluations have been shown to be sensitive to specific implementation details that often require extensive evaluation to characterize. 9192\n\nAccess to open-weights models that are substantially similar to those used in commercial products is thus necessary to support a robust evaluation ecosystem , both to guide further development of the technology and to support the development of appropriate standards.\n\n- b. How can making model weights widely available improve the safety, security, and trustworthiness of AI and the robustness of public preparedness against potential AI risks?\n\nOpen-weight models can help in identifying attacks before they can be leveraged at scale; for example, recent work on 'stealing' the last layer was developed with LLaMA 2 93 . Research has also found some adversarial attacks apply to open and closed models, 94 flagging the need to understand limitations of safety techniques. As discussed in 3a, reproducibility increases trust in AI systems by ensuring scientific rigor. Replicable evaluations with full transparency of replicable pieces can resolve misunderstandings or misleading comparisons. 95 Deployers canalso update models as needed, without being locked into hosted model contracts and agreements - which is particularly relevant when model vulnerabilities are identified or when a plausibly more secure model becomes available.\n\nProminent privacy concerns arise from API deployment and user data storage 9697 and training on user data that contains private information. 98 This tendency is particularly concerning as access to private information has been identified as a significantly stronger risk factor than model capacity in some misinformation settings. 99 Open-weight models can mitigate this risk by enabling a more controlled deployment environment and obviating the need to send data to third party organizations. Open practices can enable broader red-teaming and evaluation practices focused on privacy that can help secure all models. 100\n\n89 GPT-4 API general availability and deprecation of older models in the Completions API\n\n90 DALL\u00b7E 3\n\n91 What's going on with the Open LLM Leaderboard?\n\n92 Open LLM Leaderboard: DROP deep dive\n\n93 Stealing Part of a Production Language Model\n\n94 Universal and Transferable Adversarial Attacks on Aligned Language Models\n\n95 What's going on with the Open LLM Leaderboard?\n\n96 March 20 ChatGPT outage: Here's what happened\n\n97 AI Incident database: Incident 513: ChatGPT Banned by Italian Authority Due to OpenAI's Lack of Legal Basis for Data Collection and Age Verification\n\n98 \"It's a Fair Game'', or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents\n\nOn the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial\n\n100 Privacy risks in deployment: Introducing the Chatbot Guardrails Arena 99\n\n\n\n- c. Could open model weights, and in particular the ability to retrain models, help advance equity in rights and safety-impacting AI systems (e.g. healthcare, education, criminal justice, housing, online platforms etc.)?\n\nYes; in addition to examples in 3a, LLaMA's adaptation into Alpaca 101 significantly eased accessibility while maintaining high performance. Some API restrictions prevent equity-advancing research. 102 Open practices, such as open data work, can help address historical biases and prevent 'hate-scaling'. 103\n\n- e. How do these benefits change, if at all, when the training data or the associated source code of the model is simultaneously widely available?\n\nAccess to the training data of an open-weights foundation model provides substantial additional benefits from the perspective of research, rights and governance, and safety. 104 Direct access top the training dataset can support research on model privacy and memorization, 105 risks tied to hate content 106 or NCII generation 107 , intentional and unintentional impacts of data filtering approaches 108 , among others. Interactive or managed access to a dataset can similarly help answer questions about data provenance and privacy 109 and support data governance centered on the rights of data subjects. 110 Support for projects that develop foundation models in fully open settings by making both trained weights and access or extensive documentation of their dataset available can enable more socially responsible and inclusive technology development. 111\n\n- 4. Are there other relevant components of open foundation models that, if simultaneously widely available, would change the risks or benefits presented by widely available model weights? If so, please list them and explain their impact.\n\nIn addition to examining artifacts listed in 1, often-overlooked artifacts include process and documentation. Project governance shapes technical artifacts, including goals, trade-offs, funding, and mechanisms for internal feedback. In fully open development, processes include goal setting, value alignment, and enabling various stakeholders to question upstream choices. The BigScience project provides this level of openness for a multilingual LLM, 112 and the BigCode project 113 took a similar approach to develop a code LLM, including a new form of Governance Card to report relevant information including funding, main development choices and reasoning, handling of cybersecurity risks and personal data, and annotator pay for data augmentation work. 114\n\n101 Alpaca: A Strong, Replicable Instruction-Following Model\n\n102 Dialect prejudice predicts AI decisions about people'", + "char_slice": [ + 27981, + 38680 + ] + }, + { + "headings": [ + "## Hugging Face Response to the National Telecommunications and Information Administration Request for Comments on Dual Use Foundation Artificial Intelligence Models with Widely Available Model Weights", + "## About Hugging Face", + "## Questions" + ], + "markdown": "data governance centered on the rights of data subjects. 110 Support for projects that develop foundation models in fully open settings by making both trained weights and access or extensive documentation of their dataset available can enable more socially responsible and inclusive technology development. 111\n\n- 4. Are there other relevant components of open foundation models that, if simultaneously widely available, would change the risks or benefits presented by widely available model weights? If so, please list them and explain their impact.\n\nIn addition to examining artifacts listed in 1, often-overlooked artifacts include process and documentation. Project governance shapes technical artifacts, including goals, trade-offs, funding, and mechanisms for internal feedback. In fully open development, processes include goal setting, value alignment, and enabling various stakeholders to question upstream choices. The BigScience project provides this level of openness for a multilingual LLM, 112 and the BigCode project 113 took a similar approach to develop a code LLM, including a new form of Governance Card to report relevant information including funding, main development choices and reasoning, handling of cybersecurity risks and personal data, and annotator pay for data augmentation work. 114\n\n101 Alpaca: A Strong, Replicable Instruction-Following Model\n\n102 Dialect prejudice predicts AI decisions about people's character, employability, and criminality\n\n103 On Hate Scaling Laws For Data-Swamps\n\n104 Training Data Transparency in AI: Tools, Trends, and Policy Recommendations\n\nOn Hate Scaling Laws For Data-Swamps\n\n106 105 How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN \ud83d\udcda \ud83d\uddf3\n\n107 Multimodal datasets: misogyny, pornography, and malignant stereotypes\n\n108 A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity\n\n109 The ROOTS Search Tool: Data Transparency for LLMs\n\n110 Data Governance in the Age of Large-Scale Data-Driven Language Technology\n\n111 Social Context of LLMs - the BigScience Approach, Part 3: Data Governance and Representation | Montreal AI Ethics Institute\n\n112 BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model\n\n113 BigCode Project\n\n114 The BigCode Project Governance Card\n\n\n\nDocumentation approaches include model cards 115 and datasheets 116 that can be shared broadly. Accompanying weights with information necessary to assess their usability for a given use case helps ensure they are used in safer contexts. Data quality is broadly recognized to directly affect ML system performance, 117 and dataset limitations, such as toxicity and harmful social biases, are typically reflected in a trained model. 118 Datasets are often the most practical artifact of study to understand the capabilities, risks, and limitations of a model. Sufficient access to a training dataset 119 can help answer relevant questions about a model's characteristics 120 .\n\n- 5. What are the safety-related or broader technical issues involved in managing risks and amplifying benefits of dual-use foundation models with widely available model weights?\n\nWe need significantly more investment in evaluation to foster practices of safety, privacy, and fairness by design. Results from reliable evaluation methodologies and design decisions such as data selection can determine whether a model should be used, and in what conditions.\n\n- a. What model evaluations, if any, can help determine the risks or benefits associated with making weights of a foundation model widely available?\n\nCritical variables for assessing risks and benefits include type of model and design process and data selection. Model evaluations are important in determining usability and use case, and in shaping moderation decisions, but should be complemented with sufficient documentation of both the development process and governance mechanisms (see responses to 3e and 4) and the specific evaluation methodology (see response to 3a). Past moderation decisions on Hugging Face about whether sharing specific models was consistent with our terms of service has depended on all of those factors, as model evaluations by themselves can fail to sufficiently address social dynamics and the context of use and development.\n\nModels trained specifically for sensitive applications, such as identifying personal data (e.g. the StarPII model 121 used for private information redaction in the BigCode project training data) or producing hate speech, 122 require different governance and access to more specific stakeholders. Processes for determining release method will differ by deployer organization and calls for collective decision making boards are unanswered, although discourse is ongoing. 123124\n\n115 Model Cards for Model Reporting\n\n116 Datasheets for Datasets\n\n117 The Effects of Data Quality on Machine Learning Performance\n\n120 The ROOTS Search Tool: Data Transparency for LLMs 119 \ud83d\udcda Training Data Transparency in AI: Tools, Trends, and Policy Recommendations \ud83d\uddf3 118 A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity\n\n121 StarPII release decision\n\n122 Handling and Presenting Harmful Text in NLP Research\n\n123 Publication Norms for Responsible AI - Partnership on AI\n\n124 The Time Is Now to Develop Community Norms for the Release of Foundation Models\n\n\n\n- b. Are there effective ways to create safeguards around FMs, either to ensure that model weights do not become available, or to protect system integrity or human well-being (including privacy) and reduce security risks in those cases where weights are widely available?\n\nPrivacy by design includes steps throughout development including removing unnecessary personal data from a training dataset (see BigScience and BigCode examples in 2c) and following data minimization principles 125 helps reduce risks described in 2c. However, privacy risks also come from the extent to which current deployed systems encourage users to share personal data during their interactions. 126 Ensuring that the product design minimizes instances where this information is stored or transmitted helps prevent damaging breaches.\n\nInterventions at the fine-tuning level, such as in Constitutional AI or reinforcement learning with human feedback to aid instruction following, can help steer a model towards more desirable behaviors. While these interventions have an important role to play in adding friction to misuses of models, they are also limited in their robustness and effectiveness 127 and insufficient to address fundamental issues with models. 128129 Input and output monitoring can also be used to automatically block queries that are incompatible with a model's term of services. However,this approach presents all of the familiar limitations and potential discriminate impacts of automatic content moderation. In particular, the social cost of developing these safeguard datasets 130 and of live content moderation 131 should be considered and minimized.\n\nModel weight encryption is an ongoing research area, with proposals 132 but no wide deployment.\n\n- c. What are the prospects for developing effective safeguards in the future?\n\nIn addition to building on 5b, safety by design, tight scoping of objectives, and intentional and robust data curation all are paths forward. Using AI as safeguards on AI systems at the deployment level may mitigate some risks, can easily compound biases and have disproportionate impact on minority groups using the systems. 133\n\n- d. Are there ways to regain control over and/or restrict access to and/or limit use of weights of an open foundation model that, either inadvertently or purposely, have already become widely available? What are the approximate costs of these methods today? How reliable are they?\n\nClearly defined platform governance frameworks and disclosure mechanisms can most effectively limit availability for systems that require this guardrail due to vulnerabilities or incompatibility with existing regulations. For models found to present critical vulnerabilities, AI can draw on cybersecurity practices to notify users of updates. More research is needed to\n\n125 Artificial intelligence: the CNIL opens a consultation on the creation of datasets for AI\n\n127 Jailbreaking Black Box Large Language Models in Twenty Queries 126 \"It's a Fair Game\", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conv\n\n128 Dialect prejudice predicts AI decisions about people's character, employability, and criminality\n\n130 129 The Male CEO and the Female Assistant: Probing Gender Biases in Text-To-Image Models Through Paired Stereotype Test\n\nInmates in Finland are training AI as part of prison labor - The Verge\n\n131 Inside Facebook's African Sweatshop | TIME\n\n132 NN-Lock: A Lightweight Authorization to Prevent IP Threats of Deep Learning Models | ACM Journal on Emerging Technologies in Computing Systems\n\n133 Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection\n\n\n\nscope what constitutes a critical vulnerability and to standardize evaluations. For models that violate existing regulations or platform content policies, removal from the platform can help limit their spread and mitigate their negative uses. On the Hugging Face platform, we have taken actions to limit the use of models trained in a way that leads them to disproportionately produce harassing text (e.g. GPT4chan 134 ) or models that were trained to reproduce a person's voice or likeness without their consent.\n\n- e. What if any secure storage techniques or practices could be considered necessary to prevent unintentional distribution of model weights?\n\nApproaches can include a trusted research environment that provides researchers full access to model components through an isolated virtual machine (e.g. HathiTrust Data Capsules 135 ) to ensure security, but are resource-intensive in terms of both computational and human support to adapt the environment to specific research needs. FMs and LLMs are notably computationally intensive. Public repositories with access management (such as Hugging Face gated repositories 136 ) can conveniently balance easy access for legitimate researchers and stakeholders and limited release of model weights. In staged or gated releases, researchers may enter legal agreements with terms of use including preventing model weight distribution.\n\n- f. Which components of a foundation model need to be available, and to whom, in order to analyze, evaluate, certify, or red-team the model? To the extent possible, 14 please identify specific evaluations or types of evaluations and the component", + "char_slice": [ + 37246, + 48101 + ] + }, + { + "headings": [ + "## Hugging Face Response to the National Telecommunications and Information Administration Request for Comments on Dual Use Foundation Artificial Intelligence Models with Widely Available Model Weights", + "## About Hugging Face", + "## Questions" + ], + "markdown": "way that leads them to disproportionately produce harassing text (e.g. GPT4chan 134 ) or models that were trained to reproduce a person's voice or likeness without their consent.\n\n- e. What if any secure storage techniques or practices could be considered necessary to prevent unintentional distribution of model weights?\n\nApproaches can include a trusted research environment that provides researchers full access to model components through an isolated virtual machine (e.g. HathiTrust Data Capsules 135 ) to ensure security, but are resource-intensive in terms of both computational and human support to adapt the environment to specific research needs. FMs and LLMs are notably computationally intensive. Public repositories with access management (such as Hugging Face gated repositories 136 ) can conveniently balance easy access for legitimate researchers and stakeholders and limited release of model weights. In staged or gated releases, researchers may enter legal agreements with terms of use including preventing model weight distribution.\n\n- f. Which components of a foundation model need to be available, and to whom, in order to analyze, evaluate, certify, or red-team the model? To the extent possible, 14 please identify specific evaluations or types of evaluations and the component(s) that need to be available for each.\n\nDifferent tasks and level of access needed per task may not be obvious just as risk landscapes for dual-use FMs cannot be fully taxonomized at a given time. While this mapping exercise has not and likely cannot be exhaustively conducted, some examples include bias evaluations listed in 3a and Table 1 of the paper Black-Box Access is Insufficient for Rigorous AI Audits details audit tasks and levels of access needed. 137\n\n- g. Are there means by which to test or verify model weights? What methodology or methodologies exist to audit model weights and/or foundation models?\n\nHugging Face has used a simple model hash to verify whether two models are the same item. For open-weight models, verification is as simple as for any other large files. For APIs, verification is significantly more complex given the stochastic nature of the outputs and complex stack. As outlined above, this can be a source of security vulnerabilities for critical infrastructure that relies on externally managed APIs.\n\n134 ykilcher/gpt-4chan \u00b7 Hugging Face\n\n135 Data Capsules\n\n136 Gated models\n\n137 [2401.14446] Black-Box Access is Insufficient for Rigorous AI Audits\n\n\n\n- 7. What are current or potential voluntary, domestic regulatory, and international mechanisms to manage the risks and maximize the benefits of foundation models with widely available weights? What kind of entities should take a leadership role across which features of governance?\n- a. What security, legal, or other measures can reasonably be employed to reliably prevent wide availability of access to a foundation model's weights, or limit their end use?\n\nPopular measures include terms of licenses with use clauses 138 and gating mechanisms in addition to safeguards enumerated in recent work. 139\n\n- b. How might the wide availability of open foundation model weights facilitate, or else frustrate, government action in AI regulation?\n\nAs noted in 2, 3, and 5, wide availability of open foundation model weights facilitates government action in AI regulation by supporting research on evaluation and standards, fostering more transparent and accountable development, and promoting fair competition. As noted in 2, governance of widely available open-weights models can also require different interventions than API-only models, with different benefits and challenges.\n\n- c. When, if ever, should entities deploying AI disclose to users or the general public that they are using open foundation models either with or without widely available weights?\n\nEntities deploying AI systems should typically disclose that fact, as noted in the AI Bill of Rights. 140 In general, extending the concept of a Software Bill of Materials (SBOM) 141 to AI systems, including identifying which versions of foundation models (open or closed) are being used would have beneficial implications for cybersecurity and reliability.\n\n- d. What role, if any, should the U.S. government take in setting metrics for risk, creating standards for best practices, and/or supporting or restricting the availability of foundation model weights? i. Should other government or non-government bodies, currently existing or not, support the government in this role? Should this vary by sector?\n\nEvaluation presents two main challenges. First, specific risk evaluations is an open research area, and risk priorities vary by organization. Evaluations as a field require significant investment; widely used benchmarks for both performance and risks are criticized for often providing insufficient or even misleading results. 142 Second, model evaluation often depends on the type of models evaluated and type of risk 143 , especially for topics like privacy risks. Continual investment is needed to ensure evaluations reflect risks as models evolve. Sectoral contexts improve metric robustness.\n\n138 OpenRAIL: Towards open and responsible AI licensing frameworks\n\n139 The Gradient of Generative AI Release: Methods and Considerations\n\n140 Blueprint for an AI Bill of Rights | OSTP | The White House\n\n141 Software Bill of Materials (SBOM) | CISA\n\n142 Dialect prejudice predicts AI decisions about people's character, employability, and criminality\n\n143 Evaluating the Social Impact of Generative AI Systems in Systems and Society\n\n\n\nThe U.S. government action can include establishing standards for best practices building on existing work 144 and prioritize requirements of safety by design across both the AI development chain and its deployment environments. 145 General conditions should treat models that are broadly commercialized and broadly shared similarly. Actions should foster a research ecosystem that has sufficient access to the artifacts and infrastructure to conduct research, incentives to share lessons and artifacts for reproducibility, access to broad subject matter experts including social scientists.\n\n- e. What should the role of model hosting services (e.g. HuggingFace, GitHub, etc.) be in making dual-use models with open weights more or less available? Should hosting services host models that do not meet certain safety standards? By whom should those standards be prescribed? 16\n\nAt Hugging Face, use and utility are contextual. Our platform provides features for access management to enable developers to tailor model availability and breadth of access to strike the best balance between benefits and risks. We universally encourage extensive and transparent documentation, mandating documentation for artifacts over 10,000 downloads and providing guides and resources. Content guidelines 146 address specific harms, including breach of consent 147 and use of models for intentional harm. Sharing platforms enable research on systems that may serve different purposes, and in many cases purely as research artifacts.\n\n- f. Should there be different standards for government as opposed to private industry when it comes to sharing model weights of open foundation models or contracting with companies who use them?\n\nGiven its responsibility to stakeholders, the government should meet sufficient procurement requirements to ensure accountability, rights-respecting deployment of technology, and to ensure that AI adoption does provide a net benefit for considered use cases. We appreciate the EO directive to do so and the work of the OMB in this direction.\n\n- g. What should the U.S. prioritize in working with other countries on this topic, and which countries are most important to work with?\n\nInternational collaboration is key, especially among U.S. allies such as the UK, Singapore, and France. Vulnerability and security incident sharing is best raised among allies, especially for those with dedicated AI bodies such as the U.S. and UK AI Safety Institutes.\n\nInternational standards, including among countries with differing values and regulatory approaches, are needed. An urgent need is disclosure standards, outlining what should be disclosed broadly, which can be fine-tuned per jurisdiction to meet requirements for example in\n\n144 NIST Risk Management Framework (RMF)\n\n145 Hugging Face Response to OMB RFC on federal use of AI\n\n146 Announcing our new Content Guidelines and Policy\n\n147 Announcing our new Content Guidelines and Policy\n\n\n\nadhering to local laws. Evaluation standards can help build cross-cultural and multilingual approaches to model safety.\n\nh. What insights from other countries or other societal systems are most useful to consider?\n\nLessons can be drawn from the EU's established AI Office, which maintains standards and operational definitions as AI evolves, and is empowered to make case-by-case definitions about which models might present a systemic risk to reflect the highly contextual nature of risk assessments. The U.S. should designate an accountable organization for metrics, standards, and requirements, and sufficiently resourcing them to keep pace with AI development. The National Institute of Standards and Technology is well positioned and should be well resourced.\n\ni. Are there effective mechanisms or procedures that can be used by the government or companies to make decisions regarding an appropriate degree of availability of model weights in a dual-use foundation model or the dual-use foundation model ecosystem? Are there methods for making effective decisions about open AI deployment that balance both benefits and risks? This may include responsible capability scaling policies, preparedness frameworks, et cetera.\n\nAs discussed in 5a, no definitive processes or standards exist. In addition to discourse referenced in 5a, ongoing cross-sectoral collaborations are sharing best practices. 148 Strict guidelines about safety by design processes can be helpful for developers as they have control over relevant properties, including what data selection, whether the data covers high risk topics such as nuclear devices, and whether the model is trained to produce malware. Limitations in the state of evaluation development makes quantifying model behavior technically difficult.\n\nAs discussed in 5d, models that are trained specifically on PII/malware/hate speech may have different release conditions. Non-regulatory practices such as vulnerability sharing can draw from cybersecurity. In high risk settings, existing controls on information flow should apply. 149 Narrow models for specific domains or applications should use practices from that domain.\n\nj. Are there particular individuals/entities who should or should not have access to open-weight foundation models? If so", + "char_slice": [ + 46801, + 57674 + ] + }, + { + "headings": [ + "## Hugging Face Response to the National Telecommunications and Information Administration Request for Comments on Dual Use Foundation Artificial Intelligence Models with Widely Available Model Weights", + "## About Hugging Face", + "## Questions" + ], + "markdown": "or companies to make decisions regarding an appropriate degree of availability of model weights in a dual-use foundation model or the dual-use foundation model ecosystem? Are there methods for making effective decisions about open AI deployment that balance both benefits and risks? This may include responsible capability scaling policies, preparedness frameworks, et cetera.\n\nAs discussed in 5a, no definitive processes or standards exist. In addition to discourse referenced in 5a, ongoing cross-sectoral collaborations are sharing best practices. 148 Strict guidelines about safety by design processes can be helpful for developers as they have control over relevant properties, including what data selection, whether the data covers high risk topics such as nuclear devices, and whether the model is trained to produce malware. Limitations in the state of evaluation development makes quantifying model behavior technically difficult.\n\nAs discussed in 5d, models that are trained specifically on PII/malware/hate speech may have different release conditions. Non-regulatory practices such as vulnerability sharing can draw from cybersecurity. In high risk settings, existing controls on information flow should apply. 149 Narrow models for specific domains or applications should use practices from that domain.\n\nj. Are there particular individuals/entities who should or should not have access to open-weight foundation models? If so, why and under what circumstances?\n\nResearch institutions, especially those operating in public interest and outside of commercial product development, should not only have access but also infrastructural support. Government resourcing 150 can not only provide financial and technical support, but also ensure that researchers with sufficient breadth of expertise and external stakeholders have access to technical artifacts that may be used in rights-impacting settings. As noted in 3a, third party auditing is most effective when entities who can conditions without developer engagement. 151 Open-weight models can help create these conditions, along with varying\n\n148 PAI's Deployment Guidance for Foundation Model Safety\n\n149 How AI Can Be Regulated Like Nuclear Energy\n\n150 National Artificial Intelligence Research Resource Pilot | NSF\n\n151 Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance\n\n\n\naccess to components such as datasets, including e.g. extensive documentation and query access. 152\n\n- 8. In the face of continually changing technology, and given unforeseen risks and benefits, how can governments, companies, and individuals make decisions or plans today about open foundation models that will be useful in the future?\n- a. How should these potentially competing interests of innovation, competition, and security be addressed or balanced?\n\nPolicy decisions can jointly advance interests of innovation, competition, and security. Work across technological settings (e.g. in the contexts of election systems 153 or biosecurity in protein design 154 ) has shown that a priori antagonistic goals can often both benefit from openness given appropriately tailored approaches. In specific settings where the tensions are harder to resolve, specificity in how these tensions are managed and narrowly tailored policies are particularly important. 155 Across both settings of widely available weights and API-only development, extensive documentation, replicable evaluation, and transparency into design choices of FMs all contribute positively to all of these interests.\n\n- b. Noting that E.O. 14110 grants the Secretary of Commerce the capacity to adapt the threshold, is the amount of computational resources required to build a model, such as the cutoff of 1026 integer or floating-point operations used in the 17 Executive Order, a useful metric for thresholds to mitigate risk in the long-term, particularly for risks associated with wide availability of model weights?\n\nConcerns with thresholds include their robustness, purpose/trigger, enforcement, and verifiability. Methods for determining and proxies for model capability have differing effectiveness. Thresholds along any singular variable, such as training compute, will not give robust insight to model capability and will adapt with time and incentives. Additionally, while model evaluations should eventually play a major role in helping assess proper use and governance of foundation models, our scientific understanding of these artifacts and evaluation systems are not currently mature or robust enough to serve this purpose; and will require significant additional investment and access to open models as outlined in our response to 3a.\n\nFor determining what potential thresholds should trigger, we recommend starting with a voluntary reporting framework. Rather than red-teaming results, disclosure should include what replicable evaluations have been run, including on a spectrum of risks. This will build regulatory capacity and foster a stronger evaluation ecosystem. Thresholds should not be singular hardlines. 156 Levels of mandates should apply by contextual use case and mode of\n\n152\n\nTraining Data Transparency in AI: Tools, Trends, and Policy Recommendations\n\n154 Protein design meets biosecurity | Science\n\n153 Transparency versus security: early analysis of antagonistic requirements \ud83d\udcda\n\n\ud83d\uddf3\n\n155 Openness versus Secrecy in Scientific Research Abstract - PMC\n\n156\n\nDrawing Lines: Tiers for Foundation Models\n\n\n\ndeployment , whether in research or commercial applications. Robust transparency requirements should apply to all commercial products.\n\nDisclosure is critical and lessons can be drawn from cybersecurity to determining levels of disclosure and what can be made public. Meeting a base level requirement should mandate some level of disclosure. Models whose deployment is supported by extensive cloud compute capacity, due to model size or volume of users, should warrant additional scrutiny including cybersecurity audits given the scale of their impact. Robust documentation can also be beneficial for commercial usability and trust.\n\n## Conclusion\n\nAI innovation would not be possible without open access and open science. Openness broadly continues to benefit the field. Since the AI field is a fast-evolving space with many arising considerations, expanding scope to many system artifacts can help to better weigh risks and benefits. Many risks apply across model weight availability and tailored threat models can narrow intervention options. Hugging Face greatly appreciates the opportunity to provide insights and we remain eager to support NTIA and provide our expertise moving forward.", + "char_slice": [ + 56235, + 62924 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2025_Hugging_Face_Response_to_AI_Action_Plan.pdf", + "md_content": "\n\n## Hugging Face Response to Request for Information on the Development of an Artificial Intelligence Action Plan\n\nMarch 2025\n\n## About Hugging Face\n\nHugging Face is a community-driven U.S. company that democratizes responsible artificial intelligence (AI) and machine learning (ML). Our platform is the most widely used for sharing and collaborating on ML systems, fostering open-source and open-science initiatives. We host machine learning models and datasets within an infrastructure that enables efficient data processing, analysis, and research. Additionally, we provide educational resources, courses, and tooling to lower the barrier to AI participation for individuals from all backgrounds.\n\n## Executive Summary\n\nHugging Face appreciates the opportunity to submit comments to the Office of Science and Technology Policy (OSTP) regarding the development of an Artificial Intelligence Action Plan. As a leading AI company committed to democratizing artificial intelligence through open-source and collaborative approaches, we believe that thoughtful policy can support innovation while ensuring that AI development remains competitive, and aligned with American values.\n\nBased on Hugging Face's experience as a leading AI platform serving 7 million users and hosting over 1.5 million public models across diverse domains, we propose an AI Action Plan centered on three interconnected pillars:\n\n- 1. Strengthen Open and Open-Source AI Ecosystems : technical innovation comes from diverse actors across institutions. Support for infrastructure like NAIRR and investment in open science and data allows these contributions to have an additive effect and accelerate robust innovation.\n- 2. Prioritize Efficient and Reliable Adoption : spreading the benefits of the technology by facilitating its adoption along the value chain requires actors across sectors of activity to shape its development. More efficient, modular, and robust AI models require research and infrastructural investments to enable the broadest possible participation and innovation, enabling diffusion of technology across the U.S. economy.\n\n- 3. Promote Security and Standards : Decades of practices in open-source software cybersecurity, information security, and standards can inform safer AI technology. Promoting traceability, disclosure, and interoperability standards will foster a more resilient and robust technology ecosystem.\n\n\n\nFounded in 2016, Hugging Face has become the main platform supporting AI development through open-source contributions spanning language, biology, robotics, law, finance, and beyond. Our recommendations are grounded in practical experience developing tools that have become industry standards while supporting a diverse ecosystem of researchers, startups, and established companies.\n\n## The Role and Progress of Open and Transparent AI\n\nModern AI is built on decades of open research, with commercial giants relying heavily on open source contributions like PyTorch and groundbreaking research on transformer architectures, attention mechanisms, and training methodologies. Further, recent breakthroughs like OLMO2 a relatively small model with fully transparent training methods and data matching OpenAI's o1-mini's performance - and OlympicCoder - an even smaller models exceeding the performance of the latest Claude model on complex coding problems - demonstrate that open research remains a promising path to developing systems that match the performance of commercial models, and can often surpass them especially in terms of efficiency and performance in specific domains. Perhaps most striking is the rapid compression of development timelines-what once required over 100B parameter models just two years ago can now be accomplished with 2B parameter models, suggesting an accelerating path to parity. This trend towards more accessible, efficient, and collaborative AI development shows that open approaches to AI development have a critical role to play in enabling a successful AI strategy that maintains technical leadership and supports more widespread and secure adoption of the technology. We go into further detail in the rest of this document.\n\n## Open Source and Open Science are Key to U.S. Leadership\n\nOpen models, infrastructure, and scientific practices constitute the foundation of AI innovation, allowing a diverse ecosystem of researchers, companies, and developers to build upon shared knowledge. Hugging Face's platform hosts AI models and datasets from both small actors (startups, universities) and large organizations (Microsoft, Google, OpenAI, Meta), demonstrating how open approaches accelerate progress and democratize access to AI capabilities. Our own research and software contributions, such as the Transformers and Diffusers libraries - which have become the standard framework for implementing and sharing\n\n\n\nstate-of-the-art models - exemplify how open science drives the field forward by enabling rapid iteration and validation across the global research community.\n\nThe United States must lead in open-source AI and open science, which can enhance American competitiveness by fostering a robust ecosystem of innovation and ensuring a healthy balance of competition and shared innovation. Research has shown that open technical systems act as force multipliers for economic impact, with an estimated 2000x multiplier effect - meaning $4 billion invested in open systems could potentially generate $8 trillion in value for companies using them. These economic benefits extend to national economies as well. Without any open-source software contributions, the average country would lose 2.2% of its GDP. Open source drove between \u20ac65 and 95 billion of European GDP in 2018 alone-a finding so significant that the European Commission cited it when establishing new rules to streamline the process for open-sourcing government software. This demonstrates how open-source impact translates directly into policy action and economic advantage at the national level, underlining the importance of open source as a public good.\n\nThe commercial adoption of open models is driven by several practical factors. First, cost efficiency - developing AI models from scratch requires significant investment, so leveraging open foundations reduces R&D expenses. Second, customization - organizations can adapt and deploy models specifically tailored to their use cases rather than relying on one-size-fits-all solutions. Third, reduced vendor lock-in-open models give companies greater control over their technology stack and independence from single providers. Finally, open models have caught up to and in certain cases, surpassed the capabilities of closed, proprietary systems: Olympic-Coder, released as part of Hugging Face's Open R1 project, surpasses Claude 3.7, the latest proprietary model from Anthropic, in terms of coding performance. All this is particularly valuable for startups and mid-sized companies, which can access cutting-edge technology without massive infrastructure investments. Banks, pharmaceutical companies and other industries have been adapting open models to specific market needs, demonstrating how open-source foundations support a vibrant commercial ecosystem across the value chain.\n\n## Policy Recommendations:\n\n- \u25cf Enhance National Research Infrastructure : Fully implement and expand the National AI Research Resource (NAIRR) pilot. Hugging Face's active participation in the NAIRR pilot has demonstrated the value of providing researchers with access to computing resources, datasets, and collaborative tools.\n- \u25cf Allocate Public Computing Resources for Open Source : The public should have ways to participate via public AI infrastructure. A way to do this would be to dedicate a portion of publicly-funded computing infrastructure to support open-source AI projects, reducing barriers to innovation for smaller research teams and companies that cannot afford proprietary systems.\n\n- \u25cf Enable access to data for developing open systems: Create sustainable data ecosystems through targeted policies that address the decreasing data commons. Publishers are increasingly signing data licensing deals with proprietary AI model developers to the tune of hundreds of millions of dollars, meaning that quality data acquisition costs are now approaching or even surpassing computational expenses of training frontier models, threatening to lock out small open developers from access to quality data. Support organizations that contribute to public data repositories and streamlined compliance pathways that reduce legal barriers to responsible data sharing.\n- \u25cb Develop Open, High-Quality Datasets : Invest in the creation, curation, and maintenance of robust, representative datasets that can support the next generation of AI research and applications. Expand initiatives like the IBM AI Alliance Trusted Data Catalog and support projects like IDI's AI driven Digitization of the public collections in the Boston Public Library.\n- \u25cb Strengthen Rights-Respecting Data Access Frameworks : Establish clear guidelines for data usage, including standardized protocols for anonymization, consent management, and usage tracking. Support public-private partnerships to create specialized data trusts for high-value domains like healthcare and climate science, ensuring that individuals and organizations maintain appropriate control over their data while enabling innovation.\n- \u25cf Invest in Stakeholder-Driven Innovation : Create and support programs that enable organizations across diverse sectors (healthcare, manufacturing, education) to develop customized AI systems for their specific needs, rather than relying exclusively on general-purpose systems from major providers. This enables broader participation in the AI ecosystem and ensures that the benefits of AI extend throughout the economy.\n- \u25cf Strengthen Scientific Centers of Excellence : Expand NIST's role as a convener for AI experts across academia, industry, and government to share lessons and develop best practices. In particular, the AI Risk Management Framework has played a significant role in identifying stages of AI development and research questions that are critical to ensuring more robust and secure technology deployment for all. The tools we develop at Hugging Face, from model documentation to evaluation libraries, are directly shaped by these questions.\n- \u25cf Support High-Quality Data for Performance and Reliability Evaluation : AI development depends heavily on data, both to train models and to reliably evaluate their progress, strengths, risks and limitations. Fostering greater access to public data in a safe and secure way and ensuring that the evaluation data used to characterize models is sound and evidence-based will accelerate progress in both performance and reliability of the technology.\n\n\n\n\n\n## Prioritize Efficient and Reliable Adoption\n\nSmaller companies and startups face significant barriers to AI adoption due to high costs and limited resources. According to IDC, global AI spending will reach $632 billion in 2028, but these costs remain prohibitive for many small organizations. Meanwhile, energy scarcity presents a growing concern, with the International Energy Agency projecting that data centers' electricity consumption could double from 2022 levels to 1,000 TWh by 2026 - equivalent to Japan's entire electricity demand.\n\nWhile training AI models is energy-intensive, inference - due to its scale and frequency - can ultimately exceed training energy consumption. Ensuring broad AI accessibility requires both hardware optimizations and scalable software frameworks. A range of organizations are developing models tailored to their specific needs, and U.S. leadership in efficiency-focused AI development presents a strategic advantage. The DOE's AI for Energy initiative further supports research into energy-efficient AI, facilitating wider adoption without excessive computational demands.\n\nOpen-source AI tools also bring financial returns, 51% of surveyed companies currently utilizing open-source AI tools report seeing positive ROI, as compared to just 41% of those not using open source. Knowledge sharing and innovation enabling organizations to efficiently use their existing compute resources, such as Hugging Face's Ultra-Scale playbook, help organizations of all sizes efficiently train their own AI models on hardware that they already have access to without requiring massive infrastructure investments. Techniques like quantization, pruning, and model distillation significantly reduce computational requirements while maintaining performance, as demonstrated by Hugging Face's SmolLM and SmolVLM models. Lightweight frameworks further enhance AI deployment in edge devices and resource-constrained environments. Beyond language models, open-source robotics helps lower costs, supporting research and innovation in fields where hardware expenses present a barrier.\n\nReliability is equally important for widespread AI adoption. Organizations need systems that function consistently as intended, particularly in critical domains. Studies have shown error rates as high as 19% in some healthcare AI applications, while general-purpose AI systems often fail to meet the specific needs of specialized sectors, emphasizing the need for tailored solutions. Locally hosted open models are also, by design, more reliable - unlike proprietary models accessed via APIs, locally hosted models provide version stability that is often a requirement for critical use cases.\n\n## Policy Recommendations:\n\n- \u25cf Invest in Efficient AI Research : Support R&D investments toward developing models that maintain high performance while reducing computational and energy requirements for\n\n\n\ninference and other non-training workflows, including methods to transfer large model capabilities to drastically smaller models and models that can run on edge devices.\n\n- \u25cf Support Efficiency Measurements and Benchmarks : Develop federal standards for measuring and reporting AI system efficiency, creating market incentives for technologies that use limited energy resources more effectively. Examples of tools include the Energy Star rating for models.\n- \u25cf Encourage Appropriate Technology Selection : Promote the principle of using the most efficient tool that meets performance requirements (such as resource utilization considerations in the NIST AI RMF), ensuring appropriate resource utilization across the AI ecosystem.\n- \u25cf Support Cross Agency Collaboration of Expertise : Create and continue to support mechanisms for AI expertise to be shared across government agencies, preventing siloed knowledge and enabling consistent approaches to AI policy and procurement.\n- \u25cf Build Evaluation Capacity : Provide training and further develop technical capabilities within organizations, including government agencies, to evaluate AI systems they procure or deploy, ensuring informed decision-making. This requires both technical infrastructure and human expertise to create, choose, and run evaluations, and to interpret evaluation results in context.\n- \u25cf Support Representative Evaluation Infrastructure : Invest in developing evaluation frameworks that accurately represent the variety of potential AI applications and users across different contexts. Developer-run private evaluations are not currently comparable and have limited scrutiny, whereas open evaluations enable broader oversight.\n\n## Promote Security and Standards\n\nAs AI systems become increasingly integrated in all aspects of digital infrastructure, ensuring that they meet appropriate standards of documentation and information security will be essential to ensuring the resilience of the entire ecosystem.\n\nFirst, while AI systems do present some new challenges - particularly in terms of scale - they remain first and foremost digital information systems; so AI security practices should be informed by decades of experience in cybersecurity. These include for example ensuring sufficient documentation of all components of a system, so model cards and training information for AI systems can play a similar role for AI security as the widely recognized Software Bill Of Materials. As for software, recognizing the importance of open source AI as a necessary component of every critical infrastructure sector, and enabling Secure by Design development practices by ensuring federal agencies have capacity to work on every stage of the AI development chain will provide a strong foundation for a more secure digital ecosystem.\n\n\n\nOpen science in AI and open source AI systems are particularly relevant to the long-term security of critical infrastructure in the U.S. Developing open infrastructure to democratize state-of-the-art AI training techniques enables organizations to train their own performant models in controlled environments. 'Maximally open' AI models, which provide full transparency into their training datasets and processes, enable extensive investigation to support security certifications in critical settings. 'Open-weight' models stored in secure formats that can be deployed in air-gapped environments play a critical role for adoption of the technology in the most sensitive environments.\n\nOpen and open source development also has a significant role to play in setting favorable standards. One motivation for developing open source software for both private and government actors has been its ability to shape global practices. Work from international partners such as the UK AI Security Institute or Singapore's Digital Trust Center on open tooling has proven influential. Investing in open source software will support U.S. leadership in global standard-setting processes. In light of these considerations, prioritizing open and open-source AI as a critical component of the U.S. AI security strategy will be crucial for its success.\n\n## Policy Recommendations:\n\n- \u25cf Support Strong Documentation and Transparency Standards: Ensure that all actors using AI systems in security-impacting applications have sufficient information about the system's development and characteristics to proactively identify risks in their deployment context.\n- \u25cf Build Internal Capacity to Develop Secure AI Systems: Facilitate the development of purpose-specific AI systems for government use in cases where the use of general-purpose commercial systems may carry information security risks.\n- \u25cf Develop Public-Private Evaluation Partnerships that Prioritize Open Development: Establish frameworks for government agencies to collaborate with private-sector AI developers on evaluation efforts, combining government use cases with industry technical expertise, with an explicit mandate to prioritize open evaluation data and tooling - building on initiatives like the AI Safety Institute Consortium.\n- \u25cf Extend Existing Cybersecurity Practices to AI Systems: Ensure that cybersecurity and resilience of the software and data transfers underlying AI development and deployment remain a priority, in particular by leveraging decades of experience in securing open source systems.\n\n\n\n## Conclusion\n\nContinuing to lead AI research, development, and deployment in a direction that promotes innovation, efficiency, and competition requires that we act now. Open technical systems, from open science and open source AI, to open software tools, boost economies and accelerate innovation, providing a strong incentive for the U.S. to maintain leadership in open AI development. This would strengthen the growing ecosystem of AI applications that meet the needs of the entire range of actors in the U.S. economy by balancing the complementary relationship between open and proprietary innovation.\n\nBy leveraging the strengths of open-source approaches, prioritizing efficiency, and promoting interoperability standards, we are confident that the AI Action Plan will foster an environment where American companies and researchers thrive while developing technologies that benefit society broadly.\n\nHugging Face remains committed to contributing to this ecosystem through our open-source tools, collaborative platforms, and research initiatives. We appreciate the opportunity to provide these recommendations and look forward to continuing engagement with the OSTP and government stakeholders in shaping effective AI policy.\n\n## Submitted by:\n\nAvijit Ghosh, Yacine Jernite, and Irene Solaiman Hugging Face", + "chunks": [ + { + "headings": [ + "Hugging Face Response to Request for Information on the Development of an Artificial Intelligence Action Plan" + ], + "markdown": "\n\n## Hugging Face Response to Request for Information on the Development of an Artificial Intelligence Action Plan\n\nMarch 2025\n\n## About Hugging Face\n\nHugging Face is a community-driven U.S. company that democratizes responsible artificial intelligence (AI) and machine learning (ML). Our platform is the most widely used for sharing and collaborating on ML systems, fostering open-source and open-science initiatives. We host machine learning models and datasets within an infrastructure that enables efficient data processing, analysis, and research. Additionally, we provide educational resources, courses, and tooling to lower the barrier to AI participation for individuals from all backgrounds.\n\n## Executive Summary\n\nHugging Face appreciates the opportunity to submit comments to the Office of Science and Technology Policy (OSTP) regarding the development of an Artificial Intelligence Action Plan. As a leading AI company committed to democratizing artificial intelligence through open-source and collaborative approaches, we believe that thoughtful policy can support innovation while ensuring that AI development remains competitive, and aligned with American values.\n\nBased on Hugging Face's experience as a leading AI platform serving 7 million users and hosting over 1.5 million public models across diverse domains, we propose an AI Action Plan centered on three interconnected pillars:\n\n- 1. Strengthen Open and Open-Source AI Ecosystems : technical innovation comes from diverse actors across institutions. Support for infrastructure like NAIRR and investment in open science and data allows these contributions to have an additive effect and accelerate robust innovation.\n- 2. Prioritize Efficient and Reliable Adoption : spreading the benefits of the technology by facilitating its adoption along the value chain requires actors across sectors of activity to shape its development. More efficient, modular, and robust AI models require research and infrastructural investments to enable the broadest possible participation and innovation, enabling diffusion of technology across the U.S. economy.\n\n- 3. Promote Security and Standards : Decades of practices in open-source software cybersecurity, information security, and standards can inform safer AI technology. Promoting traceability, disclosure, and interoperability standards will foster a more resilient and robust technology ecosystem.\n\n\n\nFounded in 2016, Hugging Face has become the main platform supporting AI development through open-source contributions spanning language, biology, robotics, law, finance, and beyond. Our recommendations are grounded in practical experience developing tools that have become industry standards while supporting a diverse ecosystem of researchers, startups, and established companies.\n\n## The Role and Progress of Open and Transparent AI\n\nModern AI is built on decades of open research, with commercial giants relying heavily on open source contributions like PyTorch and groundbreaking research on transformer architectures, attention mechanisms, and training methodologies. Further, recent breakthroughs like OLMO2 a relatively small model with fully transparent training methods and data matching OpenAI's o1-mini's performance - and OlympicCoder - an even smaller models exceeding the performance of the latest Claude model on complex coding problems - demonstrate that open research remains a promising path to developing systems that match the performance of commercial models, and can often surpass them especially in terms of efficiency and performance in specific domains. Perhaps most striking is the rapid compression of development timelines-what once required over 100B parameter models just two years ago can now be accomplished with 2B parameter models, suggesting an accelerating path to parity. This trend towards more accessible, efficient, and collaborative AI development shows that open approaches to AI development have a critical role to play in enabling a successful AI strategy that maintains technical leadership and supports more widespread and secure adoption of the technology. We go into further detail in the rest of this document.\n\n## Open Source and Open Science are Key to U.S. Leadership\n\nOpen models, infrastructure, and scientific practices constitute the foundation of AI innovation, allowing a diverse ecosystem of researchers, companies, and developers to build upon shared knowledge. Hugging Face's platform hosts AI models and datasets from both small actors (startups, universities) and large organizations (Microsoft, Google, OpenAI, Meta), demonstrating how open approaches accelerate progress and democratize access to AI capabilities. Our own research and software contributions, such as the Transformers and Diffusers libraries - which have become the standard framework for implementing and sharing\n\n\n\nstate-of-the-art models - exemplify how open science drives the field forward by enabling rapid iteration and validation across the global research community.\n\nThe United States must lead in open-source AI and open science, which can enhance American competitiveness by fostering a robust ecosystem of innovation and ensuring a healthy balance of competition and shared innovation. Research has shown that open technical systems act as force multipliers for economic impact, with an estimated 2000x multiplier effect - meaning $4 billion invested in open systems could potentially generate $8 trillion in value for companies using them. These economic benefits extend to national economies as well. Without any open-source software contributions, the average country would lose 2.2% of its GDP. Open source drove between \u20ac65 and 95 billion of European GDP in 2018 alone-a finding so significant that the European Commission cited it when establishing new rules to streamline the process for open-sourcing government software. This demonstrates how open-source impact translates directly into policy action and economic advantage at the national level, underlining the importance of open source as a public good.\n\nThe commercial adoption of open models is driven by several practical factors. First, cost efficiency - developing AI models from scratch requires significant investment, so leveraging open foundations reduces R&D expenses. Second, customization - organizations can adapt and deploy models specifically tailored to their use cases rather than relying on one-size-fits-all solutions. Third, reduced vendor lock-in-open models give companies greater control over their technology stack and independence from single providers. Finally, open models have caught up to and in certain cases, surpassed the capabilities of closed, proprietary systems: Olympic-Coder, released as part of Hugging Face's Open R1 project, surpasses Claude 3.7, the latest proprietary model from Anthropic, in terms of coding performance. All this is particularly valuable for startups and mid-sized companies, which can access cutting-edge technology without massive infrastructure investments. Banks, pharmaceutical companies and other industries have been adapting open models to specific market needs, demonstrating how open-source foundations support a vibrant commercial ecosystem across the value chain.\n\n## Policy Recommendations:\n\n- \u25cf Enhance National Research Infrastructure : Fully implement and expand the National AI Research Resource (NAIRR) pilot. Hugging Face's active participation in the NAIRR pilot has demonstrated the value of providing researchers with access to computing resources, datasets, and collaborative tools.\n- \u25cf Allocate Public Computing Resources for Open Source : The public should have ways to participate via public AI infrastructure. A way to do this would be to dedicate a portion of publicly-funded computing infrastructure to support open-source AI projects, reducing barriers to innovation for smaller research teams and companies that cannot afford proprietary systems.\n\n- \u25cf Enable access to data for developing open systems: Create sustainable data ecosystems through targeted policies that address the decreasing data commons. Publishers are increasingly signing data licensing deals with proprietary AI model developers to the tune of hundreds of millions of dollars, meaning that quality data acquisition costs are now approaching or even surpassing computational expenses of training frontier models, threatening to lock out small open developers from access to quality data. Support organizations that contribute to public data repositories and streamlined compliance pathways that reduce legal barriers to responsible data sharing.\n- \u25cb Develop Open, High-Quality Datasets : Invest in the creation, curation, and maintenance of robust, representative datasets that can support the next generation of AI research and applications. Expand initiatives like the IBM AI Alliance Trusted Data Catalog and support projects like IDI's AI driven Digitization of the public collections in the Boston Public Library.\n- \u25cb Strengthen Rights-Respecting Data Access Frameworks : Establish clear guidelines for data usage, including standardized protocols for anonymization, consent management, and usage tracking. Support public-private partnerships to create specialized data trusts for high-value domains like healthcare and climate science, ensuring that individuals and organizations maintain appropriate control over their data while enabling innovation.\n- \u25cf Invest in Stakeholder-Driven Innovation : Create and support programs that enable organizations across diverse sectors (healthcare, manufacturing, education) to develop customized AI systems for their specific needs, rather than relying exclusively on general-purpose systems from major providers. This enables broader participation in the AI ecosystem and ensures that the benefits of AI extend throughout the economy.\n- \u25cf Strengthen Scientific Centers of Excellence : Expand NIST's role as a convener for AI experts across academia, industry, and government to share lessons and develop best practices. In particular, the AI Risk Management Framework has played a significant role in identifying stages of AI development and research questions that are critical to ensuring more robust and secure technology deployment for all. The tools we develop at Hugging Face, from model documentation to evaluation libraries, are directly shaped by these questions.\n- \u25cf Support High-Quality Data for Performance and Reliability Evaluation : AI development depends heavily on data, both to train models and to reliably evaluate their progress, strengths, risks and limitations. Fostering greater access to public data in a safe and secure way and ensuring that the evaluation data used to characterize models is sound and evidence-based will accelerate progress in both performance and reliability of the technology.\n\n\n\n\n\n## Prioritize Efficient and Reliable Adoption\n\nSmaller companies and startups face significant barriers to AI adoption due to high costs and limited resources. According to IDC, global AI spending will reach $632 billion in 2028, but these costs remain prohibitive for many small organizations. Meanwhile, energy scarcity presents a growing concern, with the International Energy Agency projecting that data centers' electricity consumption could double from 2022 levels to 1,000 TWh by 2026 - equivalent to", + "char_slice": [ + 0, + 11440 + ] + }, + { + "headings": [ + "## Hugging Face Response to Request for Information on the Development of an Artificial Intelligence Action Plan", + "## About Hugging Face", + "## Executive Summary", + "## The Role and Progress of Open and Transparent AI", + "## Open Source and Open Science are Key to U.S. Leadership", + "## Policy Recommendations:", + "## Prioritize Efficient and Reliable Adoption" + ], + "markdown": "develop best practices. In particular, the AI Risk Management Framework has played a significant role in identifying stages of AI development and research questions that are critical to ensuring more robust and secure technology deployment for all. The tools we develop at Hugging Face, from model documentation to evaluation libraries, are directly shaped by these questions.\n- \u25cf Support High-Quality Data for Performance and Reliability Evaluation : AI development depends heavily on data, both to train models and to reliably evaluate their progress, strengths, risks and limitations. Fostering greater access to public data in a safe and secure way and ensuring that the evaluation data used to characterize models is sound and evidence-based will accelerate progress in both performance and reliability of the technology.\n\n\n\n\n\n## Prioritize Efficient and Reliable Adoption\n\nSmaller companies and startups face significant barriers to AI adoption due to high costs and limited resources. According to IDC, global AI spending will reach $632 billion in 2028, but these costs remain prohibitive for many small organizations. Meanwhile, energy scarcity presents a growing concern, with the International Energy Agency projecting that data centers' electricity consumption could double from 2022 levels to 1,000 TWh by 2026 - equivalent to Japan's entire electricity demand.\n\nWhile training AI models is energy-intensive, inference - due to its scale and frequency - can ultimately exceed training energy consumption. Ensuring broad AI accessibility requires both hardware optimizations and scalable software frameworks. A range of organizations are developing models tailored to their specific needs, and U.S. leadership in efficiency-focused AI development presents a strategic advantage. The DOE's AI for Energy initiative further supports research into energy-efficient AI, facilitating wider adoption without excessive computational demands.\n\nOpen-source AI tools also bring financial returns, 51% of surveyed companies currently utilizing open-source AI tools report seeing positive ROI, as compared to just 41% of those not using open source. Knowledge sharing and innovation enabling organizations to efficiently use their existing compute resources, such as Hugging Face's Ultra-Scale playbook, help organizations of all sizes efficiently train their own AI models on hardware that they already have access to without requiring massive infrastructure investments. Techniques like quantization, pruning, and model distillation significantly reduce computational requirements while maintaining performance, as demonstrated by Hugging Face's SmolLM and SmolVLM models. Lightweight frameworks further enhance AI deployment in edge devices and resource-constrained environments. Beyond language models, open-source robotics helps lower costs, supporting research and innovation in fields where hardware expenses present a barrier.\n\nReliability is equally important for widespread AI adoption. Organizations need systems that function consistently as intended, particularly in critical domains. Studies have shown error rates as high as 19% in some healthcare AI applications, while general-purpose AI systems often fail to meet the specific needs of specialized sectors, emphasizing the need for tailored solutions. Locally hosted open models are also, by design, more reliable - unlike proprietary models accessed via APIs, locally hosted models provide version stability that is often a requirement for critical use cases.\n\n## Policy Recommendations:\n\n- \u25cf Invest in Efficient AI Research : Support R&D investments toward developing models that maintain high performance while reducing computational and energy requirements for\n\n\n\ninference and other non-training workflows, including methods to transfer large model capabilities to drastically smaller models and models that can run on edge devices.\n\n- \u25cf Support Efficiency Measurements and Benchmarks : Develop federal standards for measuring and reporting AI system efficiency, creating market incentives for technologies that use limited energy resources more effectively. Examples of tools include the Energy Star rating for models.\n- \u25cf Encourage Appropriate Technology Selection : Promote the principle of using the most efficient tool that meets performance requirements (such as resource utilization considerations in the NIST AI RMF), ensuring appropriate resource utilization across the AI ecosystem.\n- \u25cf Support Cross Agency Collaboration of Expertise : Create and continue to support mechanisms for AI expertise to be shared across government agencies, preventing siloed knowledge and enabling consistent approaches to AI policy and procurement.\n- \u25cf Build Evaluation Capacity : Provide training and further develop technical capabilities within organizations, including government agencies, to evaluate AI systems they procure or deploy, ensuring informed decision-making. This requires both technical infrastructure and human expertise to create, choose, and run evaluations, and to interpret evaluation results in context.\n- \u25cf Support Representative Evaluation Infrastructure : Invest in developing evaluation frameworks that accurately represent the variety of potential AI applications and users across different contexts. Developer-run private evaluations are not currently comparable and have limited scrutiny, whereas open evaluations enable broader oversight.\n\n## Promote Security and Standards\n\nAs AI systems become increasingly integrated in all aspects of digital infrastructure, ensuring that they meet appropriate standards of documentation and information security will be essential to ensuring the resilience of the entire ecosystem.\n\nFirst, while AI systems do present some new challenges - particularly in terms of scale - they remain first and foremost digital information systems; so AI security practices should be informed by decades of experience in cybersecurity. These include for example ensuring sufficient documentation of all components of a system, so model cards and training information for AI systems can play a similar role for AI security as the widely recognized Software Bill Of Materials. As for software, recognizing the importance of open source AI as a necessary component of every critical infrastructure sector, and enabling Secure by Design development practices by ensuring federal agencies have capacity to work on every stage of the AI development chain will provide a strong foundation for a more secure digital ecosystem.\n\n\n\nOpen science in AI and open source AI systems are particularly relevant to the long-term security of critical infrastructure in the U.S. Developing open infrastructure to democratize state-of-the-art AI training techniques enables organizations to train their own performant models in controlled environments. 'Maximally open' AI models, which provide full transparency into their training datasets and processes, enable extensive investigation to support security certifications in critical settings. 'Open-weight' models stored in secure formats that can be deployed in air-gapped environments play a critical role for adoption of the technology in the most sensitive environments.\n\nOpen and open source development also has a significant role to play in setting favorable standards. One motivation for developing open source software for both private and government actors has been its ability to shape global practices. Work from international partners such as the UK AI Security Institute or Singapore's Digital Trust Center on open tooling has proven influential. Investing in open source software will support U.S. leadership in global standard-setting processes. In light of these considerations, prioritizing open and open-source AI as a critical component of the U.S. AI security strategy will be crucial for its success.\n\n## Policy Recommendations:\n\n- \u25cf Support Strong Documentation and Transparency Standards: Ensure that all actors using AI systems in security-impacting applications have sufficient information about the system's development and characteristics to proactively identify risks in their deployment context.\n- \u25cf Build Internal Capacity to Develop Secure AI Systems: Facilitate the development of purpose-specific AI systems for government use in cases where the use of general-purpose commercial systems may carry information security risks.\n- \u25cf Develop Public-Private Evaluation Partnerships that Prioritize Open Development: Establish frameworks for government agencies to collaborate with private-sector AI developers on evaluation efforts, combining government use cases with industry technical expertise, with an explicit mandate to prioritize open evaluation data and tooling - building on initiatives like the AI Safety Institute Consortium.\n- \u25cf Extend Existing Cybersecurity Practices to AI Systems: Ensure that cybersecurity and resilience of the software and data transfers underlying AI development and deployment remain a priority, in particular by leveraging decades of experience in securing open source systems.\n\n\n\n## Conclusion\n\nContinuing to lead AI research, development, and deployment in a direction that promotes innovation, efficiency, and competition requires that we act now. Open technical systems, from open science and open source AI, to open software tools, boost economies and accelerate innovation, providing a strong incentive for the U.S. to maintain leadership in open AI development. This would strengthen the growing ecosystem of AI applications that meet the needs of the entire range of actors in the U.S. economy by balancing the complementary relationship between open and proprietary innovation.\n\nBy leveraging the strengths of open-source approaches, prioritizing efficiency, and promoting interoperability standards, we are confident that the AI Action Plan will foster an environment where American companies and researchers thrive while developing technologies that benefit society broadly.\n\nHugging Face remains committed to contributing to this ecosystem through our open-source tools, collaborative platforms, and research initiatives. We appreciate the opportunity to provide these recommendations and look forward to continuing engagement with the OSTP and government stakeholders in shaping effective AI policy.\n\n## Submitted by:\n\nAvijit Ghosh, Yacine Jernite, and Irene Solaiman Hugging Face", + "char_slice": [ + 10073, + 20557 + ] + }, + { + "headings": [ + "## Hugging Face Response to Request for Information on the Development of an Artificial Intelligence Action Plan", + "## About Hugging Face", + "## Executive Summary", + "## The Role and Progress of Open and Transparent AI", + "## Open Source and Open Science are Key to U.S. Leadership", + "## Policy Recommendations:", + "## Prioritize Efficient and Reliable Adoption", + "## Policy Recommendations:", + "## Promote Security and Standards", + "## Policy Recommendations:", + "## Conclusion", + "## Submitted by:" + ], + "markdown": "TP and government stakeholders in shaping effective AI policy.\n\n## Submitted by:\n\nAvijit Ghosh, Yacine Jernite, and Irene Solaiman Hugging Face", + "char_slice": [ + 20414, + 20557 + ] + } + ] + }, + { + "url": "https://huggingface.co/datasets/huggingface/policy-docs/resolve/main/2025_UK_Govt_Consultation_Copyright_and_Artificial_Intelligence.pdf", + "md_content": "\n\n## Hugging Face Comments on the Open Consultation on Copyright and Artificial Intelligence\n\nSubmitted to the United Kingdom Intellectual Property Office\n\n## About Hugging Face\n\nHugging Face is a community-driven company based in the U.S. and France, dedicated to democratizing responsible machine learning (ML). Our platform is the most widely used for sharing and collaborating on ML systems, fostering open-source and open-science initiatives. We host machine learning models and datasets within an infrastructure that enables efficient data processing, analysis, and research. Additionally, we provide educational resources, courses, and tooling to lower the barrier to AI participation for individuals from all backgrounds.\n\n## Supporting Open Research and Small Developer Needs in AI Copyright Frameworks\n\nBased on our experience working with the AI community, we broadly support Option 3's approach -a text and data mining exception with rights reservation. Our perspective is informed by direct engagement with open-source developers, researchers, and content creators who use our platform. Below, we outline key considerations drawn from practical experience.\n\n## The Value of Open Datasets and Transparency\n\nThe Hugging Face Datasets Hub hosts thousands of publicly available datasets, each documentation support via Dataset Cards that detail their contents, licensing terms, and limitations. This transparency framework has demonstrated concrete benefits:\n\n- \u25cf For researchers : Ensures data provenance can be verified, facilitating proper attribution of sources. Hugging Face datasets and dataset cards have supported projects such as the Data Provenance Initiative, which in turn have been used to examine fair use arguments as it relates to AI.\n- \u25cf For rights holders : Provides visibility into where and how their content is being used, and assists rights holders in clarifying licensing information.\n- \u25cf For developers : Clear licensing documentation in open datasets by rights holders, in turn, clarifies legal boundaries for dataset usage, reducing uncertainty.\n\n\n\nOur experience shows that open documentation practices enhance trust without impeding innovation . A key example is the BigScience ROOTS dataset , which was used to train the BLOOM model while maintaining transparency regarding data sources and filtering decisions via the ROOTS Search tool. Hugging Face's fully open and well-documented datasets such as FineWeb , have since powered open, benchmark leading models in their size class, such as SmolLM and SmolVLM .\n\nHugging Face transparency initiatives demonstrate that proper documentation does not create undue burdens when appropriate tools are available . In fact, research on Hugging Face datasets has shown that well-documented datasets are also the most downloaded ones95% of download traffic comes from datasets with documentation.\n\nTransparency also helps in compensation. We have observed that while licensing frameworks are being explored in the ecosystem, early attempts at compensating creators have yielded minimal financial returns for most rights holders (with one estimate pointing at payouts of $0.01 per image), while being prohibitively expensive for smaller developers to negotiate such deals. Our experience with transparent datasets and opt-out mechanisms has shown that structured transparency and open-source tooling may provide more practical and economical benefits to both creators and developers than complex licensing schemes that primarily benefit large corporations.\n\nFinally, we recommend that transparency requirements align with existing international frameworks , such as the EU AI Act , which mandates dataset summaries for training large AI models. Public dataset documentation should be encouraged rather than restricted to internal regulatory disclosures.\n\n## Practical Implementation of Opt-Out Mechanisms\n\nAs a platform hosting thousands of datasets, we have gained practical experience implementing rights-holder preferences , demonstrating that effective opt-out mechanisms are feasible:\n\n- \u25cf Dataset search tools : Features like Data Studio (with a dataset viewer and natural language dataset query) enable creators to explore dataset contents and identify potential use of their works.\n- \u25cf Integration with Spawning.ai's opt-out API : Allows content creators to register preferences regarding AI training use that is displayed via a widget on the Hugging Face datasets webpage for datasets with image URLs.\n- \u25cf BigCode's Am I In the Stack app allows creators to remove their github repositories from the database via an easy opt-out tool.\n\n\n\nThese efforts illustrate how platform-based solutions, in partnership with ecosystem actors, benefit small developers and individual researchers while respecting rights holders' requirements, reducing the burden of implementing custom technical solutions. We advocate for standardized opt-out signals (e.g., machine-readable rights expressions) across platforms to prevent fragmentation and ensure accessibility for creators of all sizes.\n\n## Balancing Open Research with Copyright Protections\n\nText and data mining (TDM) exceptions are critical for fostering research and innovation, particularly for:\n\n- \u25cf Scholarly researchers analyzing AI capabilities and risks.\n- \u25cf Open-source developers with limited legal and financial resources.\n- \u25cf Educational institutions training future AI practitioners.\n- \u25cf Community-led research initiatives such as Aya and BLOOM .\n\nTDM and TDM-like exceptions are already practiced in several jurisdictions, such as via DMCA exceptions in the United States and the TDM exception in the EU AI Act. An approach with Option 3 that preserves TDM will keep the UK in line with global jurisprudence and extend the benefits of TDM exceptions for researchers in the UK.\n\nOur BigScience initiative showcased how open, collaborative research can advance AI understanding while maintaining ethical data practices . The data governance working group developed frameworks that balance research needs with copyright protections , demonstrating the feasibility of responsible AI development.\n\nIn summation, to safeguard open research, we recommend:\n\n- \u25cf Ensuring research exemptions remain intact for non-commercial and educational uses .\n- \u25cf Avoiding rigid licensing requirements that could disproportionately impact smaller AI labs, academic institutions, and open-source developers .\n- \u25cf Supporting technical infrastructure that enables researchers to document and disclose dataset provenance while complying with copyright law.\n\n## Conclusion & Recommendations\n\nBased on our platform experience, Option 3 offers the most balanced approach to supporting innovation while upholding creator rights. However, effective implementation is crucial:\n\n- 1. Transparency requirements should build on existing community standards rather than introduce entirely new frameworks.\n\n\n\n- 2. Opt-out mechanisms should be technically supported to be accessible to small creators and developers .\n- 3. Research exceptions must be preserved to ensure AI's continued advancement in education and open science .\n- 4. Proportional compliance mechanisms should be implemented to avoid disadvantaging smaller organizations .\n- 5. Dataset registration and metadata standards should be promoted to reduce ambiguity and facilitate compliance.\n\nHugging Face remains committed to developing tools and best practices that promote responsible AI development while respecting intellectual property rights . We welcome continued engagement on practical implementation strategies.\n\n## Submitted by:\n\nAvijit Ghosh, Yacine Jernite, and Irene Solaiman With input from Bruna Trevelin Hugging Face", + "chunks": [ + { + "headings": [ + "Hugging Face Comments on the Open Consultation on Copyright and Artificial Intelligence" + ], + "markdown": "\n\n## Hugging Face Comments on the Open Consultation on Copyright and Artificial Intelligence\n\nSubmitted to the United Kingdom Intellectual Property Office\n\n## About Hugging Face\n\nHugging Face is a community-driven company based in the U.S. and France, dedicated to democratizing responsible machine learning (ML). Our platform is the most widely used for sharing and collaborating on ML systems, fostering open-source and open-science initiatives. We host machine learning models and datasets within an infrastructure that enables efficient data processing, analysis, and research. Additionally, we provide educational resources, courses, and tooling to lower the barrier to AI participation for individuals from all backgrounds.\n\n## Supporting Open Research and Small Developer Needs in AI Copyright Frameworks\n\nBased on our experience working with the AI community, we broadly support Option 3's approach -a text and data mining exception with rights reservation. Our perspective is informed by direct engagement with open-source developers, researchers, and content creators who use our platform. Below, we outline key considerations drawn from practical experience.\n\n## The Value of Open Datasets and Transparency\n\nThe Hugging Face Datasets Hub hosts thousands of publicly available datasets, each documentation support via Dataset Cards that detail their contents, licensing terms, and limitations. This transparency framework has demonstrated concrete benefits:\n\n- \u25cf For researchers : Ensures data provenance can be verified, facilitating proper attribution of sources. Hugging Face datasets and dataset cards have supported projects such as the Data Provenance Initiative, which in turn have been used to examine fair use arguments as it relates to AI.\n- \u25cf For rights holders : Provides visibility into where and how their content is being used, and assists rights holders in clarifying licensing information.\n- \u25cf For developers : Clear licensing documentation in open datasets by rights holders, in turn, clarifies legal boundaries for dataset usage, reducing uncertainty.\n\n\n\nOur experience shows that open documentation practices enhance trust without impeding innovation . A key example is the BigScience ROOTS dataset , which was used to train the BLOOM model while maintaining transparency regarding data sources and filtering decisions via the ROOTS Search tool. Hugging Face's fully open and well-documented datasets such as FineWeb , have since powered open, benchmark leading models in their size class, such as SmolLM and SmolVLM .\n\nHugging Face transparency initiatives demonstrate that proper documentation does not create undue burdens when appropriate tools are available . In fact, research on Hugging Face datasets has shown that well-documented datasets are also the most downloaded ones95% of download traffic comes from datasets with documentation.\n\nTransparency also helps in compensation. We have observed that while licensing frameworks are being explored in the ecosystem, early attempts at compensating creators have yielded minimal financial returns for most rights holders (with one estimate pointing at payouts of $0.01 per image), while being prohibitively expensive for smaller developers to negotiate such deals. Our experience with transparent datasets and opt-out mechanisms has shown that structured transparency and open-source tooling may provide more practical and economical benefits to both creators and developers than complex licensing schemes that primarily benefit large corporations.\n\nFinally, we recommend that transparency requirements align with existing international frameworks , such as the EU AI Act , which mandates dataset summaries for training large AI models. Public dataset documentation should be encouraged rather than restricted to internal regulatory disclosures.\n\n## Practical Implementation of Opt-Out Mechanisms\n\nAs a platform hosting thousands of datasets, we have gained practical experience implementing rights-holder preferences , demonstrating that effective opt-out mechanisms are feasible:\n\n- \u25cf Dataset search tools : Features like Data Studio (with a dataset viewer and natural language dataset query) enable creators to explore dataset contents and identify potential use of their works.\n- \u25cf Integration with Spawning.ai's opt-out API : Allows content creators to register preferences regarding AI training use that is displayed via a widget on the Hugging Face datasets webpage for datasets with image URLs.\n- \u25cf BigCode's Am I In the Stack app allows creators to remove their github repositories from the database via an easy opt-out tool.\n\n\n\nThese efforts illustrate how platform-based solutions, in partnership with ecosystem actors, benefit small developers and individual researchers while respecting rights holders' requirements, reducing the burden of implementing custom technical solutions. We advocate for standardized opt-out signals (e.g., machine-readable rights expressions) across platforms to prevent fragmentation and ensure accessibility for creators of all sizes.\n\n## Balancing Open Research with Copyright Protections\n\nText and data mining (TDM) exceptions are critical for fostering research and innovation, particularly for:\n\n- \u25cf Scholarly researchers analyzing AI capabilities and risks.\n- \u25cf Open-source developers with limited legal and financial resources.\n- \u25cf Educational institutions training future AI practitioners.\n- \u25cf Community-led research initiatives such as Aya and BLOOM .\n\nTDM and TDM-like exceptions are already practiced in several jurisdictions, such as via DMCA exceptions in the United States and the TDM exception in the EU AI Act. An approach with Option 3 that preserves TDM will keep the UK in line with global jurisprudence and extend the benefits of TDM exceptions for researchers in the UK.\n\nOur BigScience initiative showcased how open, collaborative research can advance AI understanding while maintaining ethical data practices . The data governance working group developed frameworks that balance research needs with copyright protections , demonstrating the feasibility of responsible AI development.\n\nIn summation, to safeguard open research, we recommend:\n\n- \u25cf Ensuring research exemptions remain intact for non-commercial and educational uses .\n- \u25cf Avoiding rigid licensing requirements that could disproportionately impact smaller AI labs, academic institutions, and open-source developers .\n- \u25cf Supporting technical infrastructure that enables researchers to document and disclose dataset provenance while complying with copyright law.\n\n## Conclusion & Recommendations\n\nBased on our platform experience, Option 3 offers the most balanced approach to supporting innovation while upholding creator rights. However, effective implementation is crucial:\n\n- 1. Transparency requirements should build on existing community standards rather than introduce entirely new frameworks.\n\n\n\n- 2. Opt-out mechanisms should be technically supported to be accessible to small creators and developers .\n- 3. Research exceptions must be preserved to ensure AI's continued advancement in education and open science .\n- 4. Proportional compliance mechanisms should be implemented to avoid disadvantaging smaller organizations .\n- 5. Dataset registration and metadata standards should be promoted to reduce ambiguity and facilitate compliance.\n\nHugging Face remains committed to developing tools and best practices that promote responsible AI development while respecting intellectual property rights . We welcome continued engagement on practical implementation strategies.\n\n## Submitted by:\n\nAvijit Ghosh, Yacine Jernite, and Irene Solaiman With input from Bruna Trevelin Hugging Face", + "start_char_id": 0, + "end_char_id": 7761 + } + ] + } +] \ No newline at end of file