Automatic Generation of Model and Data Cards: A Step Towards Responsible AI (2024)

Abstract: In an era of model and data proliferation in machine learning/AI especially marked by the rapid advancement of open-sourced technologies, there arises a critical need for standardized consistent documentation. Our work addresses the information incompleteness in current human-written model and data cards. We propose an automated generation approach using Large Language Models (LLMs). Our key contributions include the establishment of CardBench, a comprehensive dataset aggregated from over 4.8k model cards and 1.4k data cards, coupled with the development of the CardGen pipeline comprising a two-step retrieval process. Our approach exhibits enhanced completeness, objectivity, and faithfulness in generated model and data cards, a significant step in responsible AI documentation practices ensuring better accountability and traceability.

## introduction the landscape of artificial intelligence (ai) has undergone a profound transformation with the recent surge in open-sourced models @xcite and datasets @xcite . the trend has been significantly accelerated by the advent of disruptive technologies such as transformers @xcite . since this proliferation of accessible models and datasets can have their applications significantly influence various aspects of society, it becomes increasingly important to underscore the necessity for standardized consistent documentation to communicate their performance characteristics accurately @xcite . ## cardbench dataset cardbench contains 4,829 human-generated model cards and 328 data cards with paper and github references. ## baselines we evaluate our two-step retrieval and generation processes in cardgen against two baselines: we compare our cardgen pipeline with these two baselines in section 8.2. ## evaluation setup we evaluate cardgen on various standard as well as state-of-the-art metrics to measure the faithfulness, relevance, and other aspects of the generation quality. additionally, we incorporate human evaluation for the pipeline to address three key chal- ## conclusion in this study, we introduce a novel task focused on the automatic generation of model cards and data cards. this task is facilitated by the creation of the cardbench dataset, and the development of the cardgen pipeline leveraging state-of-theart llms. the system is designed to assist in the generation of understandable, comprehensive, and consistent models and data cards, thereby providing a valuable contribution to the field of responsible ai.