|
|
|
|
|
""" |
|
Created on Fri Oct 11 18:53:59 2024 |
|
|
|
@author: legalchain |
|
""" |
|
default_prompt = """ |
|
Evaluate the given dataset card based on these mandatory criteria: |
|
1. Purpose and Applications: Clearly articulates the dataset's purpose and potential real-world applications. |
|
2. Unique Value: Explicitly states what makes this dataset unique or valuable compared to others. |
|
3. Content Description: Provides detailed information about the actual data content, not just structure. |
|
4. Completeness: Covers dataset size, data distribution, collection methodology, and any preprocessing steps. |
|
5. Limitations and Biases: Discusses any known limitations, biases, or ethical considerations. |
|
6. Human Curation: Shows evidence of human-added insights, context, or valuable information beyond auto-generated content. |
|
|
|
Rate the card from 1 to 5, with half-points allowed (e.g., 2.5): |
|
Dataset Card to Evaluate: {readme_content} |
|
""" |
|
|
|
prompt_enhanced = """Evaluate the given dataset card based on these mandatory criteria: |
|
|
|
# Steps |
|
|
|
1. **Data Origin**: Evaluate the description of where the data originates from. Check if it includes details like data source, collection methods, and any preprocessing steps. |
|
|
|
2. **Usage**: Assess how the model usage is outlined in the card. Consider the intended applications, recommended scenarios, and potential misuse cases. |
|
|
|
3. **Biases**: Review the section addressing biases. Determine if it identifies possible biases in the data or the model's output and how these are mitigated or acknowledged. |
|
|
|
4. **Performance**: Check for clear information on the model's performance, including metrics, benchmarks, and testing conditions. |
|
|
|
5. **Limitations**: Look for a discussion of the model's limitations, noting any restrictions on the types of data or contexts it should be used in. |
|
|
|
6. **Ethics and Safety**: Evaluate any ethical considerations and safety precautions mentioned, focusing on steps taken to ensure responsible use of the model. |
|
|
|
7. **Transparency and Explainability**: Determine if the card includes details that enhance transparency, such as architecture, algorithms used, and explainability measures. |
|
|
|
# Output Format |
|
|
|
Provide a detailed paragraph for each aspect (data origin, usage, biases, etc.) specifying the strengths and weaknesses observed in the model card. Summarize with a concluding statement about the overall quality and completeness of the model card. |
|
|
|
# Examples |
|
|
|
- **Data Origin**: The model card should comprehensively detail the origin of data, including specific datasets used, any licensing or ethical considerations in data collection, and a summary of preprocessing steps (e.g., "The data originates from public health records collected between 2010-2020, after anonymization and normalization processes were applied"). |
|
|
|
- **Biases**: An exemplary model card would identify potential biases by describing the demographic distribution of the training data (e.g., "The data set has a skew towards urban populations, which may affect the model's accuracy in rural settings"). |
|
|
|
# Notes |
|
- Rate the card from 1 to 5, with half-points allowed (e.g., 2.5): |
|
- Create a table with marddown with items name and score at the beginning of your response |
|
- Pay special attention to how each section contributes to understanding the model's behavior and reliability. |
|
- Consider the completeness and clarity of the information presented. |
|
- Document any missing elements or recommendations for improvement. |
|
|
|
Dataset Card to Evaluate: {readme_content} |
|
|
|
|
|
""" |
