Post
1238
Cohere for AI, Argilla, and Hugging Face are collaborating on an Open Science Project to enhance multilingual model evaluations. The project focuses on the widely-used MMLU dataset, which spans 57 subjects like mathematics, computer science, and law. However, existing translations often miss linguistic and cultural nuances, thus embedding biases. 🤔
To address this, they have annotated a subset of the MMLU test set and are inviting global perspectives to review prompts, highlighting cultural specifics and required knowledge. They have mentioned that insights will help shape future multilingual model evaluations, ensuring they are more inclusive and accurate. 🗺️ 📝 🙌
▶️ To get started go to: CohereForAI/MMLU-evaluation
🌍 They also have an Aya Discord server for collaboration with other participants: https://discord.gg/9gVhdfnQMN
To address this, they have annotated a subset of the MMLU test set and are inviting global perspectives to review prompts, highlighting cultural specifics and required knowledge. They have mentioned that insights will help shape future multilingual model evaluations, ensuring they are more inclusive and accurate. 🗺️ 📝 🙌
▶️ To get started go to: CohereForAI/MMLU-evaluation
🌍 They also have an Aya Discord server for collaboration with other participants: https://discord.gg/9gVhdfnQMN