osamashabih/fatima-model-evaluation

Multilingual Prompt Consistency Evaluation

1. Research Question

Does Qwen3-4B-Instruct-2507 respond consistently to equivalent requests written in English, Romanized Hindi, and Hinglish?

This is an exploratory question. The experiment does not assume that a performance gap exists before analyzing the results.

2. Model

• Model: Qwen/Qwen3-4B-Instruct-2507
• Evaluation environment: Google Colab
• GPU: NVIDIA Tesla T4
• Prompt tasks: 10
• Language styles: English, Romanized Hindi, Hinglish
• Planned responses: 30

3. Methodology

Ten everyday tasks were written in three language styles. The model generated one response for each prompt.

The outputs were saved for comparison. Preliminary analysis examined response length and numbered formatting.

4. Preliminary Observations

All 30 prompts completed successfully.

Mean response lengths from the initial analysis:

• English: 66.1 words
• Hinglish: 66.5 words
• Romanized Hindi: 45.0 words

These figures describe response length, not answer quality. Shorter answers are not necessarily worse answers.

5. Limitations

• The experiment contains only 10 task types.
• Equivalent prompts may differ in wording and difficulty.
• The preliminary format score is not a measure of correctness.
• Manual quality-review scores have not yet been completed.
• Results have not established a general language performance gap.

6. Next Steps

1. Review task completion, meaning preservation, and language quality.
2. Check whether scores are applied consistently across languages.
3. Analyze errors and report uncertainty and limitations.
4. Propose data-curation and fine-tuning experiments based on observed errors.

7. Files

• evaluation_prompts.csv: prompts for all task and language combinations.
• model_evaluation_outputs.csv: generated model responses.
• model_evaluation_outputs_scored.csv: preliminary formatting analysis.
• language_comparison.csv: preliminary language-wise summary.
• manual_review_sheet.csv: blank sheet for human evaluation scores.

8. Responsible Reporting

This is a small exploratory evaluation, not a comprehensive benchmark. Conclusions and improvement recommendations will be updated after manual review and validation of the results.