yinggzhang/WeGenBench-Consistency-COT

WeGen-Consistency-CoT

A sharp-eyed multimodal judge for text-to-image prompt consistency.
Give it a generated image and its source prompt; it scores like a reviewer, deducts like a quality inspector, and explains like an annotation expert.

WeGen-Consistency-CoT is a text-to-image consistency evaluation model fully fine-tuned from Qwen3-VL-8B-Instruct. It is designed for long, detailed prompts and focuses on whether a generated image faithfully follows the prompt. The model returns a structured judgment with a 1-10 score, total deduction, overall assessment, and fine-grained deduction reasons.

This model is intended to be used as the consistency judge in the WeGenBench evaluation pipeline. For the benchmark, data format, and full evaluation workflow, please refer to the official repository: WeChatCV/WeGenBench.

Instead of producing only a black-box score, the model decomposes prompt-image mismatches into interpretable error categories, such as missing entities, wrong counts, incorrect attributes, reversed actions, mismatched materials, distorted text, or composition errors. It is useful for automatic text-to-image evaluation, badcase mining, data filtering, regression testing, and human review assistance.

Highlights

• CoT-style deduction grading: The model first gives an overall assessment, then lists each deduction category and reason.
• Prompt-faithfulness first: It evaluates prompt-image consistency rather than aesthetic quality, realism, or artistic preference.
• Fine-grained error taxonomy: It supports categories such as entity, appearance, activity, counting, shape, material, text, and composition.
• Long-prompt ready: Training uses max_length=32768, making the model suitable for complex prompts with many visual constraints.
• Built with ms-swift: The model is trained with swift sft on top of Qwen3-VL-8B-Instruct for reproducible fine-tuning and evaluation.

What It Does

Input:

1. A generated image to be evaluated
2. The original prompt used to generate the image

Output:

Score: 7/10, Total deduction: 3
Overall assessment: The main subject, composition, and overall style are mostly aligned with the prompt, but the biological details of the rear body do not fully match the requested insect abdomen.
Deduction details:
1. appearance: The prompt asks for a bee-like abdomen and tail, but the generated rear body looks more like a furry cat tail instead of a typical insect abdomen. Deduct 3 points.

The model follows a deduction-based consistency evaluation scheme. It starts from a maximum score of 10 and deducts points for explicit, observable prompt constraints that are not correctly reflected in the generated image.

Recommended Prompt

The recommended evaluation prompt is:

Your task is to evaluate text-to-image consistency. Based on the given image and prompt, judge how well the generated image matches the prompt. Give a score from 1 to 10. First provide an overall assessment, then list the deduction details and error reasons. The prompt is:

Append the original generation prompt after this instruction and pass the image together with the text.

Output Format

For automatic parsing and downstream evaluation, we recommend the following format:

Score: <1-10>/10, Total deduction: <0-10>
Overall assessment: <overall consistency judgment>
Deduction details:
1. <category>: <prompt requirement>, <actual image mismatch>. Deduct <points> points.
2. <category>: <prompt requirement>, <actual image mismatch>. Deduct <points> points.

Common deduction categories:

Category	Meaning
entity	Missing entity, wrong entity, or mismatched subject
appearance	Incorrect appearance, color, clothing, or local visual attribute
activity	Wrong action, pose, gesture, or interaction
counting	Incorrect number of objects or people
shape	Incorrect shape, structure, or geometry
material	Incorrect material, texture, or surface quality
text	Missing, distorted, unreadable, or incorrect text
composition	Incorrect viewpoint, layout, subject scale, or framing

Inference with ms-swift

We recommend using TransformersEngine from ms-swift:

from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest

MODEL_PATH = "yinggzhang/WeGenBench-Consistency-COT"
IMAGE_PATH = "example.png"
PROMPT = "A classic BMW sedan is parked in an indoor environment, with smooth body lines..."

judge_prompt = (
    "Your task is to evaluate text-to-image consistency. Based on the given image and prompt, "
    "judge how well the generated image matches the prompt. Give a score from 1 to 10. "
    "First provide an overall assessment, then list the deduction details and error reasons. "
    "The prompt is:"
)

engine = TransformersEngine(MODEL_PATH, max_batch_size=1)
request_config = RequestConfig(max_tokens=1024, temperature=0)

request = InferRequest(
    messages=[{"role": "user", "content": judge_prompt + PROMPT}],
    images=[IMAGE_PATH],
)

response = engine.infer([request], request_config)[0]
print(response.choices[0].message.content)

Best For

• Automatic text-to-image consistency scoring
• Cross-model comparison and leaderboard pre-screening
• Constraint-following analysis for complex prompts
• Badcase attribution and error distribution analysis
• Image-text data cleaning, filtering, and quality regression
• Human annotation assistance

Limitations

• This model evaluates prompt consistency, not general aesthetics.
• It should not be used as the final authority for highly specialized domains such as medical, legal, or safety-critical imagery.
• Scores may still be affected by visual understanding errors, prompt ambiguity, and annotation style.
• CoT-style explanations are useful for review and debugging, but production systems should still parse structured fields and keep human spot checks.
• If the input prompt contains subjective, vague, or conflicting constraints, the model may produce conservative judgments.

Citation

If you find this model useful, please cite:

@misc{liang2026wegenbenchmultidimensionaldiagnosticbenchmark,
      title={WeGenBench: A Multidimensional Diagnostic Benchmark towards Text-to-Image Model Optimization},
      author={Qian Liang and Xiaomin Li and Ying Zhang and Jia Xu and Lihao Ni and Hongrui Li and Jingjing Li and Jing Lyu and Chen Li},
      year={2026},
      eprint={2606.20100},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.20100},
}

Contact Us

For questions, feedback, or collaboration, please contact yinggzhang@tencent.com.