AutoCatalogAI V2

AutoCatalogAI V2 is a multi-task computer vision model for automatically extracting fashion product attributes and generating structured catalog metadata from a single product image.

The model predicts seven attributes:

Gender
Master category
Subcategory
Article type
Base colour
Season
Usage

It is built on top of openai/clip-vit-base-patch32 and improves the original AutoCatalogAI V1 model through a lightweight colour-feature branch, hierarchical residual connections, controlled fine-tuning, and optional consistency correction.

Model Overview

AutoCatalogAI V2 uses a shared CLIP image encoder followed by multiple task-specific classification heads.

Product Image
      │
      ├── CLIP Vision Encoder
      │         │
      │         ├── Gender Head
      │         ├── Master Category Head
      │         ├── Subcategory Head
      │         ├── Article Type Head
      │         ├── Season Head
      │         └── Usage Head
      │
      └── Colour Statistics Branch
                │
                └── Base Colour Head

The model also uses lightweight hierarchical residual connections:

Master Category → Subcategory
Subcategory → Article Type
Article Type → Season
Article Type → Usage

These connections improve consistency between related product attributes without replacing the independent task heads.

Improvements Over V1

Compared with AutoCatalogAI V1, V2 introduces:

A dedicated colour-feature branch
Hierarchical residual connections between related tasks
Two-stage fine-tuning
Mild capped class balancing for selected tasks
Validation-based safe checkpoint selection
Optional consistency correction
Separate single-image and batch latency benchmarks
Fixed classification-report generation for classes absent from the test split

The V2 training process starts from the proven V1 checkpoint rather than training the complete model from scratch.

Dataset

The model was trained and evaluated using:

ashraq/fashion-product-images-small

Dataset split:

Split	Percentage
Training	70%
Validation	15%
Test	15%

The final test set contains 6,611 product images.

Predicted Attributes

Internal Task Name	Description
`gender`	Product target gender category
`masterCategory`	High-level product category
`subCategory`	Product subcategory
`articleType`	Specific product type
`baseColour`	Primary product colour
`season`	Associated product season
`usage`	Intended product usage

Corrected Test Results

The following results include lightweight consistency correction based on attribute relationships learned from the training split.

Attribute	Accuracy	Macro F1	Weighted F1	Top-3 Accuracy
Gender	91.92%	81.20%	91.71%	99.74%
Master Category	99.53%	84.90%	99.42%	99.94%
Subcategory	96.42%	75.82%	96.21%	99.73%
Article Type	87.64%	66.37%	86.87%	98.15%
Base Colour	69.72%	36.50%	68.55%	90.70%
Season	75.50%	76.80%	75.32%	98.97%
Usage	91.91%	50.52%	91.65%	99.82%

Overall Corrected Metrics

Metric	Result
Average Accuracy	87.52%
Average Macro F1	67.44%
Average Weighted F1	87.10%
Average Top-3 Accuracy	98.15%
Exact-Match Accuracy	40.63%
Test Samples	6,611

Exact-match accuracy requires all seven attributes to be predicted correctly for the same image.

Raw Model Results

The raw results below are calculated before consistency correction.

Metric	Result
Average Accuracy	87.48%
Average Macro F1	67.41%
Average Weighted F1	87.07%
Average Top-3 Accuracy	98.15%
Exact-Match Accuracy	40.46%

The small difference between raw and corrected performance shows that most of the improvement comes directly from the trained model rather than rule-based post-processing.

V1 and V2 Comparison

Metric	V1	V2
Average Accuracy	83.35%	87.52%
Average Macro F1	65.68%	67.44%
Average Weighted F1	84.21%	87.10%
Average Top-3 Accuracy	97.11%	98.15%
Exact-Match Accuracy	27.94%	40.63%
Base-Colour Accuracy	60.11%	69.72%
Season Accuracy	70.70%	75.50%
Usage Accuracy	86.04%	91.91%

V2 improves exact-match accuracy by approximately 12.69 percentage points and base-colour accuracy by approximately 9.61 percentage points over V1.

Training Strategy

Training was performed in two stages.

Stage 1

The CLIP image encoder remained frozen
Newly introduced colour and hierarchy components were trained
Selected classification heads were fine-tuned
A stable V1-equivalent checkpoint was preserved as a fallback

Stage 2

The final CLIP vision layer was unfrozen
All task heads were trained using controlled learning rates
Mild capped class weights were used only for selected imbalanced tasks
The best checkpoint was selected using validation performance and safety constraints

The final checkpoint was selected from:

stage_2_epoch_3

Inference Performance

Latency was measured on an NVIDIA Tesla P100 GPU.

Single-Image Inference

Metric	Latency
Average	6.04 ms
Median / P50	5.99 ms
P95	6.49 ms

Batch Inference

Metric	Latency
Average per image	1.48 ms
Median / P50 per image	1.48 ms
P95 per image	1.50 ms

Latency depends on hardware, batch size, PyTorch version, and runtime configuration.

Repository Files

The repository contains the artifacts required to load and use the custom model:

model.pt
config.json
label_maps.json
consistency_rules.json
metrics.json
README.md

Additional evaluation reports and prediction files may also be included.

Downloading the Model

The complete repository can be downloaded using huggingface_hub:

from huggingface_hub import snapshot_download

model_directory = snapshot_download(
    repo_id="mohsin416/autocatalogai-clip-multitask-v2"
)

print(model_directory)

Loading

AutoCatalogAI V2 uses a custom PyTorch architecture and cannot be loaded directly with:

AutoModelForImageClassification.from_pretrained(...)

Use the AutoCatalogAI V2 project architecture to:

Load config.json
Load label_maps.json
Initialize the custom multi-task model
Load the state dictionary from model.pt
Generate colour features during preprocessing
Optionally apply consistency_rules.json

Example project-level usage:

from autocatalog.inference.predictor import AutoCatalogPredictor

predictor = AutoCatalogPredictor(
    repo_id="mohsin416/autocatalogai-clip-multitask-v2"
)

result = predictor.predict("product_image.jpg")

print(result["prediction"])
print(result["catalog_output"])

The predictor implementation must support the V2 colour branch and hierarchical residual layers.

Example Output

{
  "prediction": {
    "gender": {
      "label": "Women",
      "confidence": 0.8828
    },
    "masterCategory": {
      "label": "Footwear",
      "confidence": 0.9999
    },
    "subCategory": {
      "label": "Flip Flops",
      "confidence": 0.9888
    },
    "articleType": {
      "label": "Flip Flops",
      "confidence": 0.9903
    },
    "baseColour": {
      "label": "Red",
      "confidence": 0.7010
    },
    "season": {
      "label": "Summer",
      "confidence": 0.7783
    },
    "usage": {
      "label": "Casual",
      "confidence": 0.9980
    }
  },
  "catalog_output": {
    "suggested_title": "Women Red Casual Flip Flops",
    "search_tags": [
      "women flip flops",
      "red flip flops",
      "casual flip flops",
      "summer footwear",
      "women casual wear",
      "red casual fashion",
      "flip flops"
    ]
  }
}

Intended Uses

AutoCatalogAI V2 is intended for:

Fashion product catalog automation
Product attribute extraction
E-commerce metadata generation
Product title suggestion
Search-tag generation
Fashion dataset analysis
Multi-task learning demonstrations
Computer vision portfolio and research projects

Out-of-Scope Uses

The model is not intended for:

Identifying or classifying people
Inferring a person's gender from an image
Safety-critical decision-making
Legal, medical, or financial applications
General-purpose object recognition outside the supported fashion domain
Images containing multiple unrelated products without preprocessing

The gender output represents the dataset's product-target category and must not be interpreted as a prediction about a person.

Limitations

Base Colour

Base-colour prediction remains the most difficult task because:

Many products contain multiple colours
Similar labels such as red, pink, peach, and orange overlap visually
Background and lighting affect colour statistics
Dataset colour annotations may be subjective or noisy

The model achieves 69.72% Top-1 and 90.70% Top-3 accuracy for base colour.

Class Imbalance

The gap between average accuracy and macro F1 indicates weaker performance on rare labels. Common classes generally receive stronger predictions than underrepresented classes.

Season and Usage

Season and usage are not always directly visible in an image. These attributes can depend on product descriptions, regional conventions, and seller metadata.

Domain Shift

The model was evaluated on images from the same dataset distribution used during development. Performance may decrease on:

Real-world marketplace photos
Complex backgrounds
Low-resolution images
Occluded products
Multiple-product images
Unusual camera angles
Strong lighting or colour filters

Evaluation Notes

Label mappings were kept consistent across training, validation, and test splits
Test metrics were calculated on 6,611 held-out images
Top-3 accuracy checks whether the correct class appears among the three highest-probability predictions
Exact-match accuracy requires all seven attributes to be correct
Consistency correction uses mappings derived only from the training split
Corrected and raw metrics are reported separately for transparency

Upstream Models and Data

AutoCatalogAI V2 depends on:

Version

Model: AutoCatalogAI V2
Architecture: CLIP ViT-B/32 multi-task classifier
Checkpoint source: AutoCatalogAI V1
Test samples: 6,611
Primary framework: PyTorch + Transformers

Downloads last month: 55

Model tree for mohsin416/autocatalogai-clip-multitask-v2

Base model

openai/clip-vit-base-patch32

Finetuned

(123)

this model

mohsin416
/

autocatalogai-clip-multitask-v2