hur03/capturemate-category-classifier-v2-4class

CaptureMate Category Classifier v2 4-Class

OCR text-based screenshot category classifier for CaptureMate.

This model classifies OCR-extracted screenshot text into one of four app categories: schedule, shopping, place, and memo.

unknown samples are excluded from training. Ambiguous or low-confidence predictions should be handled by the application using a confidence threshold.

Model Details

Model Description

This model is a fine-tuned KLUE-RoBERTa based text classification model for CaptureMate, an iOS screenshot organization and action recommendation app.

The model receives OCR text extracted from screenshots and predicts the most relevant screenshot category.

• Developed by: CaptureMate
• Model type: Text classification
• Language(s): Korean, English, mixed OCR text
• License: Not specified
• Finetuned from model: KLUE-RoBERTa base
• Number of labels: 4

Labels

ID	Label	Description
0	schedule	Schedule, reservation, ticket, event, or date-related screenshots
1	shopping	Shopping, product, price, payment, or commerce-related screenshots
2	place	Place, map, restaurant, store, travel, or location-related screenshots
3	memo	Text, article, note, content, or general information screenshots

Uses

Direct Use

Use this model to classify OCR-extracted screenshot text into CaptureMate categories.

Downstream Use

This model can be used inside a screenshot processing pipeline:

Screenshot
-> OCR text extraction
-> Text preprocessing
-> Category classification
-> Category-specific action recommendation

Out-of-Scope Use

This model is not designed to:

• Classify original images directly
• Perform OCR
• Extract structured fields such as dates, prices, addresses, or product names
• Predict an unknown category directly
• Classify content outside the CaptureMate screenshot domain

Bias, Risks, and Limitations

The model is trained on a small CaptureMate-specific screenshot OCR dataset. It may not generalize well to unrelated domains or OCR text from different user behavior patterns.

Known limitations:

• The model always predicts one of the four labels.
• There is no direct unknown output.
• Very short or noisy OCR text can lead to unstable predictions.
• Ambiguous cases such as ticket screenshots, food blog screenshots, or shopping-like memo content may be confused.

Recommendations

Use a confidence threshold at inference time.

Example:

if confidence < 0.6:
    category = "unknown"
else:
    category = predicted_label

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "YOUR_USERNAME/capturemate-category-classifier-v2-4class"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

text = "2025.4.7 티켓 수령 공연 예약 정보"

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=256
)

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.softmax(outputs.logits, dim=-1)
    pred_id = probs.argmax(dim=-1).item()
    confidence = probs[0][pred_id].item()

label = model.config.id2label[pred_id]

print(label, confidence)

Training Details

Training Data

The model was trained on OCR text extracted from screenshot images collected for the CaptureMate project.

Dataset split:

Split	Samples
Train	340
Validation	73
Test	74

unknown samples were removed before creating the final training, validation, and test splits.

Training Procedure

Preprocessing

• OCR text was extracted from screenshots.
• Empty text samples were removed.
• Labels outside the four target categories were excluded.
• Text was tokenized with the KLUE-RoBERTa tokenizer.
• Maximum sequence length: 256 tokens.

Training Hyperparameters

Hyperparameter	Value
Learning rate	1e-5
Epochs	8
Train batch size	16
Eval batch size	16
Warmup ratio	0.1
Weight decay	0.01
Best model metric	macro F1
Seed	42

Evaluation

Testing Data, Factors & Metrics

Testing Data

The test set contains 74 OCR text samples across four categories.

Metrics

The model was evaluated using:

• Accuracy
• Macro F1
• Macro precision
• Macro recall

Macro metrics are important because the dataset is not perfectly balanced across categories.

Results

Metric	Validation	Test
Accuracy	98.63%	94.59%
Macro F1	98.89%	94.55%
Macro Precision	99.07%	93.91%
Macro Recall	98.75%	95.47%

Test Set Per-Class Results

Class	Precision	Recall	F1-score	Support
schedule	90.00%	100.00%	94.74%	9
shopping	96.15%	96.15%	96.15%	26
place	94.74%	100.00%	97.30%	18
memo	94.74%	85.71%	90.00%	21

Confusion Matrix

Rows are true labels, columns are predicted labels.

True \ Pred	schedule	shopping	place	memo
schedule	9	0	0	0
shopping	0	25	0	1
place	0	0	18	0
memo	1	1	1	18

Technical Specifications

Model Architecture and Objective

• Architecture: KLUE-RoBERTa sequence classification
• Objective: Single-label multi-class classification
• Input: OCR text
• Output: One of four CaptureMate categories

Software

• transformers
• datasets
• torch
• scikit-learn

Model Card Authors

CaptureMate

Model Card Contact

CaptureMate project maintainer