npm • GitHub • Apache 2.0 License
Fiptcha is a lightweight, CPU-efficient vision-language CAPTCHA solver built for browser automation within tight memory constraints. Fine-tuned from microsoft/Florence-2-base-ft on CAPTCHA challenge data derived from inclusionAI/VenusBench-CAPTCHA and dynamically quantized to INT8 ONNX (~431 MB total across 5 submodels), Fiptcha runs purely via onnxruntime and transformers / tokenizers—requiring no GPU or PyTorch at inference time.
Fiptcha is designed to act as a local CAPTCHA solving engine (<OCR> and <CAPTION_TO_PHRASE_GROUNDING> for text, icon/text click, slider puzzle, drag-end, image rotation, and visual reasoning challenges) on resource-constrained laptops, Apple Silicon devices, and CPU-only containers.
| Artifact | Backbone | Precision / Format | Disk Size |
|---|---|---|---|
| ONNX INT8 Submodels (Default) | microsoft/Florence-2-base-ft |
Dynamic per-channel QInt8 ONNX |
~431 MB (total) |
fiptcha-florence2-merged (Checkpoint) |
microsoft/Florence-2-base-ft |
float32 Safetensors |
~883.4 MB |
Submodel Breakdown
| File | INT8 Size | Role |
|---|---|---|
vision_encoder.onnx |
90 MB | DaViT vision backbone + 2D/temporal embeddings + projection |
embed_tokens.onnx |
38 MB | Shared token embedding lookup |
encoder_model.onnx |
42 MB | Multimodal BART-style 6-layer encoder |
decoder_model.onnx |
131 MB | Autoregressive 6-layer decoder (initial step) |
decoder_with_past_model.onnx |
124 MB | Autoregressive 6-layer decoder with KV cache |
Inputs and outputs
- Input:
- A CAPTCHA screenshot / image resized and normalized to
768x768(CLIPImageProcessor). - A Florence-2 task prompt:
<OCR>for distorted-text / OCR CAPTCHAs<CAPTION_TO_PHRASE_GROUNDING><target>(e.g.,slider puzzle target,icon click target,text click target,drag end target,image rotation target,visual reasoning target)
- A CAPTCHA screenshot / image resized and normalized to
- Output:
- Structured action or grounding output, such as
Click(box=(x, y)),Type(content='...')for<OCR>or<loc_x1><loc_y1><loc_x2><loc_y2>bounding-box coordinates (0–999 scale) for<CAPTION_TO_PHRASE_GROUNDING>.
- Structured action or grounding output, such as
Citation
@misc{fiptcha_2026,
title = {Fiptcha: Lightweight CPU-Efficient Local CAPTCHA Solver},
author = {Figranium},
year = {2026},
url = {https://huggingface.co/figranium/fiptcha}
}
Model Data
Data used for model training and how the data was processed.
Training Dataset
Fiptcha was fine-tuned on a combined mixture of three public CAPTCHA datasets (augmented with repetition factors and split into train.jsonl and val.jsonl), plus an optional local click-target set:
inclusionAI/VenusBench-CAPTCHA(testsplit, upsampled15x):- OCR CAPTCHAs: Text-based challenges parsed from
action_rawand annotated with<OCR>prompts. - Visual / Grounding CAPTCHAs: Click and interaction target boxes annotated with
<CAPTION_TO_PHRASE_GROUNDING>{prompt}to predict localized target boxes (<loc_x1><loc_y1><loc_x2><loc_y2>).
- OCR CAPTCHAs: Text-based challenges parsed from
OpenCaptchaWorld/Open_CaptchaWorld(trainsplit, upsampled10x):- Point-Click / Multi-Point CAPTCHAs: Point coordinates
(x, y)mapped into padded bounding boxes for visual localization prompts. - VQA / Question-Answering CAPTCHAs: General alphanumeric text and label verification questions mapped to a
<VQA>prefix and target answers.
- Point-Click / Multi-Point CAPTCHAs: Point coordinates
hammer888/captcha-data(40,000samples):- Distorted text-based images representing classic visual OCR CAPTCHAs, mapped to the
<OCR>task prefix.
- Distorted text-based images representing classic visual OCR CAPTCHAs, mapped to the
- Optional CAPTCHA-X Manifest (upsampled
15x, when present):- A specialized local click-target dataset (
raw_captchax/manifest.jsonl).
- A specialized local click-target dataset (
Data Preprocessing
- Image Preprocessing: Collected images were cached and normalized into
train.jsonl/val.jsonlcatalogs, resized to768x768(resample=3), and normalized with standard ImageNet/CLIP mean ([0.485, 0.456, 0.406]) and std ([0.229, 0.224, 0.225]), producing577visual tokens. - Prompt Formatting: Each example is formatted as
(image, prefix, suffix)pairs across<OCR>,<VQA>, and<CAPTION_TO_PHRASE_GROUNDING>task prefixes.
Implementation Information
Details about the model internals and runtime optimization.
- Architecture:
Florence2ForConditionalGeneration— DaViT vision encoder (depths=[1, 1, 9, 1],dim_embed=[128, 256, 512, 1024]) paired with a6-layer encoder /6-layer decoder BART-style language model (d_model=768,12attention heads,vocab_size=51,289). - Quantization & Export: Exported from PyTorch (
attn_implementation="eager") to FP32 ONNX (opset_version=17) with dynamic batch and sequence axes, then quantized usingonnxruntime.quantization.quantize_dynamic(per_channel=True,weight_type=QuantType.QInt8,MatMulConstBOnly=True).
Evaluation & Benchmarks
Fiptcha (ONNX INT8) is evaluated on the held-out VenusBench-CAPTCHA validation split (val.jsonl, 44 samples across 6 visual grounding categories and OCR).
VenusBench-CAPTCHA Validation Results (INT8 ONNX · CPU)
| Category | Samples | IoU@0.5 | Mean IoU |
|---|---|---|---|
drag end target |
8 | 100.0% (8/8) | 0.903 |
image rotation target |
3 | 100.0% (3/3) | 0.940 |
text click target |
8 | 87.5% (7/8) | 0.641 |
slider puzzle target |
6 | 83.3% (5/6) | 0.715 |
visual reasoning target |
6 | 83.3% (5/6) | 0.644 |
icon click target |
8 | 37.5% (3/8) | 0.346 |
| Grounding Overall | 39 | 79.5% (31/39) | 0.669 |
- OCR (
5samples): 40.0% (2/5) exact match onType(content='...'). - Latency (CPU, greedy
num_beams=1): 2.523s median (2.526s mean) per sample usingonnxruntime.
Methodology & Setup
- Task Formulation: Each sample is evaluated with its corresponding
<CAPTION_TO_PHRASE_GROUNDING><target>or<OCR>prompt using greedy decoding (num_beams=1,max_new_tokens=64) andskip_special_tokens=Falseto preserve<loc_...>coordinate tokens. - Grounding Metric: For each ground-truth bounding box, we take the maximum IoU against all predicted bounding boxes, average across ground-truth boxes in the sample, and report
IoU@0.5andMean IoU.
Ethics and Safety
Fiptcha is intended for authorized browser automation testing, accessibility workflows, and research on CAPTCHA robustness. Users are responsible for complying with applicable terms of service and laws when automating interactions with web applications.
Usage and Limitations
Usage
Install the Node.js runtime from github.com/figranium/fiptcha:
npm install fiptcha
To download the quantized ONNX INT8 model weights locally from Hugging Face:
hf download figranium/fiptcha --local-dir ./fiptcha-int8
System Requirements
- Disk Space: ~431 MB (
vision_encoder.onnx,embed_tokens.onnx,encoder_model.onnx,decoder_model.onnx,decoder_with_past_model.onnx, plus tokenizer/processor files) - Hardware: Standard x86_64 or ARM64 CPU; no GPU required
Limitations
- Domain Specificity: Fine-tuned primarily on UI/web CAPTCHA screenshots (
VenusBench-CAPTCHA); performance degrades on out-of-domain low-resolution synthetic text CAPTCHAs. - Single-Step Inference: Produces a single
<OCR>or<CAPTION_TO_PHRASE_GROUNDING>prediction per image; multi-round challenge orchestration is handled by thefiptchaNode.js runtime.
- Downloads last month
- -
Model tree for figranium/fiptcha
Base model
microsoft/Florence-2-base-ftDatasets used to train figranium/fiptcha
OpenCaptchaWorld/Open_CaptchaWorld
hammer888/captcha-data
Evaluation results
- IoU@0.5 on VenusBench-CAPTCHA (val.jsonl, 39 grounding samples)validation set self-reported79.500
- Mean IoU on VenusBench-CAPTCHA (val.jsonl, 39 grounding samples)validation set self-reported0.669