Fiptcha

npm  •  GitHub  •  Apache 2.0 License

Fiptcha is a lightweight, CPU-efficient vision-language CAPTCHA solver built for browser automation within tight memory constraints. Fine-tuned from microsoft/Florence-2-base-ft on CAPTCHA challenge data derived from inclusionAI/VenusBench-CAPTCHA and dynamically quantized to INT8 ONNX (~431 MB total across 5 submodels), Fiptcha runs purely via onnxruntime and transformers / tokenizers—requiring no GPU or PyTorch at inference time.

Fiptcha is designed to act as a local CAPTCHA solving engine (<OCR> and <CAPTION_TO_PHRASE_GROUNDING> for text, icon/text click, slider puzzle, drag-end, image rotation, and visual reasoning challenges) on resource-constrained laptops, Apple Silicon devices, and CPU-only containers.

Artifact Backbone Precision / Format Disk Size
ONNX INT8 Submodels (Default) microsoft/Florence-2-base-ft Dynamic per-channel QInt8 ONNX ~431 MB (total)
fiptcha-florence2-merged (Checkpoint) microsoft/Florence-2-base-ft float32 Safetensors ~883.4 MB

Submodel Breakdown

File INT8 Size Role
vision_encoder.onnx 90 MB DaViT vision backbone + 2D/temporal embeddings + projection
embed_tokens.onnx 38 MB Shared token embedding lookup
encoder_model.onnx 42 MB Multimodal BART-style 6-layer encoder
decoder_model.onnx 131 MB Autoregressive 6-layer decoder (initial step)
decoder_with_past_model.onnx 124 MB Autoregressive 6-layer decoder with KV cache

Inputs and outputs

  • Input:
    • A CAPTCHA screenshot / image resized and normalized to 768x768 (CLIPImageProcessor).
    • A Florence-2 task prompt:
      • <OCR> for distorted-text / OCR CAPTCHAs
      • <CAPTION_TO_PHRASE_GROUNDING><target> (e.g., slider puzzle target, icon click target, text click target, drag end target, image rotation target, visual reasoning target)
  • Output:
    • Structured action or grounding output, such as Click(box=(x, y)),Type(content='...') for <OCR> or <loc_x1><loc_y1><loc_x2><loc_y2> bounding-box coordinates (0–999 scale) for <CAPTION_TO_PHRASE_GROUNDING>.

Citation

@misc{fiptcha_2026,
  title  = {Fiptcha: Lightweight CPU-Efficient Local CAPTCHA Solver},
  author = {Figranium},
  year   = {2026},
  url    = {https://huggingface.co/figranium/fiptcha}
}

Model Data

Data used for model training and how the data was processed.

Training Dataset

Fiptcha was fine-tuned on a combined mixture of three public CAPTCHA datasets (augmented with repetition factors and split into train.jsonl and val.jsonl), plus an optional local click-target set:

  • inclusionAI/VenusBench-CAPTCHA (test split, upsampled 15x):
    • OCR CAPTCHAs: Text-based challenges parsed from action_raw and annotated with <OCR> prompts.
    • Visual / Grounding CAPTCHAs: Click and interaction target boxes annotated with <CAPTION_TO_PHRASE_GROUNDING>{prompt} to predict localized target boxes (<loc_x1><loc_y1><loc_x2><loc_y2>).
  • OpenCaptchaWorld/Open_CaptchaWorld (train split, upsampled 10x):
    • Point-Click / Multi-Point CAPTCHAs: Point coordinates (x, y) mapped into padded bounding boxes for visual localization prompts.
    • VQA / Question-Answering CAPTCHAs: General alphanumeric text and label verification questions mapped to a <VQA> prefix and target answers.
  • hammer888/captcha-data (40,000 samples):
    • Distorted text-based images representing classic visual OCR CAPTCHAs, mapped to the <OCR> task prefix.
  • Optional CAPTCHA-X Manifest (upsampled 15x, when present):
    • A specialized local click-target dataset (raw_captchax/manifest.jsonl).

Data Preprocessing

  • Image Preprocessing: Collected images were cached and normalized into train.jsonl / val.jsonl catalogs, resized to 768x768 (resample=3), and normalized with standard ImageNet/CLIP mean ([0.485, 0.456, 0.406]) and std ([0.229, 0.224, 0.225]), producing 577 visual tokens.
  • Prompt Formatting: Each example is formatted as (image, prefix, suffix) pairs across <OCR>, <VQA>, and <CAPTION_TO_PHRASE_GROUNDING> task prefixes.

Implementation Information

Details about the model internals and runtime optimization.

  • Architecture: Florence2ForConditionalGeneration — DaViT vision encoder (depths=[1, 1, 9, 1], dim_embed=[128, 256, 512, 1024]) paired with a 6-layer encoder / 6-layer decoder BART-style language model (d_model=768, 12 attention heads, vocab_size=51,289).
  • Quantization & Export: Exported from PyTorch (attn_implementation="eager") to FP32 ONNX (opset_version=17) with dynamic batch and sequence axes, then quantized using onnxruntime.quantization.quantize_dynamic (per_channel=True, weight_type=QuantType.QInt8, MatMulConstBOnly=True).

Evaluation & Benchmarks

Fiptcha (ONNX INT8) is evaluated on the held-out VenusBench-CAPTCHA validation split (val.jsonl, 44 samples across 6 visual grounding categories and OCR).

VenusBench-CAPTCHA Validation Results (INT8 ONNX · CPU)

Category Samples IoU@0.5 Mean IoU
drag end target 8 100.0% (8/8) 0.903
image rotation target 3 100.0% (3/3) 0.940
text click target 8 87.5% (7/8) 0.641
slider puzzle target 6 83.3% (5/6) 0.715
visual reasoning target 6 83.3% (5/6) 0.644
icon click target 8 37.5% (3/8) 0.346
Grounding Overall 39 79.5% (31/39) 0.669
  • OCR (5 samples): 40.0% (2/5) exact match on Type(content='...').
  • Latency (CPU, greedy num_beams=1): 2.523s median (2.526s mean) per sample using onnxruntime.

Methodology & Setup

  • Task Formulation: Each sample is evaluated with its corresponding <CAPTION_TO_PHRASE_GROUNDING><target> or <OCR> prompt using greedy decoding (num_beams=1, max_new_tokens=64) and skip_special_tokens=False to preserve <loc_...> coordinate tokens.
  • Grounding Metric: For each ground-truth bounding box, we take the maximum IoU against all predicted bounding boxes, average across ground-truth boxes in the sample, and report IoU@0.5 and Mean IoU.

Ethics and Safety

Fiptcha is intended for authorized browser automation testing, accessibility workflows, and research on CAPTCHA robustness. Users are responsible for complying with applicable terms of service and laws when automating interactions with web applications.


Usage and Limitations

Usage

Install the Node.js runtime from github.com/figranium/fiptcha:

npm install fiptcha

To download the quantized ONNX INT8 model weights locally from Hugging Face:

hf download figranium/fiptcha --local-dir ./fiptcha-int8

System Requirements

  • Disk Space: ~431 MB (vision_encoder.onnx, embed_tokens.onnx, encoder_model.onnx, decoder_model.onnx, decoder_with_past_model.onnx, plus tokenizer/processor files)
  • Hardware: Standard x86_64 or ARM64 CPU; no GPU required

Limitations

  • Domain Specificity: Fine-tuned primarily on UI/web CAPTCHA screenshots (VenusBench-CAPTCHA); performance degrades on out-of-domain low-resolution synthetic text CAPTCHAs.
  • Single-Step Inference: Produces a single <OCR> or <CAPTION_TO_PHRASE_GROUNDING> prediction per image; multi-round challenge orchestration is handled by the fiptcha Node.js runtime.
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for figranium/fiptcha

Quantized
(10)
this model

Datasets used to train figranium/fiptcha

Evaluation results