X-Chat-1.7B

X-Chat-1.7B is a multimodal research model for language-guided object detection. It combines a Qwen3-1.7B language backbone, a SigLIP2 semantic vision encoder, and a Swin-Tiny/DINO detection branch. Given an image and a category name or referring expression, the model generates <DET> tokens and uses their hidden states to predict bounding boxes.

This repository distributes the consolidated X-Chat inference checkpoint. The implementation, configuration, installation instructions, and inference entry points live in the QJ-ROBOTS/X-Chat GitHub repository.

• Code: github.com/QJ-ROBOTS/X-Chat
• Organization: Qianjue Robotics
• Tasks: category-conditioned detection and referring-expression detection
• Checkpoint: full-parameter FP16 model state consolidated from DeepSpeed ZeRO

X-Chat is not a standard transformers architecture. Do not load this file with AutoModel.from_pretrained(). Download pytorch_model.bin and run it through the X-Chat project as shown below.

Checkpoint file

File	Description
pytorch_model.bin	Complete FP16 X-Chat Stage 3 inference state

• Source checkpoint: iter_272000.pth
• State-dict compatibility: 1378 / 1378 keys matched
• Shape mismatches during validation: none
• SHA-256: 4ff60de6162f7415041ad96f0d5edc220930c3dc00a4c937fc36622aaf368339

The 1.7B name refers to the language backbone. The full multimodal model also contains the visual encoders, detector, and cross-modal projection modules.

Requirements

• Linux x86_64
• Python 3.10
• NVIDIA GPU with a CUDA 12.4-compatible driver
• The environment pinned by the X-Chat repository
• Local Qwen3-1.7B, SigLIP2 SO400M Patch14 384, and DINO Swin-Tiny base-model directories

The consolidated checkpoint includes the trained Stage 1, Stage 2, and Stage 3 model state. Separate Stage 1 and Stage 2 checkpoint files are not required for inference. The three base-model directories are still used to construct the architecture, tokenizer, and image processors before the consolidated state is loaded.

Quick start

1. Install X-Chat

git clone https://github.com/QJ-ROBOTS/X-Chat.git
cd X-Chat

uv python install 3.10
uv sync --frozen --extra train --extra gpu --extra demo

2. Prepare runtime directories

Run the following commands from the X-Chat repository root. Keep the trailing slashes on the four project paths.

export XCHAT_ROOT="$(pwd)"
export CODE_DIR="${XCHAT_ROOT}/xchat/"
export DATA_DIR="${XCHAT_ROOT}/datas/"
export INIT_DIR="${XCHAT_ROOT}/inits/"
export WORK_DIR="${XCHAT_ROOT}/wkdrs/"
export HF_HOME="${XCHAT_ROOT}/inits/huggingface"

mkdir -p "${DATA_DIR}" "${INIT_DIR}" "${WORK_DIR}" "${HF_HOME}"

Place the required base assets in this layout:

inits/
├── Qwen3-1.7B/
├── siglip2-so400m-patch14-384/
└── dino-swin-tiny-patch4-window7-224/

See the X-Chat README for the supported versions and complete setup notes. Base assets must be obtained under their respective licenses.

3. Download the checkpoint

mkdir -p "${WORK_DIR}releases/xchat-1.7b"

uv run --frozen --extra demo hf download \
  qj-robot/X-Chat-1.7B pytorch_model.bin \
  --local-dir "${WORK_DIR}releases/xchat-1.7b"

Set the paths used by the inference commands:

export CONFIG_FILE="xchat/xchat/configs/xchat/s3_mixed_finetune/xchat_qwen3_1x7b_siglip2_so400m_p14_384_swin_tiny_dino_gpu16_mixed_finetune.py"
export CHECKPOINT="wkdrs/releases/xchat-1.7b/pytorch_model.bin"
export IMAGE_FILE="path/to/your/image.jpg"

4. Category-conditioned detection

The local CLI expects an ins: prefix followed by one or more comma-separated categories.

uv run --frozen --extra demo --extra gpu xchat-demo "${CONFIG_FILE}" \
  --pth_model "${CHECKPOINT}" \
  --image "${IMAGE_FILE}" \
  --prompt "ins: person, cup" \
  --task_name img_gendet \
  --score_thr 0.3 \
  --output_dir outputs/gendet \
  --cfg-options \
    model.s1_pretrained_pth=None \
    model.s2_pretrained_pth=None

5. Referring-expression detection

uv run --frozen --extra demo --extra gpu xchat-demo "${CONFIG_FILE}" \
  --pth_model "${CHECKPOINT}" \
  --image "${IMAGE_FILE}" \
  --prompt "the blue cup on the left" \
  --task_name img_refdet \
  --score_thr 0.3 \
  --output_dir outputs/refdet \
  --cfg-options \
    model.s1_pretrained_pth=None \
    model.s2_pretrained_pth=None

Rendered predictions are written to the selected output directory. The repository also provides a Gradio interface and COCO-format batch inference.

Supported workflows

Workflow	Task ID	Prompt form	Output
Category-conditioned detection	img_gendet	ins: apple, cup	Requested classes, scores, and boxes
Referring-expression detection	img_refdet	the red apple on the left	Grounded phrase, score, and box

Conversation and segmentation components are present in the research codebase, but this checkpoint is released and documented for the two detection workflows above.

Validation

The consolidated checkpoint was loaded with the public X-Chat Stage 3 configuration while separate Stage 1 and Stage 2 loading was disabled. All 1,378 state entries matched without shape mismatches. A category-conditioned test produced a <DET> response and a rendered bounding-box prediction.

Limitations

• This is a research preview and has not been validated for safety-critical or production robotics decisions.
• Detection quality depends on prompt wording, image domain, resolution, and the selected score threshold.
• Open-vocabulary predictions may miss targets, localize the wrong instance, or produce false positives.
• The current release requires the X-Chat codebase and separately prepared base-model assets; it is not a standalone Transformers checkpoint.
• Hardware requirements below the tested CUDA environment have not been fully characterized.

License and third-party components

Use and redistribution are subject to the X-Chat repository terms and the licenses of Qwen3, SigLIP2, DINO/Swin, datasets, and other third-party components. Review the latest license information in the GitHub repository before use or redistribution.

中文说明

X-Chat-1.7B 是千诀机器人发布的语言引导目标检测研究模型。本模型仓库仅 托管合并后的完整推理权重；代码、配置和推理入口请使用 QJ-ROBOTS/X-Chat。

使用时需要注意：

1. 下载 pytorch_model.bin，不要上传或使用包含优化器状态的原始 iter_272000.pth/ DeepSpeed 目录进行推理；
2. 按 GitHub README 准备 Qwen3、SigLIP2 和 DINO 基础模型目录；
3. 推理命令中设置 model.s1_pretrained_pth=None 和 model.s2_pretrained_pth=None；
4. 类别条件检测使用 img_gendet 和 ins: 提示词，指代表达定位使用 img_refdet。

更多安装、Web Demo、批量推理和训练说明请参阅 X-Chat GitHub README。