pcczc15/Qwen3-VL-2B-Instruct-FT-CyberNH

Cyber-NH LLM Runtime

This directory is intentionally kept outside the CyberNH project tree. By default, CyberNH expects it at:

/Users/chongzhang/CyberNH-LLM

The runtime is wired for the ModelScope model Qwen/Qwen3-VL-2B-Instruct.

If you move this directory again, start project scripts with:

CYBERNH_LLM_DIR=/absolute/path/to/CyberNH-LLM ./S1_Start_llm.sh

Setup

cd /Users/chongzhang/CyberNH-LLM
./setup_modelscope.sh
./download_model.sh

The CyberNH project scripts read /Users/chongzhang/CyberNH-LLM/.env automatically. The dashboard calls an OpenAI-compatible local endpoint at:

http://localhost:8000/v1/chat/completions

Serve

To run only the local LLM service:

cd /Users/chongzhang/CyberNH
./S1_Start_llm.sh

This starts the ModelScope/Transformers OpenAI-compatible endpoint and then opens a CLI chat loop. On Apple Silicon/macOS it uses MPS by default. The first run installs the required inference packages into /Users/chongzhang/CyberNH-LLM/.venv.

CLI chat commands:

/help      show commands
/reset     clear conversation history
/history   show current message count
/exit      quit

By default, if S1_Start_llm.sh starts the model for chat, it stops that model process when you exit the chat. To keep it running after chat exits:

CYBERNH_LLM_KEEP_ALIVE=1 ./S1_Start_llm.sh

To start only the service without entering chat:

CYBERNH_LLM_CHAT=0 ./S1_Start_llm.sh

For a background process with readiness checking:

CYBERNH_LLM_CHAT=0 CYBERNH_LLM_BACKGROUND=1 ./S1_Start_llm.sh

Runtime knobs:

CYBERNH_LLM_DEVICE=auto   # auto, mps, cuda, or cpu
CYBERNH_LLM_DTYPE=auto    # auto, float16, bfloat16, or float32
CYBERNH_LLM_ADAPTER_DIR=/Users/chongzhang/CyberNH-LLM/adapters/system-scenarios-lora

The Qwen3-VL runtime requires transformers>=4.57.0; ./S1_Start_llm.sh / /Users/chongzhang/CyberNH-LLM/serve_transformers.sh install the runtime packages automatically unless CYBERNH_LLM_INSTALL_DEPS=0.

CYBERNH_LLM_ADAPTER_DIR loads the CyberNH system-scenario LoRA adapter. This adapter teaches the model that [System Scenario 1], [System Scenario 2], and [System Scenario 3] stand for the Worker, Senior, and Assistant system prompts.

The lower-level Transformers entrypoint is:

cd /Users/chongzhang/CyberNH-LLM
./serve_transformers.sh

The optional vLLM entrypoint remains at /Users/chongzhang/CyberNH-LLM/serve_vllm.sh for NVIDIA Linux hosts that have vLLM installed.

In LLM mode, Cyber-NH requires this endpoint to be reachable and requires valid JSON decisions from the model.

Start Dashboard

./01_run_sim.sh uses CYBERNH_START_LLM=auto by default. It requires an OpenAI-compatible endpoint at CYBERNH_LLM_BASE_URL; if the endpoint is not reachable and the local LLM server cannot be started, the dashboard exits immediately.

cd /Users/chongzhang/CyberNH
./01_run_sim.sh

To force local LLM startup:

CYBERNH_START_LLM=1 ./01_run_sim.sh

To use a remote OpenAI-compatible endpoint:

CYBERNH_START_LLM=0 \
CYBERNH_LLM_BASE_URL=http://your-llm-host:8000/v1 \
CYBERNH_LLM_API_KEY=EMPTY \
./01_run_sim.sh

Watch LLM I/O

cd /Users/chongzhang/CyberNH
./L2_listen_llm.sh --auto

Only Worker-Agent LLM request and reply blocks are printed.