LongCap usage

Use Linux x86-64, Python 3.12, a C/C++ compiler, an NVIDIA driver compatible with CUDA 13.0, and an FP8-capable GPU with sufficient memory, such as an H100 80 GB.

Download and install

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install huggingface_hub

Download the code and the checkpoint for your task. No Hugging Face login is required.

# Captioning
hf download longcap/LongCap_ICLR --local-dir LongCap_ICLR --include 'code/*' 'LongCap/*'

# Diarization and transcription
hf download longcap/LongCap_ICLR --local-dir LongCap_ICLR --include 'code/*' 'LongCap_Diar/*'

Install from the downloaded code/ directory:

cd LongCap_ICLR/code
python -m pip install --only-binary=:all: --require-hashes -r requirements/bootstrap.lock
python -m pip install --only-binary=:all: --require-hashes -r requirements/server.lock
python -m pip install --no-build-isolation --no-deps .

For a client-only installation, use requirements/client.lock instead of requirements/server.lock and omit --no-build-isolation from the final command. Run the following commands from code/ with the same Python environment activated in each terminal.

Captioning

Start the server:

longcap verify-model --model ../LongCap
longcap serve --model ../LongCap --profile long

In another terminal:

longcap caption --audio /path/to/recording.wav --model ../LongCap --output outputs/caption

Read the captions from outputs/caption/predictions/predictions.jsonl. Event start and end times are in seconds from the start of the recording.

Diarization and transcription

Stop the previous server before reusing its port, then start the diarization server:

longcap verify-model --model ../LongCap_Diar
longcap serve --model ../LongCap_Diar --profile long

In another terminal:

longcap diarize --audio /path/to/recording.wav --model ../LongCap_Diar --frequency-penalty 0.05 --output outputs/diarization

Read the segments from outputs/diarization/predictions/diarization.jsonl. Each segment contains start/end seconds, a speaker label, and text.

Docker server

With Docker and NVIDIA Container Toolkit installed, run from code/:

docker build -f docker/Dockerfile -t longcap:review .
mkdir -p cache
docker run --rm --gpus all --shm-size=16g -p 127.0.0.1:20000:20000 \
  -v "$PWD/../LongCap:/models/longcap-35b-a3b:ro" \
  -v "$PWD/cache:/cache" longcap:review

For diarization, replace ../LongCap with ../LongCap_Diar in the model mount. Use the client commands above from the host. The host client and container server must use the same checkpoint. Stop any other server using port 20000 first.

Batch input and saved runs

Use WAV or FLAC input. To process multiple recordings, replace --audio /path/to/recording.wav with --manifest recordings.jsonl:

{"key": "recording_001", "audio_path": "audio/recording_001.wav", "duration": 73.5}

Put one recording on each line. Audio paths are relative to the manifest; duration is the number of seconds to process and must not exceed the file length.

Rerun the same command to resume saved work. Use a new output directory when changing inputs, tasks, models, or decoding settings. Exit code 2 means output is incomplete; inspect predictions/conversion.jsonl and raw/client.json in the output directory. To regenerate predictions from saved replies, run longcap convert --run /path/to/output.

Use longcap --help or longcap <command> --help for more options.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support