unsloth

/

diffusiongemma-26B-A4B-it-GGUF

Like 395

Follow

Unsloth AI 35.2k

I have tried it on 2 Google accounts, in 5 sessions so far. Shall rerun the rest with /usr/bin/time -v cmake --build build -j --config Release --target llama-diffusion-cli to show the logs...

OOM-ed once again with some +12 GB RAM spike at [ 38%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmf-instance-ncols_15.cu.o

Note to self:
Try:

• -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF and -DCMAKE_CXX_FLAGS_RELEASE="-O2 -g0"
• -j 2 (limits the concurrent compilation threads)

The latter helped to compile without OOM:

CMAKE_BUILD_TYPE=Release
-- ccache found, compilation results will be cached. Disable with GGML_CCACHE=OFF.
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native 
-- CUDA Toolkit found
-- Using CMAKE_CUDA_ARCHITECTURES=75-real CMAKE_CUDA_ARCHITECTURES_NATIVE=75-real
-- CUDA host compiler is GNU 11.4.0
-- Including CUDA backend
-- ggml version: 0.14.0
-- ggml commit:  9b4dae81f-dirty
-- OpenSSL found: 3.0.2
CMake Deprecation Warning at tools/ui/CMakeLists.txt:16 (message):
  LLAMA_WEBUI_HF_BUCKET is deprecated, use LLAMA_UI_HF_BUCKET instead


-- Configuring done (50.5s)
 -- Generating done (237.7s)
-- Build files have been written to: /content/drive/MyDrive/Shared/Downloads/GitHub/llama.cpp/build

real    4m48.773s
user    0m1.418s
sys     0m2.222s
[ -3%] Built target cpp-httplib
[ -1%] Built target llama-common-base
[  0%] Built target ggml-base
[  3%] Built target ggml-cpu



[ 41%] Built target ggml-cuda
[ 43%] Built target ggml
[ 87%] Built target llama
[ 96%] Built target llama-common
[ 96%] Built target llama-diffusion
[ 96%] Built target llama-diffusion-cli
        Command being timed: "cmake --build build -j --config Release --target llama-diffusion-cli -j2"
        User time (seconds): 1.02
        System time (seconds): 2.09
        Percent of CPU this job got: 6%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:49.94
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 21120
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 24370
        Voluntary context switches: 374024
        Involuntary context switches: 1819
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

OK, and these make it run the actual model safely on Google Colab:

#!/bin/bash

# DiffusionGemma Experimental Runner (Tesla T4 Optimized)
# Optimal for 16GB VRAM / 12GB System RAM

# MODEL SELECTION
# Q8_0 is too large (26GB) for T4. Q4_K_M (16GB) is the mandatory baseline.
MODEL="/content/unsloth/diffusiongemma-26B-A4B-it-GGUF/diffusiongemma-26B-A4B-it-Q4_K_M.gguf"

# PARAMETER LOGIC:
# -ngl 20: Maximum safe GPU offload. Leaves ~3GB headroom for CUDA driver and 
#          DiffusionGemma's large canvas/sampling buffers. Going higher OOMs.
# -n 512:  Targets 2 blocks (512 tokens). Higher values increase compute buffer size.
# -cnv:    Enables multi-turn conversation mode.
# --diffusion-visual: Optional. Enable to see live denoising.

# ./build/bin/llama-diffusion-cli -m $MODEL -ngl 99 -n 2048 # TOO AGGRESSIVE for T4
# ./build/bin/llama-diffusion-cli -m $MODEL -ngl 0 # TOO SLOW (CPU mode)

 nsys profile --stats=true ./build/bin/llama-diffusion-cli \
    -m "$MODEL" \
    -ngl 20 \
    -n 512 \
     "$@"

Update: https://unsloth.ai/docs/models/diffusiongemma#unsloth-studio-guide is a much easier method, but one needs to:

1. pip install unsloth # yes, just forget any .sh or venv and like
2. npm install -g localtunnel --verbose # we like to see what happens
   2b. /usr/local/lib/python3.12/dist-packages/studio/setup.sh # may be needed after server restart, not sure why so yet, because "run install.sh" error demands it: 'Studio not set up. Run install.sh first.' *
3. unsloth studio -p 8888 -H 0.0.0.0 # it rightly complains that no public URI yet
4. lt --port 8888
5. We copy the IP shown to the right box
6. We may need to refresh the server's GUI (not sure why so yet, otherwise the GGUFs etc are not downloaded yet)

Works.

It probably can be done via .ipynb there, to be tested....
Update, yes, it works, see below, silly me.

Oh, I should have RTFM-ed. Of course: https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb ... one could just click it.