Instructions to use startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b")

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b

SGLang

How to use startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b with Docker Model Runner:
```
docker model run hf.co/startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b
```

gdn-gatedfa-340m-pas-fa-layer12-10b

This is a 340M controlled-pretraining checkpoint released for the paper Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus.

It is one member of the Massive Activations HLA model release and the corresponding Hugging Face Collection. The official analysis and reproducibility code is available at StartluxLabs/Massive-Activations-HLA.

Checkpoint details

Field	Value
Model	`gdn-gatedfa-340m-pas-fa-layer12-10b`
Scale	340M
Training tokens	10B
Experiment	PAS
Full-attention layers	12 (one-based)
Output-gating variant	GDN and full-attention output gates enabled
Final training step	38146
Weight format	Safetensors

Compatibility and reproducibility scope

The checkpoint is a weights-only research artifact. Its full-attention layers use a post-SDPA, head-specific sigmoid output gate inspired by the G1 design in Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (official code). The exact GatedDeltaNet integration is not distributed, so this checkpoint is not part of the public from-scratch quickstart. The repository loader raises an explicit compatibility message unless a user deliberately configures a compatible local implementation.

Loading

REPO_ID = "startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b"

The files can be downloaded without a Hugging Face subfolder argument:

from huggingface_hub import snapshot_download

local_path = snapshot_download(REPO_ID)

Downloading the weights does not make this variant directly executable; see the compatibility scope above.

Reproduction

Use the official GitHub repository for the tested analysis workflow, PAS/ISP metrics, lifecycle atlases, installation instructions, and model registry:

Intended use and limitations

This checkpoint is a research artifact for studying massive activations, hybrid linear-attention architectures, PAS/ISP morphology, attention placement, output gating, and scale. It is not instruction-tuned, safety-tuned, or validated for production deployment. It has not been comprehensively evaluated for downstream accuracy, factuality, bias, robustness, privacy, or safety.

The checkpoint uses the custom Transformers architecture GatedDeltaNetForCausalLM (model_type="gated_deltanet").

Training data

The model was trained from scratch on open data. A dataset-level composition and sampling breakdown is not included in this release; users should not infer language or domain coverage beyond the published project materials.

Citation

Please cite the accompanying paper. The arXiv link and final BibTeX entry will be added after the preprint metadata is public.

License

The released model artifacts are available under the Apache License 2.0. See LICENSE. Third-party software and datasets retain their own licenses and terms.

Downloads last month: 179

Safetensors

Model size

0.4B params

Tensor type

F32

Collection including startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b

Massive Activations in Hybrid Linear Attention Models

Collection

Official PAS, ISP, gating-ablation, and scale-study checkpoints for Massive Activations in Hybrid Linear Attention Models. • 10 items • Updated 1 day ago

Paper for startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

Paper • 2505.06708 • Published May 10, 2025 • 11