LLoPS-360M

LLoPS-360M is the 360M language model from Learnable Low-Rank Polynomial Sketch for Effective Linear Attention. It replaces BASED global linear attention with a learnable degree-4 low-rank polynomial sketch and is continually trained on SlimPajama-6B at sequence length 2,048.

This is a raw pretrained language model. It has not been instruction-tuned or audited for downstream deployment.

Model details

• Base model: hazyresearch/based-360m
• Polynomial degree: 4 (llops_power=2)
• Linear term: disabled
• Tokenizer: GPT-2
• Continual-training data: SlimPajama-6B

Usage

Install the code from the LLoPS repository, then load the model:

from transformers import AutoTokenizer
from based.models.gpt_unified import GPTSketchLMHeadModel

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = GPTSketchLMHeadModel.from_pretrained_hf(
    "zongyueq/LLoPS-360M",
    device="cuda",
)

Zero-shot results

WinoGrande	PIQA	LAMBADA	HellaSwag	ARC-E	ARC-C	FDA	SWDE	SQuAD
52.09	66.87	29.25	37.06	43.81	25.43	7.44	31.41	32.21

PIQA, HellaSwag, ARC-Easy, and ARC-Challenge use normalized accuracy; LAMBADA reports accuracy. All tasks are evaluated at zero shot.

License

Apache-2.0. The implementation is derived from HazyResearch/BASED; see the repository NOTICE for upstream attribution.