LLoPS-1B

LLoPS-1B is the 1B-scale language model from Learnable Low-Rank Polynomial Sketch for Effective Linear Attention. It replaces BASED global linear attention with a learnable degree-2 low-rank polynomial sketch with a learned linear term and is continually trained on SlimPajama-6B at sequence length 2,048.

This is a raw pretrained language model. It has not been instruction-tuned or audited for downstream deployment.

Model details

• Base model: hazyresearch/based-1b
• Polynomial degree: 2 (llops_power=1)
• Linear term: enabled
• Tokenizer: GPT-2
• Continual-training data: SlimPajama-6B

Usage

Install the code from the LLoPS repository, then load the model:

from transformers import AutoTokenizer
from based.models.gpt_unified import GPTSketchLMHeadModel

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = GPTSketchLMHeadModel.from_pretrained_hf(
    "zongyueq/LLoPS-1B",
    device="cuda",
)

Zero-shot results

WinoGrande	PIQA	LAMBADA	HellaSwag	ARC-E	ARC-C	FDA	SWDE	SQuAD
51.38	68.82	38.06	42.76	47.14	26.96	26.13	48.06	36.26

PIQA, HellaSwag, ARC-Easy, and ARC-Challenge use normalized accuracy; LAMBADA reports accuracy. All tasks are evaluated at zero shot.

License

Apache-2.0. The implementation is derived from HazyResearch/BASED; see the repository NOTICE for upstream attribution.