SproutKO-130M

Runtime documentation: English · 日本語 · 한국어

SproutKO-130M is a Korean pretrained decoder-only Transformer with 129,983,232 unique parameters. It is a base model for text completion, not an instruction-tuned chat model.

• Code: sprout-research/SproutKO-130M
• Release verification: release/VERIFICATION.md

The release configuration records 7,000,031,232 training tokens at step 106,812. The pretraining sequence length is 2,048. The configured maximum generation length, including the prompt, is 4,096.

transformers.AutoModelForCausalLM and AutoTokenizer are not supported. Use the custom sproutko runtime.

How to use

This is a gated model. Request or accept access while signed in to Hugging Face. Then install the runtime from the GitHub repository, authenticate the Hugging Face CLI, and load this Hub bundle.

python -m pip install .
hf auth login
sproutko-generate --checkpoint sprout-research/SproutKO-130M --prompt "한국어는" --max-new-tokens 64 --temperature 0 --device cpu

The CLI uses the user access token cached by hf auth login. Without approved model access and authentication, Hub downloads fail with a gated-repository authorization error.

import torch
from sproutko import SproutKOForCausalLM, SproutKOTokenizer, generate

source = "sprout-research/SproutKO-130M"
model = SproutKOForCausalLM.from_pretrained(source)
tokenizer = SproutKOTokenizer.from_pretrained(source)

ids = torch.tensor([tokenizer.encode("한국어는")], dtype=torch.long)
output = generate(
    model,
    ids,
    max_new_tokens=64,
    temperature=0,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0].tolist()))

The example uses the token cached by hf auth login. Both from_pretrained loaders also accept revision, token, and cache_dir; pass token explicitly when cached CLI authentication is not available. To pin a release, pass the same Hub commit revision to both loaders.

To load a local copy, download config.json, model.safetensors, and sproutko-tokenizer-32k-v1.json from the same Hub revision into one directory, then pass that path as source or --checkpoint.

Architecture

A causal decoder with pre-normalization, RMSNorm, Rotary Position Embeddings (RoPE), Grouped Query Attention (GQA), and a SwiGLU feed-forward network. Token embeddings and the output projection share weights.

Property	SproutKO-130M
Unique parameters	129,983,232
Vocabulary size	32,000
Hidden size	768
Transformer layers	16
Query / KV heads	12 / 4
Head dimension	64
Feed-forward intermediate size	2,176
Maximum generation sequence length, including prompt	4,096
Pretraining sequence length	2,048
RoPE theta	10,000
RMSNorm epsilon	1e-6
Tied word embeddings	Yes

Tokenizer

SproutKO-Tokenizer-32K-v1 is a custom 32,000-token BPE tokenizer backed by the Rust tokenizers library. The artifact is sproutko-tokenizer-32k-v1.json.

• Text is normalized to Unicode NFC and line endings are standardized. BOM characters are removed.
• Byte fallback covers characters outside the learned vocabulary.
• Load the artifact through SproutKOTokenizer, not AutoTokenizer.

Special token	ID
<pad>	0
<s> (BOS)	1
</s> (EOS)	2
<unk>	3

Always use weights and tokenizer files from the same release.

Intended use and limitations

• Use a text prompt for completion with this base model. It is not instruction-tuned and is not a chat model.
• Prompts must contain at least one token. The combined prompt and requested output length is limited to 4,096 tokens.
• Training used 2,048-token sequences. Quality beyond that length is not guaranteed.
• Generated text can be inaccurate or biased.
• This bundle does not include training data, optimizer state, training scripts, or evaluation reports.

Verification

On CPU, the inference runtime matches the original runtime on nine tokenizer samples and on logits plus 12-token greedy output for these weights. Details are in the release verification record. GPU inference has not been verified.

Files

File	Purpose
config.json	Model architecture and bundle filenames
model.safetensors	Model weights
sproutko-tokenizer-32k-v1.json	Vocabulary, merges, and embedded tokenizer backend
LICENSE	Apache-2.0
README.md	This model card

License

Code, exported weights, and tokenizer are licensed under Apache-2.0.

---

SproutKO-130M

런타임 문서: English · 日本語 · 한국어

SproutKO-130M은 129,983,232개 고유 파라미터의 한국어 사전학습 decoder-only Transformer입니다. 텍스트를 이어 쓰는 기본 모델이며, instruction-tuned 채팅 모델이 아닙니다.

• 코드: sprout-research/SproutKO-130M
• 릴리스 검증 기록: release/VERIFICATION.md

릴리스 설정에 기록된 학습량은 106,812스텝, 7,000,031,232토큰입니다. 사전학습 시퀀스 길이는 2,048입니다. 프롬프트를 포함한 최대 생성 길이는 4,096입니다.

transformers.AutoModelForCausalLM과 AutoTokenizer는 지원하지 않습니다. 자체 sproutko 런타임을 사용하세요.

사용 방법

이 모델은 gated 모델입니다. Hugging Face에 로그인한 상태에서 접근을 요청하거나 이용 조건에 동의하세요. 그런 다음 GitHub 저장소에서 런타임을 설치하고 Hugging Face CLI에 로그인한 뒤 이 Hub 번들을 불러옵니다.

python -m pip install .
hf auth login
sproutko-generate --checkpoint sprout-research/SproutKO-130M --prompt "한국어는" --max-new-tokens 64 --temperature 0 --device cpu

CLI는 hf auth login으로 저장한 사용자 액세스 토큰을 사용합니다. 모델 접근 승인과 인증이 없으면 gated 저장소 권한 오류로 다운로드에 실패합니다.

import torch
from sproutko import SproutKOForCausalLM, SproutKOTokenizer, generate

source = "sprout-research/SproutKO-130M"
model = SproutKOForCausalLM.from_pretrained(source)
tokenizer = SproutKOTokenizer.from_pretrained(source)

ids = torch.tensor([tokenizer.encode("한국어는")], dtype=torch.long)
output = generate(
    model,
    ids,
    max_new_tokens=64,
    temperature=0,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0].tolist()))

위 예제는 hf auth login으로 저장한 토큰을 사용합니다. 두 from_pretrained 로더는 revision, token, cache_dir도 지원하므로 CLI 로그인 정보를 사용할 수 없는 환경에서는 token을 명시적으로 전달하세요. 특정 릴리스를 고정하려면 두 로더에 동일한 Hub 커밋 revision을 전달하세요.

로컬로 쓰려면 같은 Hub revision에서 config.json, model.safetensors, sproutko-tokenizer-32k-v1.json을 한 폴더에 받은 뒤 그 경로를 source 또는 --checkpoint로 지정하세요.

아키텍처

Pre-Norm 구조에 RMSNorm, Rotary Position Embeddings(RoPE), Grouped Query Attention(GQA), SwiGLU 피드포워드 네트워크를 사용하는 causal decoder입니다. 토큰 임베딩과 출력 프로젝션은 가중치를 공유합니다.

항목	SproutKO-130M
고유 파라미터 수	129,983,232
어휘 크기	32,000
Hidden size	768
Transformer 레이어 수	16
Query / KV head 수	12 / 4
Head dimension	64
피드포워드 중간 차원	2,176
프롬프트를 포함한 최대 생성 시퀀스 길이	4,096
사전학습 시퀀스 길이	2,048
RoPE theta	10,000
RMSNorm epsilon	1e-6
토큰 임베딩·출력 가중치 공유	사용

토크나이저

SproutKO-Tokenizer-32K-v1은 Rust tokenizers 라이브러리를 백엔드로 사용하는 자체 32,000토큰 BPE 토크나이저입니다. 파일명은 sproutko-tokenizer-32k-v1.json입니다.

• Unicode NFC 정규화와 줄바꿈 통일을 적용하고 BOM 문자를 제거합니다.
• 학습된 어휘 밖의 문자는 byte fallback으로 처리합니다.
• AutoTokenizer가 아니라 SproutKOTokenizer로 불러오세요.

특수 토큰	ID
<pad>	0
<s> (BOS)	1
</s> (EOS)	2
<unk>	3

가중치와 토크나이저는 항상 동일한 릴리스의 파일을 함께 사용하세요.

사용 범위와 한계

• 기본 모델에 텍스트 프롬프트를 입력하여 이어질 내용을 생성합니다. instruction-tuned 모델이 아니며 채팅 모델도 아닙니다.
• 프롬프트에는 최소 한 개의 토큰이 필요합니다. 프롬프트와 요청한 출력 길이의 합은 최대 4,096토큰입니다.
• 학습 길이는 2,048토큰입니다. 그 밖의 길이에서는 품질을 보장하지 않습니다.
• 생성된 텍스트는 부정확하거나 편향될 수 있습니다.
• 이 번들에는 학습 데이터, optimizer 상태, 학습 스크립트, 평가 보고서가 포함되지 않습니다.

검증

CPU에서 추론 런타임은 원본 런타임과 토크나이저 샘플 9개, 그리고 이 가중치의 로짓 및 12토큰 greedy 출력이 일치합니다. 자세한 내용은 릴리스 검증 기록을 보세요. GPU 추론은 검증하지 않았습니다.

파일

파일	용도
config.json	모델 구조와 번들 파일명
model.safetensors	모델 가중치
sproutko-tokenizer-32k-v1.json	어휘, 병합 규칙, 내장 토크나이저 백엔드
LICENSE	Apache-2.0
README.md	이 모델 카드

라이선스

코드, 배포 가중치, 토크나이저는 Apache-2.0 라이선스를 따릅니다.