pihu21057w/astral-134m-base

Astral

Astral is a base causal language model designed for English-language text generation and language understanding tasks. It is a pretrained foundation model trained on approximately 30 billion tokens and built around a modified Llama 3–inspired architecture. Astral is intended as a strong starting point for downstream adaptation rather than a ready-to-deploy chat model.

Overview

Astral is a base model package for researchers and developers who want to work with a locally hosted, Transformers-compatible foundation model. The repository includes:

• a custom configuration class for model metadata and hyperparameters
• a custom causal language modeling implementation for generation and fine-tuning workflows
• a custom tokenizer implementation compatible with the model vocabulary and special-token conventions
• auto-mapping metadata so Transformers can discover the local classes automatically

Because Astral is a base model, it requires supervised fine-tuning, instruction tuning, or domain adaptation before it is suitable for production chat, assistant, or task-specific behavior.

Highlights

• Base pretrained English-only model
• Approximately 30B tokens of pretraining
• Modified Llama 3–inspired architecture
• Designed for fine-tuning and adaptation
• Compatible with Hugging Face Transformers loading flow
• Supports local loading with trust_remote_code=True

Model Summary

Property	Value
Model type	Astral base causal language model
Architecture	Modified Llama 3–inspired decoder-only transformer
Training scope	English-only pretraining
Pretraining tokens	Approximately 30B
Hidden size	576
Intermediate size	1536
Number of layers	30
Attention heads	9
Key/value heads	3
Context length	8192
Vocabulary size	49152
Activation	SiLU
Precision	float16
Use cache	Enabled

Repository Structure

• config.json — model configuration metadata and auto-map entries
• configuration_astral.py — custom Astral config class
• modeling_astral.py — custom Astral causal language model implementation
• tokenizer_astral.py — custom tokenizer implementation
• special_tokens_map.json — tokenizer special-token definitions
• generation_config.json — generation defaults for sampling and decoding

Requirements

Install the core dependencies before running inference:

pip install torch transformers tokenizers accelerate

If you are loading a large model, a GPU-enabled environment is strongly recommended.

Quick Start

Load the model locally

Astral is a base model and should be treated as a pretrained foundation checkpoint. It is not intended to be used directly as a polished assistant without further fine-tuning.

from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

model_dir = "."

config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    config=config,
    trust_remote_code=True,
)

Generate text

prompt = "Astral is"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Configuration Notes

The model metadata in config.json is aligned with the custom Astral config class so that generation and loading behave consistently. In particular:

• the architecture is exposed as AstralForCausalLM
• the model type is registered as astral
• the tokenizer and model classes are mapped through the auto_map block for automatic discovery

This is important when using custom code with Transformers because otherwise the loader cannot resolve the local implementation automatically.

Tokenizer Notes

The tokenizer module is designed to work with the same special-token conventions expected by the model configuration. The implementation uses a BPE-based tokenizer and includes compatible special token definitions in special_tokens_map.json.

If you are using a real pretrained vocabulary artifact, make sure the tokenizer files and model weights are available in the same directory structure expected by your deployment environment.

Generation Settings

The default generation behavior is defined in generation_config.json and can be overridden per call. Typical settings include:

• sampling enabled for creative generation
• temperature control for response variability
• maximum new token length tuning for latency and quality tradeoffs

Intended Use

Astral is suitable for:

• base-model experimentation
• continued pretraining and domain adaptation
• supervised fine-tuning
• instruction tuning and conversational specialization
• research and prototype development

Limitations

• Astral is an English-only base model and is not optimized out-of-the-box for multilingual tasks.
• The model requires fine-tuning before it is appropriate for instruction-following, assistant-style behavior, or task-specific deployment.
• This package assumes the pretrained weights are already available and compatible with the provided architecture.
• Tokenizer and model behavior depend on the exact checkpoint and vocabulary assets supplied by the user.

License

Use the license that applies to your underlying pretrained model and tokenizer assets. The repository structure itself is provided for inference integration and should be reviewed alongside any model checkpoint terms.

Citation and Attribution

If you are using this package in a research or production workflow, please preserve attribution to the original architecture and tokenization design that the implementation is based on, and clearly document the source of your pretrained weights.