Endless-1b

Model Description

Endless-1b is a decoder-only transformer language model trained with a three-stage approach: pretraining on diverse text corpora, supervised fine-tuning (SFT) on conversational data, and direct preference optimization (DPO) for alignment. The model architecture balances efficiency with decent performance on conversational and code generation tasks.

Model Details

Architecture

• Model Type: Decoder-only Transformer
• Parameters: 950 million (0.95B)
• Architecture Dimensions:
  • Hidden Size: 1,728
  • Layers: 24
  • Attention Heads: 16
  • Feed-Forward Dimension: 6,912
  • Context Length: 1,024 tokens
  • Vocabulary Size: 50,257 (GPT-2 tokenizer)

Training Approach

The model was trained in three stages:

Stage 1: Pretraining

• Trained from scratch on diverse text corpora
• Datasets:
  • FineWeb (web-scale dataset)
  • Wikipedia
  • Project Gutenberg
  • Additional book collections
• Hardware: Nvidia H100

Stage 2: Supervised Fine-Tuning (SFT)

• Fine-tuned on high-quality conversational and instruction-following data
• Datasets:
  • LMSYS Chat conversations
  • OpenOrca instruction dataset
  • UltraChat conversational data
  • Additional synthetic and freely available dialogue datasets
• Total training tokens: Approximately 2 billion
• Batch size: 128 (effective, with gradient accumulation)
• Learning rate: 3e-6 with cosine decay and warmup
• Hardware: Kaggle TPU v3-8

Stage 3: Direct Preference Optimization (DPO)

• Aligned using preference pairs from UltraFeedback dataset
• Filtered for English-only content and Python/JavaScript programming tasks
• DPO beta: 0.1
• Batch size: 32 (effective)
• Learning rate: 5e-7 with cosine decay
• Hardware: NVIDIA H100 GPU

Technical Optimizations

The H100 training implementation includes:

• Flash Attention 3 for efficient self-attention computation
• SwiGLU activation functions in feed-forward networks
• RMSNorm for layer normalization
• BFloat16 mixed precision training
• Fused AdamW optimizer

Intended Use

Primary Use Cases

• Conversational AI applications
• Question answering systems
• Content generation
• Code assistance (Python and JavaScript)
• Educational tutoring

Out-of-Scope Uses

• Multilingual applications or non-English content generation
• Code generation in languages other than Python and JavaScript (e.g., C++, Java, Rust, Go)
• Advanced mathematical proofs, theorem proving, or symbolic computation
• High-stakes decision making without human oversight
• Generation of harmful, biased, or illegal content
• Medical, legal, or financial advice

Training Data

Pretraining Data

• FineWeb: Large-scale web corpus
• Wikipedia: English Wikipedia dump
• Project Gutenberg: Public domain books
• Additional Books: Curated collection of educational and literary texts

SFT Training Data

• LMSYS Chat: Real-world conversational data from ChatBot Arena
• OpenOrca: High-quality instruction-following dataset
• UltraChat: Multi-turn dialogue dataset
• Synthetic Datasets: Generated conversational examples
• Additional Open Datasets: Curated instruction and dialogue data from freely available sources

DPO Training Data

• UltraFeedback: Preference pairs with chosen and rejected responses
• Filtering Criteria:
  • English language only (intentionally monolingual for quality)
  • Programming content limited to Python and JavaScript
  • Maximum sequence length of 900 tokens
  • Excluded C++, Java, and other programming languages
  • Excluded advanced mathematics and symbolic reasoning tasks

Rationale for Filtering: The model was intentionally optimized for high-quality English conversational capabilities rather than multilingual or broad technical coverage. By focusing exclusively on English and limiting programming support to Python and JavaScript, the model achieves better performance within its specialized domain. Advanced mathematics and non-English content were excluded to concentrate model capacity on conversational fluency and practical coding assistance in the most commonly used languages.

Data Preprocessing

All training data underwent rigorous filtering to remove:

• Non-English content (intentional focus on English quality)
• Excessively long sequences beyond model capacity
• Low-quality or repetitive responses
• Code examples in unsupported programming languages (C++, Java, Ruby, Go, etc.)
• Advanced mathematical proofs and symbolic notation
• Multilingual or code-switched content

Evaluation

Performance Characteristics

The model demonstrates medium performance on its target domains:

• Single-turn English conversation coherence
• Instruction following in English
• Factual question answering in English
• Basic mathematical reasoning and arithmetic at times

Design Philosophy: Rather than attempting broad coverage across all languages and programming paradigms, this model prioritizes depth over breadth. The intentional focus on English and popular programming languages (Python/JavaScript) allows for higher quality outputs within these domains compared to models with more diffuse training objectives.

Check out the paper i published on https://scaling-laws-limit.endless.sbs

Limitations

By Design (Intentional Constraints):

• English-only optimization - not trained for multilingual tasks
• Programming support limited to Python and JavaScript only
• Mathematical reasoning focused on basic arithmetic and explanations, not advanced proofs

Technical Limitations:

• Context window limited to 1,024 tokens
• May exhibit biases present in training data
• May produce hallucinations or factually incorrect information
• Performance degrades on out-of-domain tasks (non-English, unsupported programming languages, advanced mathematics)

Usage

Installation

pip install transformers torch

Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "https://huggingface.co/end765/Endless-1b"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

Inference Example

from endless_chat import EndlessChat
chat = EndlessChat("end765/Endless-1b")

response = chat.generate("Hello!")
print(response)

# Multi-turn conversation
history = [
    {"role": "user", "content": "Hi"},
    {"role": "assistant", "content": "Hello!"}
]
response = chat.chat(history, "How are you?")
print(response)

Conversation Format

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

class EndlessChat:
    
    def __init__(self, model_name="end765/Endless-1b"):
        self.device = "cuda" if torch.cuda.is_available() else "cpu"
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(model_name).to(self.device)
        self.model.eval()
        
        if self.tokenizer.pad_token is None:
            self.tokenizer.pad_token = self.tokenizer.eos_token
    
    def generate(self, prompt, max_tokens=256, temp=0.7):
        formatted = f"<|human|>{prompt}<|end|><|assistant|>"
        inputs = self.tokenizer(formatted, return_tensors="pt").to(self.device)
        
        with torch.inference_mode():
            outputs = self.model.generate(
                **inputs,
                max_new_tokens=max_tokens,
                temperature=temp,
                top_p=0.9,
                do_sample=True,
                pad_token_id=self.tokenizer.eos_token_id
            )
        
        response = self.tokenizer.decode(outputs[0], skip_special_tokens=False)
        response = response.split("<|assistant|>")[-1].split("<|end|>")[0].strip()
        return response
    
    def chat(self, history, message, max_tokens=256, temp=0.7):
        conversation = ""
        for turn in history:
            if turn["role"] == "user":
                conversation += f"<|human|>{turn['content']}<|end|>"
            else:
                conversation += f"<|assistant|>{turn['content']}<|end|>"
        
        conversation += f"<|human|>{message}<|end|><|assistant|>"
        inputs = self.tokenizer(conversation, return_tensors="pt").to(self.device)
        
        with torch.inference_mode():
            outputs = self.model.generate(
                **inputs,
                max_new_tokens=max_tokens,
                temperature=temp,
                top_p=0.9,
                do_sample=True,
                pad_token_id=self.tokenizer.eos_token_id
            )
        
        response = self.tokenizer.decode(outputs[0], skip_special_tokens=False)
        response = response.split("<|assistant|>")[-1].split("<|end|>")[0].strip()
        return response


chat = EndlessChat("end765/Endless-1b")

print(chat.generate("Hello!"))

history = []
while True:
    user_input = input("You: ").strip()
    if user_input.lower() in ['quit', 'exit']:
        break
    if not user_input:
        continue
    
    response = chat.chat(history, user_input) if history else chat.generate(user_input)
    print(f"Assistant: {response}")
    
    history.append({"role": "user", "content": user_input})
    history.append({"role": "assistant", "content": response})

Ethical Considerations

Bias and Fairness

The model may exhibit biases present in its training data, including but not limited to:

• Cultural and demographic biases
• Programming language preferences (Python/JavaScript)
• English-centric worldview

Users should be aware of these limitations and implement appropriate safeguards when deploying the model.

Safety and Alignment

While the model underwent alignment training through DPO, it may still generate:

• Factually incorrect information
• Harmful or inappropriate content
• Biased or stereotypical responses

Human oversight is recommended for production deployments, particularly in sensitive applications.

Environmental Impact

Carbon Footprint

• Hardware:
  • Kaggle TPU v3-8 (pretraining and SFT)
  • NVIDIA H100 (DPO alignment)
• Training Time: Approximately 2 weeks hours total across all stages.

The environmental impact was minimized through:

• Efficient training techniques including mixed precision and gradient accumulation
• Optimized attention mechanisms (Flash Attention 3)
• Use of Kaggle's TPU infrastructure for compute-intensive stages

Citation

If you use this model in your research or applications, please cite:

@misc{Endless-1b2026,
  author = {Endless},
  title = {TinyTransformer-950M: Efficient Conversational Language Model with DPO Alignment},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/end765/Endless-1b}},
}

License

This model is released under the Apache 2.0 License. See LICENSE file for details.

The model was trained on datasets with various licenses. Users should review and comply with the licenses of the underlying datasets:

• Arena Human Preference Dataset
• Tulu-3 SFT Personas
• UltraFeedback

Acknowledgments

This model was trained using:

• HuggingFace Transformers library
• Flash Attention implementation
• Kaggle TPU v3-8 infrastructure (pretraining and SFT stages)
• NVIDIA H100 infrastructure (DPO alignment stage)

Special thanks to:

• Kaggle for providing free TPU resources and some datasets
• The creators of FineWeb, Wikipedia, Project Gutenberg, LMSYS, OpenOrca, UltraChat, and UltraFeedback datasets
• The open-source community

Contact

For questions, issues, or collaboration inquiries, please open an issue on the model repository or contact mnbc80115@gmail.com.

Version History

v1.0 (Current)

• Initial release
• 950M parameter model
• Three-stage training: pretraining, SFT, and DPO alignment
• Support for English conversational tasks and Python/JavaScript code generation.