pharo-llm/pharo-one-qwen2-5-32B

pharo-one

pharo-one is a collection of language models and adapters for Pharo.

The first release is a continued-pretraining checkpoint trained only to improve a compatible base model's understanding of Pharo syntax, source-code structure, message-passing conventions, and common language patterns.

It is not yet fine-tuned for code generation, code completion, test generation, bug fixing, code repair, or instruction following. Those capabilities may be introduced in later checkpoints within the collection.

Pharo is a low-resource programming language with syntax and conventions that differ significantly from mainstream languages. The purpose of this first checkpoint is language adaptation only: it teaches the model what Pharo code looks like before any downstream task-specific fine-tuning.

Collection Overview

The pharo-one collection may eventually contain multiple checkpoints trained for different tasks and model sizes.

The first checkpoint is:

• Continued-pretrained on raw Pharo source code
• Trained with a causal language-modeling objective
• Intended only for Pharo language and syntax adaptation
• Distributed as a LoRA or PEFT adapter, when applicable

Future checkpoints may add supervised fine-tuning for code generation, completion, testing, repair, or other downstream tasks.

Each checkpoint should be used according to the task and training configuration described in its repository.

The underlying base model may differ between checkpoints. Consult the model configuration, adapter configuration, or repository metadata to identify the compatible base checkpoint.

Model Details

• Collection name: pharo-one
• Model family: Decoder-only causal language models
• Programming language: Pharo / Smalltalk
• Primary domain: Source-code generation and understanding
• Framework: Transformers
• Adapter framework: PEFT / LoRA, when applicable
• Training method: Continued pre-training
• Current purpose: Pharo syntax and language adaptation
• Current task scope: Causal language modeling only

Motivation

Large language models perform well on popular programming languages such as Python, Java, JavaScript, and C++, but their performance is often weaker on low-resource languages such as Pharo.

Pharo inherits distinctive features from Smalltalk, including:

• Keyword-based message syntax
• Message-oriented control flow
• Dynamic typing
• Reflective APIs
• Image-based development
• Compact method definitions
• Test-driven development conventions

General-purpose code models may not represent these characteristics accurately. The pharo-one collection adapts coding models to Pharo syntax, APIs, idioms, and development patterns.

Training Overview

The first model in this collection uses the following training stage.

Continued Pre-Training

A compatible causal language model is adapted to Pharo source code using next-token prediction.

This stage helps the model learn:

• Pharo syntax
• Common selectors
• Standard library APIs
• Class and method conventions
• Message-passing patterns
• Naming and formatting conventions
• Typical project structures

Training examples are tokenized, concatenated, and packed into fixed-length causal language-modeling blocks.

Current Scope

The first pharo-one checkpoint is intended only to improve the model's internal representation of Pharo code.

It is trained to learn:

• Pharo syntax
• Message-send structure
• Keyword, unary, and binary messages
• Method and class-definition patterns
• Common selectors and naming conventions
• Typical source-code formatting
• Recurrent Pharo language patterns

It should not be presented as a task-specialized model for code generation, test generation, bug fixing, or fill-in-the-middle completion.

Installation

Install the required libraries:

pip install -U transformers accelerate torch peft

Some checkpoints may require additional packages. Refer to the corresponding model repository for checkpoint-specific requirements.

Usage

Loading a Standalone Model

Use this pattern for checkpoints distributed as complete models:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "<PHARO_ONE_MODEL_ID>"

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
).eval()

Loading a LoRA or PEFT Adapter

Use this pattern for checkpoints distributed as adapters:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_name = "<COMPATIBLE_BASE_MODEL_ID>"
adapter_name = "<PHARO_ONE_ADAPTER_ID>"

tokenizer = AutoTokenizer.from_pretrained(
    base_model_name,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(
    model,
    adapter_name,
).eval()

The base model must match the checkpoint used to train the adapter. Check adapter_config.json or the checkpoint repository for the required base-model identifier.

Basic Causal-Language-Modeling Example

prompt = """Collection >> selectEven

    ^ self select: [ :each |
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
    )

generated_text = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)

print(generated_text)

The model may continue the Pharo source sequence with text such as:

Collection >> selectEven

    ^ self select: [ :each | each even ]

Future Test-Generation Models

prompt = """Generate a Pharo unit test for this method:

Calculator >> add: firstNumber to: secondNumber
    ^ firstNumber + secondNumber
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=192,
        do_sample=False,
    )

completion = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)

print(completion)

A future task-specific checkpoint may produce results such as:

CalculatorTest >> testAdd

    self
        assert: (calculator add: 2 to: 3)
        equals: 5

Future Bug-Fixing Models

prompt = """Fix the following Pharo method:

Collection >> firstEven
    ^ self detect: [ :each | each odd ]
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
    )

completion = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)

print(completion)

A future task-specific checkpoint may produce corrections such as:

Collection >> firstEven

    ^ self detect: [ :each | each even ]

Prompt Formats

The first checkpoint uses ordinary causal-language-model input rather than a task-specific instruction format.

A checkpoint may use:

• Plain causal-language-model prompts
• Instruction-style prompts
• Chat templates
• Fill-in-the-middle tokens
• Structured task prefixes
• Repository-specific templates

Always consult the individual checkpoint model card for the recommended prompt format.

Training Configuration

Training settings may differ between checkpoints. A typical language-adaptation run may use:

• LoRA or full-model fine-tuning
• Causal language modeling
• Packed source-code sequences
• Automatic mixed precision
• Gradient checkpointing
• Multi-GPU distributed training
• Periodic evaluation and checkpointing
• Automatic checkpoint resumption
• Validation loss and perplexity tracking

The exact values for context length, learning rate, batch size, LoRA rank, number of epochs, and target modules are checkpoint-specific.

Evaluation

Models in the collection should be evaluated on task-appropriate Pharo datasets.

Possible metrics include:

Language Adaptation

• Validation loss
• Perplexity

Code Completion

• Exact match
• Edit similarity
• Token accuracy
• CodeBLEU
• Compilation or parsing success

Test Generation

• Test compilation rate
• Test execution rate
• Assertion correctness
• Mutation score
• Code coverage

Bug Fixing

• Exact patch match
• Plausible patch rate
• Test-suite pass rate
• Functional correctness

Evaluation results should be reported in each checkpoint repository when available.

Intended Use

The first pharo-one checkpoint is intended for:

• Continued pre-training research
• Pharo syntax adaptation
• Low-resource programming-language research
• Parameter-efficient language adaptation
• Use as a starting point for later supervised fine-tuning
• Local experimentation with Pharo source-code modeling

It is not intended to be evaluated or advertised as a specialized code-generation, test-generation, bug-fixing, or code-completion model without additional task-specific fine-tuning.

Limitations

• Generated code may be incorrect, incomplete, insecure, or non-compiling.
• The models do not execute or validate generated Pharo code.
• Generated selectors, classes, methods, or APIs may not exist.
• Performance varies across checkpoints and base-model families.
• Some models are specialized for a single task and may perform poorly on others.
• LoRA adapters require the exact compatible base model.
• FIM tokens and chat templates may differ between model families.
• The models may reproduce errors, biases, insecure patterns, or licensing characteristics found in their training data.
• Repository-level reasoning may be limited when the required context exceeds the model context window.
• Generated tests may pass without adequately validating program behavior.
• Bug-fixing outputs should be reviewed and verified with the project test suite.

Generated code should be reviewed, executed, and tested before use in production.

Responsible Use

Users should:

• Review generated code manually
• Run relevant tests
• Validate API and selector availability
• Check licensing requirements
• Avoid using generated code without verification in critical systems
• Avoid submitting private or sensitive source code to untrusted inference services

Reproducibility

Exact results may vary across:

• Base-model versions
• Tokenizers
• Dataset revisions
• Hardware
• Precision modes
• PyTorch versions
• Transformers versions
• PEFT versions
• CUDA versions
• Distributed-training configurations
• Decoding parameters

For reproducible experiments, record:

• The full checkpoint identifier
• The model revision
• The dataset revision
• The training command
• The random seed
• Dependency versions
• Hardware information
• Prompt templates
• Generation parameters

License

This repository is released under the MIT License.

Individual checkpoints may also be subject to the license and usage terms of their underlying base models and training datasets. Users are responsible for reviewing all applicable licenses before use, redistribution, or deployment.