Attention Is All You Need
Paper • 1706.03762 • Published • 147
Linux Command Transformer
A small Transformer Encoder-Decoder trained from scratch with PyTorch to generate Linux commands from natural language descriptions.
This model was developed as a personal learning project to better understand the Transformer architecture, its training process, and its limitations.
The Transformer implementation was guided by Umar Jamil's Transformer from Scratch walkthrough.
Model Architecture
The model follows the original Transformer Encoder-Decoder architecture introduced in Attention Is All You Need.
| Parameter | Value |
|---|---|
| Encoder Layers | 2 |
| Decoder Layers | 2 |
| Model Dimension | 64 |
| Attention Heads | 4 |
| Feed Forward Dimension | 128 |
| Tokenizer | GPT-2 |
| Maximum Source Length | 64 |
| Maximum Target Length | 96 |
The model was trained with randomly initialized weights. No pretrained language model weights were used.
Training
The model was trained on approximately 8,000 natural language–Linux command pairs, using an NVIDIA RTX 4060 Laptop GPU with 8 GB of VRAM.
Training configuration:
- Optimizer: AdamW
- Learning rate: 0.001
- Loss: Cross-Entropy
- Gradient clipping: 1.0
- Maximum epochs: 10
The best checkpoint was selected at epoch 6, with a validation loss of 2.0178.
Evaluation Results
The model was evaluated on 796 test examples.
| Metric | Accuracy |
|---|---|
| Exact Match | 11.68% |
| Command Accuracy | 81.03% |
| First Argument Accuracy | 36.35% |
| Command + First Argument Accuracy | 35.45% |
The model identifies the correct main Linux utility in many cases, but generating complete commands with accurate arguments remains challenging.
The argument-related metrics were calculated on 773 examples containing at least one argument. Metrics use whitespace-based command splitting rather than full shell parsing.
Usage
This is a custom PyTorch architecture and cannot be loaded directly using AutoModel.from_pretrained().
The repository provides:
model.pt— Trained model weightsconfig.json— Architecture configurationtokenizer/— GPT-2 tokenizer configuration with additional special tokens
To run inference, use the implementation provided in the associated GitHub repository.
The inference script downloads the model from Hugging Face, reconstructs the Transformer architecture, loads the weights, and generates Linux commands.
Limitations
The model was trained on a relatively small dataset, and training experiments were constrained by the available GPU resources.
Although it achieves 81.03% Command Accuracy, its Exact Match Accuracy is only 11.68%.
This model is intended for educational and experimental purposes. Generated commands should be reviewed before execution, especially commands that modify or delete files.
References
- Downloads last month
- 67