---
license: apache-2.0
language:
- en
tags:
- text-generation
- cli
- shell
- command-line
- sft
- instruction-following
pipeline_tag: text-generation
widget:
- text: "Instruction: List all files in the current directory\nCommand:"
  example_title: List files
- text: "Instruction: Find all Python files\nCommand:"
  example_title: Find Python files
- text: "Instruction: Show disk usage\nCommand:"
  example_title: Disk usage
---

# Tiny-LLM CLI SFT (54M)

A **54 million parameter** language model fine-tuned for CLI command generation.

## Model Description

This model is a Supervised Fine-Tuned (SFT) version of [jonmabe/tiny-llm-54m](https://huggingface.co/jonmabe/tiny-llm-54m), trained to generate Unix/Linux shell commands from natural language instructions.

### Training Data

- **Geddy's NL2Bash dataset**: ~2,300 natural language to bash command pairs
- **NL2Bash benchmark**: Standard benchmark for command translation
- **Synthetic examples**: Additional generated pairs
- **Total**: ~13,000 training pairs

### Training Details

| Parameter | Value |
|-----------|-------|
| Base Model | tiny-llm-54m |
| Training Steps | 2,000 |
| Best Checkpoint | Step 1,000 |
| Best Val Loss | 1.2456 |
| Learning Rate | 5e-5 |
| Batch Size | 16 |
| Hardware | NVIDIA RTX 5090 |
| Training Time | ~9 minutes |

## Architecture

- **Parameters**: 54.93M
- **Layers**: 12
- **Hidden Size**: 512
- **Attention Heads**: 8
- **Intermediate Size**: 1408
- **Max Position**: 512
- **Vocabulary**: 32,000 tokens
- **Features**: RoPE, RMSNorm, SwiGLU, Weight Tying

## Usage

### Prompt Format

```
Instruction: <natural language description>
Command:
```

### Example

```python
from model import TinyLLM
import torch

# Load model
checkpoint = torch.load("best_model.pt", map_location="cpu")
model = TinyLLM(checkpoint["config"]["model"])
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

# Generate
prompt = "Instruction: Find all Python files modified in the last day\nCommand:"
# ... tokenize and generate
```

## Limitations

⚠️ **Known Issues:**
- Tokenizer decode shows raw BPE tokens (Ġ = space, Ċ = newline)
- Model generates fragments of correct commands but output can be noisy
- Needs more training steps for reliable generation
- Small model size limits command complexity

## Improvement Plan

1. **Fix tokenizer decode** - Proper BPE to text conversion
2. **Longer training** - 5,000-10,000 steps
3. **Data quality** - Curate cleaner training pairs
4. **Lower LR** - More stable convergence with 1e-5

## License

Apache 2.0

## Citation

```bibtex
@misc{tiny-llm-cli-sft-2026,
  author = {Jon Mabe},
  title = {Tiny-LLM CLI SFT: Small Language Model for Command Generation},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/jonmabe/tiny-llm-cli-sft}
}
```