| | --- |
| | datasets: |
| | - bigcode/the-stack |
| | - bigcode/the-stack-v2 |
| | - bigcode/starcoderdata |
| | - bigcode/commitpack |
| | - nvidia/OpenCodeReasoning |
| | library_name: transformers |
| | inference: true |
| | tags: |
| | - code |
| | license: mit |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # Spec Coder V1 |
| | **Spec Coder** is a cutting-edge, open-source AI model designed to assist with fundamental coding tasks. It is built on the **Llama architecture**, allowing seamless access via tools like **llama.cpp** and **Ollama**. This makes **Spec Coder** highly compatible with a variety of systems, enabling flexible deployment both locally and in the cloud. |
| |
|
| | Trained on vast datasets, **Spec Coder** excels in generating code, completing code snippets, and understanding programming tasks across multiple languages. It can be used for code completion, debugging, and automated code generation, acting as a versatile assistant for developers. |
| |
|
| | **Spec Coder** is optimized for integration into developer tools, providing intelligent coding assistance and facilitating research in programming languages. Its advanced transformer-based architecture, with 4 billion parameters, allows it to perform tasks across different environments efficiently. |
| |
|
| | The model supports various downstream tasks including supervised fine-tuning (SFT) and reinforcement learning (RL) to improve its performance for specific programming tasks. |
| |
|
| | # Training Data |
| | - Total Training Tokens: ~4.3 trillion tokens |
| | - Corpus: The Stack, StarCoder Training Dataset, The Stack v2, CommitPack, OpenCodeReasoning, English Wikipedia |
| |
|
| | # Training Details |
| | - Context Window: 8,192 tokens |
| | - Optimization: Standard language modeling objective |
| | - Hardware: Cluster of 5 x RTX 4090 GPUs |
| | - Training Duration: ~140 days (approximately 6 months) |
| |
|
| | # Benchmarks |
| | ## RepoBench 1.1 (Python) |
| | | Model | 2k | 4k | 8k | 12k | 16k | Avg | Avg ≤ 8k | |
| | |--------------------|-------|-------|-------|-------|-------|-------|----------| |
| | | Spec-Coder-4b-V1 | 30.42%| 38.55%| 36.91%| 32.75%| 30.34%| 34.59%| 36.23% | |
| |
|
| | ## Syntax-Aware Fill-in-the-Middle (SAFIM) |
| | | Model | Algorithmic | Control | API | Average | |
| | |----------------------|-------------|---------|--------|---------| |
| | | Spec-Coder-4b-V1 | 38.22% | 41.18% | 60.45% | 46.28% | |
| |
|
| | ## HumanEval Infilling |
| | | Model | Single-Line | Multi-Line | Random Span | |
| | |----------------------|-------------|------------|-------------| |
| | | Spec-Coder-4b-V1 | 72.34% | 45.65% | 39.12% | |
| |
|
| | # Limitations |
| | - **Biases**: The model may reflect biases present in the public codebases. |
| | - **Security**: Code generated by the model may contain security vulnerabilities. It is essential to verify and audit the code generated by the model for any potential risks. |
| |
|
| | # Sample Usage |
| | Here are examples of how to run and interact with **Spec Coder**: |
| |
|
| | ```python |
| | from transformers import AutoModelForCausalLM, AutoTokenizer |
| | |
| | model_name = "SVECTOR-CORPORATION/Spec-Coder-4b-V1" |
| | model = AutoModelForCausalLM.from_pretrained(model_name) |
| | tokenizer = AutoTokenizer.from_pretrained(model_name) |
| | |
| | input_code = "def factorial(n):\n if n == 0:" |
| | |
| | inputs = tokenizer(input_code, return_tensors="pt") |
| | |
| | outputs = model.generate(inputs['input_ids'], max_length=50, num_return_sequences=1) |
| | |
| | generated_code = tokenizer.decode(outputs[0], skip_special_tokens=True) |
| | |
| | print("Generated Python code:\n", generated_code) |
| | ``` |
