| | --- |
| | license: apache-2.0 |
| | datasets: |
| | - Skylion007/openwebtext |
| | language: |
| | - en |
| | pipeline_tag: text-generation |
| | tags: |
| | - research |
| | - convolutional |
| | - fft |
| | - transformer-alternative |
| | - causal-lm |
| | --- |
| | |
| | # GCLM — Global Convolutional Language Model |
| |
|
| | ## Model Summary |
| |
|
| | **GCLM (Global Convolutional Language Model)** is an experimental causal language model that replaces traditional self-attention with a hybrid **local + global convolutional architecture**. |
| |
|
| | Instead of attention heads, GCLM uses: |
| | - **Local depthwise convolutions** for short-range context |
| | - **FFT-based global convolutions** for long-range sequence modeling |
| |
|
| | This design explores whether **global receptive fields** can be achieved efficiently *without* quadratic attention, while remaining compatible with standard autoregressive language modeling. |
| |
|
| | > GCLM is a transformer alternative — not a transformer replacement. |
| |
|
| | --- |
| |
|
| | ## Architecture Overview |
| |
|
| | - Token + learned positional embeddings |
| | - Stacked convolutional blocks: |
| | - Local depthwise + pointwise convolution |
| | - Optional global FFT convolution every *N* layers |
| | - Feedforward MLP |
| | - Residual connections + LayerNorm |
| | - Causal language modeling head |
| |
|
| | **Key properties:** |
| | - No attention mechanism |
| | - No KV cache |
| | - Linear memory scaling with sequence length |
| | - Extremely long-context friendly (tested up to 8k+ tokens) |
| |
|
| | --- |
| |
|
| | ## Training Data |
| |
|
| | The model was trained on: |
| | - **Skylion007/openwebtext** |
| |
|
| | This dataset contains raw, unfiltered internet text and may include biased, incorrect, or unsafe content. |
| |
|
| | --- |
| |
|
| | ## Intended Use |
| |
|
| | **Primary use cases:** |
| | - Research into transformer alternatives |
| | - Long-context modeling experiments |
| | - Architectural ablation studies |
| | - Educational exploration of non-attention sequence models |
| |
|
| | **Not intended for:** |
| | - Safety-critical applications |
| | - Medical, legal, or financial advice |
| | - Deployment as a production chatbot without additional alignment work |
| |
|
| | --- |
| |
|
| | ## Limitations |
| |
|
| | - This model is **research-grade**, not instruction-tuned |
| | - Outputs may be: |
| | - Incoherent |
| | - Factually incorrect |
| | - Biased or unsafe |
| | - Performance characteristics differ significantly from transformer LMs |
| | - No reinforcement learning or alignment tuning applied |
| |
|
| | --- |
| |
|
| | ## Ethical Considerations |
| |
|
| | GCLM was trained on publicly available web data and may reflect societal biases present in that data. |
| |
|
| | Users are responsible for: |
| | - Applying appropriate filtering |
| | - Avoiding harmful or misleading use cases |
| | - Evaluating outputs critically |
| |
|
| | --- |
| |
|
| | ## License |
| |
|
| | This model is released under the **Apache License 2.0**. |
| |
|
| | You are free to: |
| | - Use |
| | - Modify |
| | - Distribute |
| | - Use commercially |
| |
|
| | Attribution and license preservation are required. |
| | Patent rights are explicitly granted under this license. |
| |
|
| | --- |
| |
|
| | ## Citation |
| |
|
| | If you use GCLM in your research, please cite or reference the project. |
| |
|
| |
|
| | ## Important |
| |
|
| | The model will not be put in the repo until it has finished training. |
