| | --- |
| | title: ZeroGPU-LLM-Inference |
| | emoji: 🧠 |
| | colorFrom: indigo |
| | colorTo: purple |
| | sdk: gradio |
| | sdk_version: 5.49.1 |
| | app_file: app.py |
| | pinned: false |
| | license: apache-2.0 |
| | short_description: Streaming LLM chat with web search and controls |
| | --- |
| | |
| | # 🧠 ZeroGPU LLM Inference |
| |
|
| | A modern, user-friendly Gradio interface for **token-streaming, chat-style inference** across a wide variety of Transformer models—powered by ZeroGPU for free GPU acceleration on Hugging Face Spaces. |
| |
|
| | ## ✨ Key Features |
| |
|
| | ### 🎨 Modern UI/UX |
| | - **Clean, intuitive interface** with organized layout and visual hierarchy |
| | - **Collapsible advanced settings** for both simple and power users |
| | - **Smooth animations and transitions** for better user experience |
| | - **Responsive design** that works on all screen sizes |
| | - **Copy-to-clipboard** functionality for easy sharing of responses |
| |
|
| | ### 🔍 Web Search Integration |
| | - **Real-time DuckDuckGo search** with background threading |
| | - **Configurable timeout** and result limits |
| | - **Automatic context injection** into system prompts |
| | - **Smart toggle** - search settings auto-hide when disabled |
| |
|
| | ### 💡 Smart Features |
| | - **Thought vs. Answer streaming**: `<think>…</think>` blocks shown separately as "💭 Thought" |
| | - **Working cancel button** - immediately stops generation without errors |
| | - **Debug panel** for prompt engineering insights |
| | - **Duration estimates** based on model size and settings |
| | - **Example prompts** to help users get started |
| | - **Dynamic system prompts** with automatic date insertion |
| |
|
| | ### 🎯 Model Variety |
| | - **30+ LLM options** from leading providers (Qwen, Microsoft, Meta, Mistral, etc.) |
| | - Models ranging from **135M to 32B+** parameters |
| | - Specialized models for **reasoning, coding, and general chat** |
| | - **Efficient model loading** - one at a time with automatic cache clearing |
| |
|
| | ### ⚙️ Advanced Controls |
| | - **Generation parameters**: max tokens, temperature, top-k, top-p, repetition penalty |
| | - **Web search settings**: max results, chars per result, timeout |
| | - **Custom system prompts** with dynamic date insertion |
| | - **Organized in collapsible sections** to keep interface clean |
| |
|
| | ## 🔄 Supported Models |
| |
|
| | ### Compact Models (< 2B) |
| | - **SmolLM2-135M-Instruct** - Tiny but capable |
| | - **SmolLM2-360M-Instruct** - Lightweight conversation |
| | - **Taiwan-ELM-270M/1.1B** - Multilingual support |
| | - **Qwen3-0.6B/1.7B** - Fast inference |
| |
|
| | ### Mid-Size Models (2B-8B) |
| | - **Qwen3-4B/8B** - Balanced performance |
| | - **Phi-4-mini** (4.3B) - Reasoning & Instruct variants |
| | - **MiniCPM3-4B** - Efficient mid-size |
| | - **Gemma-3-4B-IT** - Instruction-tuned |
| | - **Llama-3.2-Taiwan-3B** - Regional optimization |
| | - **Mistral-7B-Instruct** - Classic performer |
| | - **DeepSeek-R1-Distill-Llama-8B** - Reasoning specialist |
| |
|
| | ### Large Models (14B+) |
| | - **Qwen3-14B** - Strong general purpose |
| | - **Apriel-1.5-15b-Thinker** - Multimodal reasoning |
| | - **gpt-oss-20b** - Open GPT-style |
| | - **Qwen3-32B** - Top-tier performance |
| |
|
| | ## 🚀 How It Works |
| |
|
| | 1. **Select Model** - Choose from 30+ pre-configured models |
| | 2. **Configure Settings** - Adjust generation parameters or use defaults |
| | 3. **Enable Web Search** (optional) - Get real-time information |
| | 4. **Start Chatting** - Type your message or use example prompts |
| | 5. **Stream Response** - Watch as tokens are generated in real-time |
| | 6. **Cancel Anytime** - Stop generation mid-stream if needed |
| |
|
| | ### Technical Flow |
| |
|
| | 1. User message enters chat history |
| | 2. If search enabled, background thread fetches DuckDuckGo results |
| | 3. Search snippets merge into system prompt (within timeout limit) |
| | 4. Selected model pipeline loads on ZeroGPU (bf16→f16→f32 fallback) |
| | 5. Prompt formatted with thinking mode detection |
| | 6. Tokens stream to UI with thought/answer separation |
| | 7. Cancel button available for immediate interruption |
| | 8. Memory cleared after generation for next request |
| |
|
| | ## ⚙️ Generation Parameters |
| |
|
| | | Parameter | Range | Default | Description | |
| | |-----------|-------|---------|-------------| |
| | | Max Tokens | 64-16384 | 1024 | Maximum response length | |
| | | Temperature | 0.1-2.0 | 0.7 | Creativity vs focus | |
| | | Top-K | 1-100 | 40 | Token sampling pool size | |
| | | Top-P | 0.1-1.0 | 0.9 | Nucleus sampling threshold | |
| | | Repetition Penalty | 1.0-2.0 | 1.2 | Reduce repetition | |
| |
|
| | ## 🌐 Web Search Settings |
| |
|
| | | Setting | Range | Default | Description | |
| | |---------|-------|---------|-------------| |
| | | Max Results | Integer | 4 | Number of search results | |
| | | Max Chars/Result | Integer | 50 | Character limit per result | |
| | | Search Timeout | 0-30s | 5s | Maximum wait time | |
| |
|
| | ## 💻 Local Development |
| |
|
| | ```bash |
| | # Clone the repository |
| | git clone https://huggingface.co/spaces/Luigi/ZeroGPU-LLM-Inference |
| | cd ZeroGPU-LLM-Inference |
| | |
| | # Install dependencies |
| | pip install -r requirements.txt |
| | |
| | # Run the app |
| | python app.py |
| | ``` |
| |
|
| | ## 🎨 UI Design Philosophy |
| |
|
| | The interface follows these principles: |
| |
|
| | 1. **Simplicity First** - Core features immediately visible |
| | 2. **Progressive Disclosure** - Advanced options hidden but accessible |
| | 3. **Visual Hierarchy** - Clear organization with groups and sections |
| | 4. **Feedback** - Status indicators and helpful messages |
| | 5. **Accessibility** - Responsive, keyboard-friendly, with tooltips |
| |
|
| | ## 🔧 Customization |
| |
|
| | ### Adding New Models |
| |
|
| | Edit `MODELS` dictionary in `app.py`: |
| |
|
| | ```python |
| | "Your-Model-Name": { |
| | "repo_id": "org/model-name", |
| | "description": "Model description", |
| | "params_b": 7.0 # Size in billions |
| | } |
| | ``` |
| |
|
| | ### Modifying UI Theme |
| |
|
| | Adjust theme parameters in `gr.Blocks()`: |
| |
|
| | ```python |
| | theme=gr.themes.Soft( |
| | primary_hue="indigo", |
| | secondary_hue="purple", |
| | # ... more options |
| | ) |
| | ``` |
| |
|
| | ## 📊 Performance |
| |
|
| | - **Token streaming** for responsive feel |
| | - **Background search** doesn't block UI |
| | - **Efficient memory** management with cache clearing |
| | - **ZeroGPU acceleration** for fast inference |
| | - **Optimized loading** with dtype fallbacks |
| |
|
| | ## 🤝 Contributing |
| |
|
| | Contributions welcome! Areas for improvement: |
| |
|
| | - Additional model integrations |
| | - UI/UX enhancements |
| | - Performance optimizations |
| | - Bug fixes and testing |
| | - Documentation improvements |
| |
|
| | ## 📝 License |
| |
|
| | Apache 2.0 - See LICENSE file for details |
| |
|
| | ## 🙏 Acknowledgments |
| |
|
| | - Built with [Gradio](https://gradio.app) |
| | - Powered by [Hugging Face Transformers](https://huggingface.co/transformers) |
| | - Uses [ZeroGPU](https://huggingface.co/zero-gpu-explorers) for acceleration |
| | - Search via [DuckDuckGo](https://duckduckgo.com) |
| |
|
| | --- |
| |
|
| | **Made with ❤️ for the open source community** |
| |
|
