feat: self-healing gateway, package replay, private space guard, key rotation, readme sync

- start.sh: gateway restart loop, shell capture wrappers (apt/pip/uv/npm/hermes),
pool key promotion for 16 providers, HUGGINGMES_RUN startup scripts,
env-driven package installs, JupyterLab PID guard, enhanced boot banner
- health-server.js: private space guard (HF API detect + redirect),
embed-aware dashboard buttons, /hf-redirect route
- .env.example: add HUGGINGMES_RUN, package installs, key rotation pools,
gateway restart vars, terminal vars, webhook URL
- README.md: document all new features, fix terminal section (on by default),
add ephemeral package replay and key rotation sections

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.env.example

@@ -21,6 +21,12 @@ GATEWAY_TOKEN=your_gateway_token_here
 # Allowed Telegram User IDs (comma-separated numeric IDs)
 # TELEGRAM_ALLOWED_USERS=123456789,987654321
 
+# Telegram mode: "webhook" (default, requires Cloudflare proxy) or "polling"
+# TELEGRAM_MODE=webhook
+
+# Telegram webhook URL (auto-set via Cloudflare Worker; override only if manual)
+# TELEGRAM_WEBHOOK_URL=https://your-webhook.workers.dev
+
 # ── OPTIONAL: Cloudflare Proxy & Keep-Alive ──
 # Cloudflare API token for automatic Worker proxy and KeepAlive setup
 # CLOUDFLARE_WORKERS_TOKEN=your_cloudflare_token_here
@@ -42,9 +48,53 @@ HF_TOKEN=hf_your_token_here
 # Backup interval in seconds (default: 180)
 SYNC_INTERVAL=180
 
-# ── OPTIONAL: Advanced ──
-# Telegram mode (webhook/polling)
-TELEGRAM_MODE=webhook
-
 # Include .env in backups (default: false)
 # SYNC_INCLUDE_ENV=false
+
+# ── OPTIONAL: Startup Scripts & Package Install ──
+# Run arbitrary bash on every boot (before Hermes gateway launches).
+# Supports base64: prefix for quote-heavy scripts.
+# HUGGINGMES_RUN="""
+# sudo apt-get install -y ffmpeg
+# pip install pandas
+# """
+
+# Install packages from day one (before terminal is opened)
+# HUGGINGMES_APT_PACKAGES=ffmpeg,git-lfs
+# HUGGINGMES_PIP_PACKAGES=pandas,requests
+# HUGGINGMES_NPM_PACKAGES=typescript
+
+# ── OPTIONAL: Gateway Restart Behavior ──
+# Seconds to wait between gateway restarts (default: 5)
+# GATEWAY_RESTART_DELAY=5
+
+# Max gateway restarts before container exits (default: unlimited)
+# GATEWAY_MAX_RESTARTS=10
+
+# ── OPTIONAL: API Key Rotation ──
+# Comma-separated key pools per provider for round-robin rotation.
+# The first key in a pool is also promoted to the singular env var.
+# ANTHROPIC_API_KEYS=sk-ant-key1,sk-ant-key2,sk-ant-key3
+# OPENAI_API_KEYS=sk-openai-key1,sk-openai-key2
+# GEMINI_API_KEYS=AIza-key1,AIza-key2
+# DEEPSEEK_API_KEYS=key1,key2
+# GROQ_API_KEYS=gsk_key1,gsk_key2
+# MISTRAL_API_KEYS=key1,key2
+# OPENROUTER_API_KEYS=sk-or-key1,sk-or-key2
+# XAI_API_KEYS=key1,key2
+# NVIDIA_API_KEYS=key1,key2
+# COHERE_API_KEYS=key1,key2
+# TOGETHER_API_KEYS=key1,key2
+# CEREBRAS_API_KEYS=key1,key2
+
+# ── OPTIONAL: Terminal (JupyterLab) ──
+# Terminal is ON by default when GATEWAY_TOKEN is set.
+# Set DEV_MODE=false to disable it entirely.
+# DEV_MODE=false
+
+# Override terminal password (optional — defaults to GATEWAY_TOKEN)
+# JUPYTER_TOKEN=your_separate_terminal_token_here
+
+# ── OPTIONAL: Advanced ──
+# Webhook URL for restart/backup-failure notifications
+# WEBHOOK_URL=https://your-webhook-endpoint.com/notify

README.md

@@ -23,9 +23,9 @@ secrets:
   - name: CLOUDFLARE_WORKERS_TOKEN
     description: "Cloudflare API token for automatic Worker proxy and KeepAlive setup."
   - name: DEV_MODE
-    description: "Set to 'true' to enable the JupyterLab terminal at /terminal/."
+    description: "Set to 'false' to disable the JupyterLab terminal at /terminal/. Terminal is on by default when GATEWAY_TOKEN is set."
   - name: JUPYTER_TOKEN
-    description: "Strong token to secure JupyterLab terminal access (required when DEV_MODE=true). Generate: openssl rand -hex 32"
+    description: "Override terminal password (optional). Defaults to GATEWAY_TOKEN — no extra secret needed."
 ---
 
 <!-- Badges -->
@@ -46,6 +46,8 @@ secrets:
 - [📱 Telegram Setup](#-telegram-setup)
 - [🌐 Cloudflare Proxy](#-cloudflare-proxy)
 - [💾 Backup & Persistence](#-backup--persistence)
+- [📦 Ephemeral Package Re-install](#-ephemeral-package-re-install-optional)
+- [🔑 API Key Rotation](#-api-key-rotation-optional)
 - [💓 Staying Alive](#-staying-alive-recommended-on-free-hf-spaces)
 - [🔐 Security & Advanced](#-security--advanced)
 - [💻 Terminal Access (JupyterLab)](#-terminal-access-jupyterlab)
@@ -58,10 +60,14 @@ secrets:
 - 🧠 **Hermes Core:** Runs Hermes Agent for multi-turn chat, tools, memory, and agent workflows.
 - 🔐 **Secure by Default:** Protects the dashboard and API with a single gateway token.
 - 🌐 **Built-in Connectivity:** Adds Cloudflare Worker proxy support for Telegram and other blocked outbound traffic.
-- 📊 **Dashboard:** Real-time view of uptime, sync health, and agent status at `/`.
+- 📊 **Dashboard:** Real-time view of uptime, sync health, model, provider, and agent status at `/`.
 - 💾 **Persistent Backup:** Syncs chats, config, and session data to a private HF Dataset.
 - ⏰ **Keep-Alive:** Can provision a cron-triggered Cloudflare Worker to keep the Space awake.
-- 💻 **Terminal Access:** Optional JupyterLab terminal at `/terminal/` for direct shell access (enable with `DEV_MODE=true`).
+- 💻 **Terminal Out of the Box:** JupyterLab terminal at `/terminal/` auto-enabled when `GATEWAY_TOKEN` is set — no extra config needed.
+- 🔄 **Self-Healing Gateway:** Gateway, dashboard, health server, and JupyterLab are all monitored and automatically restarted if they exit unexpectedly.
+- 📦 **Ephemeral Package Replay:** Install packages from the terminal and they survive restarts — shell wrappers record `apt`/`pip`/`uv`/`npm`/`hermes` installs and replay them on every boot.
+- 🚀 **Startup Scripts:** Run arbitrary bash at boot via `HUGGINGMES_RUN` or `HUGGINGMES_APT/PIP/NPM_PACKAGES` variables.
+- 🔑 **API Key Pool Rotation:** Supply comma-separated key pools (e.g. `ANTHROPIC_API_KEYS=key1,key2`) and the first key is promoted automatically.
 - 🤖 **Broad Provider Support:** Supports Hermes' native providers, direct API-key providers, OAuth providers, and custom OpenAI-compatible endpoints.
 
 ## 🎥 Video Tutorial
@@ -188,6 +194,56 @@ Hugging Face Spaces often block outbound calls to APIs used by Telegram and some
 
 Set `HF_TOKEN` with write access to enable backup. HuggingMes syncs workspace data to a private HF Dataset named `huggingmes-backup` every 600 seconds by default.
 
+| Variable | Default | Description |
+| :--- | :--- | :--- |
+| `HF_TOKEN` | — | HF token with **Write** access |
+| `BACKUP_DATASET_NAME` | `huggingmes-backup` | Dataset name for backup |
+| `SYNC_INTERVAL` | `600` | Backup frequency in seconds |
+
+## 📦 Ephemeral Package Re-install *(Optional)*
+
+Install packages in the terminal and they survive Space restarts — no extra config needed. Shell wrappers record every successful `apt install`, `pip install`, `uv pip install`, `npm install -g`, and `hermes plugins install` into `workspace/startup.sh`, which is backed up and replayed automatically on next boot.
+
+For packages you want installed from day one (before the terminal is even opened), use the startup variables:
+
+| Variable | What to put in it |
+| :--- | :--- |
+| `HUGGINGMES_RUN` | Full bash script to run on every startup (multi-line, heredocs, `if` blocks all work) |
+| `HUGGINGMES_APT_PACKAGES` | Space-separated apt packages to install |
+| `HUGGINGMES_PIP_PACKAGES` | Space-separated Python packages to install |
+| `HUGGINGMES_NPM_PACKAGES` | Space-separated npm packages to install globally |
+
+**Example:**
+
+```bash
+HUGGINGMES_RUN="""
+pip install pandas matplotlib
+npm install -g tsx
+sudo apt-get install -y ffmpeg
+"""
+```
+
+For scripts with complex quoting, base64-encode them:
+
+```bash
+# locally
+base64 -w0 setup.sh
+# HF Variable
+HUGGINGMES_RUN=base64:<paste-output-here>
+```
+
+## 🔑 API Key Rotation *(Optional)*
+
+Spread requests across multiple API keys to avoid rate limits. Supply a comma-separated pool — the first key is promoted to the provider's singular env var, and Hermes picks it up automatically.
+
+```bash
+ANTHROPIC_API_KEYS=sk-ant-key1,sk-ant-key2
+OPENAI_API_KEYS=sk-oai-key1,sk-oai-key2
+OPENROUTER_API_KEYS=sk-or-key1,sk-or-key2
+```
+
+Supported pool vars: `OPENROUTER_API_KEYS`, `ANTHROPIC_API_KEYS`, `OPENAI_API_KEYS`, `GOOGLE_API_KEYS`, `GEMINI_API_KEYS`, `DEEPSEEK_API_KEYS`, `KIMI_API_KEYS`, `MINIMAX_API_KEYS`, `NVIDIA_API_KEYS`, `XAI_API_KEYS`, `KILOCODE_API_KEYS`, `GLM_API_KEYS`, `ARCEEAI_API_KEYS`, `DASHSCOPE_API_KEYS`, `GMI_API_KEYS`, `TOKENHUB_API_KEYS`.
+
 ## 💓 Staying Alive
 
 With `CLOUDFLARE_WORKERS_TOKEN` set, HuggingMes can create a keep-alive worker that pings the Space's `/health` endpoint on a schedule so the free tier stays awake longer.
@@ -201,9 +257,12 @@ With `CLOUDFLARE_WORKERS_TOKEN` set, HuggingMes can create a keep-alive worker t
 | `CLOUDFLARE_WORKERS_TOKEN` | — | Cloudflare API token for proxying and keep-awake |
 | `SYNC_INTERVAL` | `600` | Backup frequency in seconds |
 | `CLOUDFLARE_KEEPALIVE_ENABLED` | `true` | Set `false` to disable keep-awake worker |
-| `TELEGRAM_MODE` | `webhook` | `webhook` or `polling` |
+| `TELEGRAM_MODE` | `webhook` | `webhook` or `polling` (webhook auto-configured from `SPACE_HOST`) |
 | `DEV_MODE` | `true` | Set `false` to disable JupyterLab terminal at `/terminal/` |
 | `JUPYTER_TOKEN` | *(uses `GATEWAY_TOKEN`)* | Override terminal password (optional) |
+| `WEBHOOK_URL` | — | Endpoint for POST JSON restart notifications |
+| `GATEWAY_RESTART_DELAY` | `5` | Seconds between gateway restart attempts |
+| `GATEWAY_MAX_RESTARTS` | `0` (unlimited) | Maximum gateway restart count before container exits |
 
 ## 💻 Terminal Access (JupyterLab)
 
@@ -237,7 +296,7 @@ docker compose up --build
 - **Dashboard (`/`)**: Real-time management and monitoring.
 - **Hermes App (`/app/`)**: Secure proxied access to the Hermes UI.
 - **API (`/v1/*`)**: Proxied OpenAI-compatible agent API.
-- **Terminal (`/terminal/`)**: JupyterLab terminal (requires `DEV_MODE=true`).
+- **Terminal (`/terminal/`)**: JupyterLab terminal (auto-enabled when `GATEWAY_TOKEN` is set; set `DEV_MODE=false` to disable).
 - **Health Check (`/health`)**: Readiness probe for HF and keep-alive.
 - **Sync Engine**: Python background task for HF Dataset persistence.
 

health-server.js

@@ -1,6 +1,7 @@
 "use strict";
 
 const http = require("http");
+const https = require("https");
 const fs = require("fs");
 const net = require("net");
 const crypto = require("crypto");
@@ -18,6 +19,59 @@ const APP_BASE = "/app";
 const LOGIN_PATH = "/login";
 const SESSION_COOKIE = "huggingmes_session";
 
+// ── Private Space redirect support ──
+const SPACE_ID = (process.env.SPACE_ID || "").trim();
+function deriveHfSpaceUrl() {
+  if (SPACE_ID) return `https://huggingface.co/spaces/${SPACE_ID}`;
+  const host = (process.env.SPACE_HOST || "").replace(/\.hf\.space$/i, "");
+  const author = (process.env.SPACE_AUTHOR_NAME || "").trim().toLowerCase();
+  if (author && host.toLowerCase().startsWith(author + "-")) {
+    const spaceName = host.slice(author.length + 1);
+    return `https://huggingface.co/spaces/${process.env.SPACE_AUTHOR_NAME}/${spaceName}`;
+  }
+  return "";
+}
+const HF_SPACE_URL = deriveHfSpaceUrl();
+
+let SPACE_IS_PRIVATE = false;
+async function detectSpacePrivacy() {
+  if (!SPACE_ID) return;
+  try {
+    const token = (process.env.HF_TOKEN || "").trim();
+    const reqOptions = {
+      hostname: "huggingface.co",
+      path: `/api/spaces/${SPACE_ID}`,
+      method: "GET",
+      headers: Object.assign(
+        { "User-Agent": "HuggingMes/health-server" },
+        token ? { Authorization: `Bearer ${token}` } : {}
+      ),
+    };
+    await new Promise((resolve) => {
+      const r = https.request(reqOptions, (res) => {
+        let body = "";
+        res.on("data", (chunk) => { body += chunk; });
+        res.on("end", () => {
+          try {
+            if (res.statusCode === 200) {
+              const data = JSON.parse(body);
+              SPACE_IS_PRIVATE = data.private === true;
+            } else if (res.statusCode === 404 && !token) {
+              SPACE_IS_PRIVATE = true;
+            }
+          } catch {}
+          resolve();
+        });
+      });
+      r.on("error", resolve);
+      r.setTimeout(5000, () => { r.destroy(); resolve(); });
+      r.end();
+    });
+    console.log(`[health-server] Space privacy: ${SPACE_IS_PRIVATE ? "private" : "public"}`);
+  } catch {}
+}
+detectSpacePrivacy();
+
 const SYNC_STATUS_FILE = "/tmp/huggingmes-sync-status.json";
 const CLOUDFLARE_KEEPALIVE_STATUS_FILE =
   "/tmp/huggingmes-cloudflare-keepalive-status.json";
@@ -347,6 +401,36 @@ async function statusPayload() {
   };
 }
 
+function renderPrivateRedirect(targetUrl) {
+  const safeUrl = escapeHtml(targetUrl);
+  return `<!doctype html><html lang="en"><head>
+  <meta charset="utf-8"/><meta name="viewport" content="width=device-width,initial-scale=1"/>
+  <meta http-equiv="refresh" content="3;url=${safeUrl}"/>
+  <title>HuggingMes — Private Space</title>
+  <style>
+    :root{color-scheme:dark}
+    body{margin:0;min-height:100vh;display:flex;align-items:center;justify-content:center;
+         font-family:Inter,ui-sans-serif,system-ui,-apple-system,sans-serif;
+         background:#08080f;color:#f6f4ff;text-align:center;padding:24px}
+    .card{border:1px solid #26243a;background:#12111b;border-radius:14px;padding:36px 32px;max-width:440px}
+    h1{margin:0 0 12px;font-size:1.5rem}
+    p{color:#b8b3d7;line-height:1.6;margin:0 0 24px}
+    .btn{display:inline-flex;align-items:center;justify-content:center;
+         background:#fff;color:#000;font-weight:850;font-size:.95rem;
+         border-radius:8px;padding:12px 28px;text-decoration:none;transition:opacity .15s}
+    .btn:hover{opacity:.85}
+    .sub{color:#7f7a9e;font-size:.78rem;margin-top:16px}
+  </style></head><body>
+  <div class="card">
+    <h1>🔒 Private Space</h1>
+    <p>This HuggingFace Space is private. You need to be logged in to <strong>huggingface.co</strong> to access it.<br><br>Redirecting you now&hellip;</p>
+    <a class="btn" href="${safeUrl}">Open on Hugging Face →</a>
+    <div class="sub">Redirecting in 3 seconds&hellip;</div>
+  </div>
+  <script>setTimeout(() => { window.location.replace(${JSON.stringify(targetUrl)}); }, 100);</script>
+</body></html>`;
+}
+
 function badge(label, state) {
   return `<span class="badge ${state ? "ok" : "off"}">${escapeHtml(label)}</span>`;
 }
@@ -529,9 +613,9 @@ function renderDashboard(data) {
       <div class="subtitle">Self-hosted - Hermes Agent</div>
     </header>
     <div class="hero-buttons">
-      <a class="hero-action" href="${APP_BASE}/" target="_blank" rel="noopener noreferrer">Open Hermes Agent →</a>
-      <a class="hero-action secondary" href="/terminal/" target="_blank" rel="noopener noreferrer">Open Terminal →</a>
-      <a class="hero-action secondary" href="/env-builder" target="_blank" rel="noopener noreferrer">ENV Builder →</a>
+      <a class="hero-action" data-space-link="app" href="${APP_BASE}/">Open Hermes Agent →</a>
+      <a class="hero-action secondary" data-space-link="terminal" href="/terminal/">💻 Open Terminal →</a>
+      <a class="hero-action secondary" data-space-link="env-builder" href="/env-builder">⚙️ ENV Builder →</a>
     </div>
     <section class="overview">
       ${tiles}
@@ -545,6 +629,28 @@ function renderDashboard(data) {
         el.textContent = 'At ' + date.toLocaleTimeString();
       }
     });
+    const inEmbeddedApp = (() => { try { return window.top !== window.self; } catch { return true; } })();
+    const isDirectHfSpaceHost = /\.hf\.space$/i.test(window.location.hostname);
+    const HF_SPACE_URL = ${JSON.stringify(HF_SPACE_URL)};
+    const SPACE_IS_PRIVATE = ${JSON.stringify(SPACE_IS_PRIVATE)};
+    if (SPACE_IS_PRIVATE && isDirectHfSpaceHost && !inEmbeddedApp && HF_SPACE_URL) {
+      const notice = document.createElement('div');
+      notice.style.cssText = 'position:fixed;inset:0;display:flex;align-items:center;justify-content:center;background:#08080f;color:#f6f4ff;font-family:sans-serif;flex-direction:column;gap:16px;z-index:9999';
+      notice.innerHTML = '<span style="font-size:1.1rem">🔒 Private Space &mdash; Redirecting&hellip;</span><a href="' + HF_SPACE_URL + '" style="color:#a5b4fc;font-size:.85rem">Click here if not redirected</a>';
+      document.body.appendChild(notice);
+      setTimeout(() => { window.location.replace(HF_SPACE_URL); }, 300);
+    }
+    // Force new-tab navigation when running inside the HF App iframe or on a raw .hf.space link
+    const openInNewTab = inEmbeddedApp || isDirectHfSpaceHost;
+    document.querySelectorAll('a[data-space-link]').forEach((a) => {
+      if (openInNewTab) {
+        a.setAttribute('target', '_blank');
+        a.setAttribute('rel', 'noopener noreferrer');
+      } else {
+        a.removeAttribute('target');
+        a.removeAttribute('rel');
+      }
+    });
   </script>
 </body>
 </html>`;
@@ -559,6 +665,25 @@ const server = http.createServer(async (req, res) => {
     return;
   }
 
+  // ── Private Space Guard (server-side) ──
+  // Intercepts browser HTML requests from raw .hf.space hosts when the Space is private.
+  // /health and /status are always exempt so uptime monitors keep working.
+  const isHtmlReq = (req.headers.accept || "").includes("text/html");
+  const isDirectHfSpaceReq = SPACE_IS_PRIVATE &&
+    HF_SPACE_URL &&
+    isHtmlReq &&
+    typeof req.headers.host === "string" &&
+    req.headers.host.endsWith(".hf.space");
+
+  if (path === "/hf-redirect" || path === "/hf-redirect/") {
+    if (HF_SPACE_URL) {
+      res.writeHead(302, { location: HF_SPACE_URL, "cache-control": "no-store" });
+      return res.end();
+    }
+    res.writeHead(404, { "content-type": "text/plain" });
+    return res.end("SPACE_ID not configured.");
+  }
+
   if (path === "/health" || path === `${APP_BASE}/health`) {
     const data = await statusPayload();
     // Always 200 — health server up means the app is running.
@@ -610,6 +735,10 @@ const server = http.createServer(async (req, res) => {
   }
 
   if (path === "/") {
+    if (isDirectHfSpaceReq) {
+      res.writeHead(200, { "content-type": "text/html; charset=utf-8" });
+      return res.end(renderPrivateRedirect(HF_SPACE_URL));
+    }
     const data = await statusPayload();
     res.writeHead(200, { "content-type": "text/html; charset=utf-8" });
     res.end(renderDashboard(data));

start.sh

@@ -17,6 +17,7 @@ TELEGRAM_WEBHOOK_PORT="${TELEGRAM_WEBHOOK_PORT:-8765}"
 SYNC_INTERVAL="${SYNC_INTERVAL:-600}"
 BACKUP_DATASET="${BACKUP_DATASET_NAME:-huggingmes-backup}"
 CF_PROXY_ENV_FILE="/tmp/huggingmes-cloudflare-proxy.env"
+STARTUP_FILE="$HERMES_HOME/workspace/startup.sh"
 
 export HERMES_HOME
 export API_SERVER_ENABLED="${API_SERVER_ENABLED:-true}"
@@ -223,6 +224,40 @@ if [ -n "${CLOUDFLARE_PROXY_URL:-}" ] && [ -z "$TELEGRAM_BASE_URL" ]; then
   export TELEGRAM_BASE_FILE_URL="${CLOUDFLARE_PROXY_URL}/file/bot"
 fi
 
+# ── Pool key promotion ──
+# Mirror first key from comma-separated pool vars into the singular env var.
+# Hermes providers read singular vars; this lets users supply pool keys like
+# ANTHROPIC_API_KEYS=key1,key2 and have them picked up automatically.
+promote_first_pool_key() {
+  local singular_var="$1"
+  local pool_var="$2"
+  local singular_val="${!singular_var:-}"
+  local pool_val="${!pool_var:-}"
+  [ -n "$singular_val" ] && return 0
+  [ -n "$pool_val" ] || return 0
+  local first
+  first=$(printf '%s' "$pool_val" | tr ',' '\n' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' | awk 'NF{print; exit}')
+  [ -n "$first" ] || return 0
+  export "${singular_var}=$first"
+}
+
+promote_first_pool_key "OPENROUTER_API_KEY"   "OPENROUTER_API_KEYS"
+promote_first_pool_key "ANTHROPIC_API_KEY"    "ANTHROPIC_API_KEYS"
+promote_first_pool_key "OPENAI_API_KEY"       "OPENAI_API_KEYS"
+promote_first_pool_key "GOOGLE_API_KEY"       "GOOGLE_API_KEYS"
+promote_first_pool_key "GEMINI_API_KEY"       "GEMINI_API_KEYS"
+promote_first_pool_key "DEEPSEEK_API_KEY"     "DEEPSEEK_API_KEYS"
+promote_first_pool_key "KIMI_API_KEY"         "KIMI_API_KEYS"
+promote_first_pool_key "MINIMAX_API_KEY"      "MINIMAX_API_KEYS"
+promote_first_pool_key "NVIDIA_API_KEY"       "NVIDIA_API_KEYS"
+promote_first_pool_key "XAI_API_KEY"          "XAI_API_KEYS"
+promote_first_pool_key "KILOCODE_API_KEY"     "KILOCODE_API_KEYS"
+promote_first_pool_key "GLM_API_KEY"          "GLM_API_KEYS"
+promote_first_pool_key "ARCEEAI_API_KEY"      "ARCEEAI_API_KEYS"
+promote_first_pool_key "DASHSCOPE_API_KEY"    "DASHSCOPE_API_KEYS"
+promote_first_pool_key "GMI_API_KEY"          "GMI_API_KEYS"
+promote_first_pool_key "TOKENHUB_API_KEY"     "TOKENHUB_API_KEYS"
+
 # ── Build config ──
 python3 - <<'PY'
 import os
@@ -288,11 +323,19 @@ path.chmod(0o600)
 PY
 
 # ── Startup Summary ──
+HERMES_RUNTIME_VERSION="$(/opt/hermes/.venv/bin/hermes --version 2>/dev/null | awk '{print $NF; exit}' || true)"
 echo ""
+if [ -n "${HERMES_RUNTIME_VERSION:-}" ]; then
+  echo "Version   : ${HERMES_RUNTIME_VERSION}"
+fi
 echo "Model     : ${MODEL_FOR_CONFIG:-unset}"
 echo "Provider  : ${PROVIDER_FOR_CONFIG:-unset}"
 if [ -n "${TELEGRAM_BOT_TOKEN:-}" ]; then
-  echo "Telegram  : enabled"
+  if [ -n "${TELEGRAM_WEBHOOK_URL:-}" ]; then
+    echo "Telegram  : webhook"
+  else
+    echo "Telegram  : polling"
+  fi
 else
   echo "Telegram  : not configured"
 fi
@@ -304,17 +347,22 @@ fi
 if [ -n "${CLOUDFLARE_PROXY_URL:-}" ]; then
   echo "Proxy     : ${CLOUDFLARE_PROXY_URL}"
 fi
+echo "Routes    : /app/ (Hermes UI), /terminal/ (JupyterLab)"
 echo "Dashboard : http://127.0.0.1:${DASHBOARD_PORT}"
 echo "Gateway   : http://127.0.0.1:${GATEWAY_API_PORT}"
 echo ""
 
-# ── JupyterLab terminal (on by default, uses GATEWAY_TOKEN) ──
+# ── JupyterLab terminal (on by default when GATEWAY_TOKEN is set) ──
+JUPYTER_PID=""
 start_jupyter() {
   if [ "${DEV_MODE:-true}" = "false" ]; then
     echo "JupyterLab disabled (DEV_MODE=false)."
     return 0
   fi
-  # Use JUPYTER_TOKEN if set, otherwise fall back to GATEWAY_TOKEN
+  # Guard: skip if already running
+  if [ -n "${JUPYTER_PID:-}" ] && kill -0 "$JUPYTER_PID" 2>/dev/null; then
+    return 0
+  fi
   local token="${JUPYTER_TOKEN:-${API_SERVER_KEY:-}}"
   if [ -z "$token" ]; then
     echo "WARNING: No GATEWAY_TOKEN or JUPYTER_TOKEN set — JupyterLab skipped (terminal would be unauthenticated)." >&2
@@ -329,7 +377,7 @@ start_jupyter() {
   local root_dir="${JUPYTER_ROOT_DIR:-$HERMES_HOME/workspace}"
   mkdir -p "$root_dir"
   ln -sfn "$HERMES_HOME" "$root_dir/HuggingMes" 2>/dev/null || true
-  echo "Starting JupyterLab terminal on port 8888 (path: /terminal/) root: $root_dir"
+  echo "Starting JupyterLab terminal on port 8888 (root: $root_dir)"
   "$VENV_PYTHON" -m jupyterlab \
     --ip 127.0.0.1 \
     --port 8888 \
@@ -355,6 +403,8 @@ start_jupyter() {
 }
 
 # ── Trap SIGTERM for graceful shutdown ──
+SYNC_LOOP_PID=""
+DASHBOARD_PID=""
 graceful_shutdown() {
   echo "Shutting down HuggingMes..."
   if [ -n "${HF_TOKEN:-}" ]; then
@@ -365,6 +415,283 @@ graceful_shutdown() {
 }
 trap graceful_shutdown SIGTERM SIGINT
 
+# ── Shell capture wrappers ──
+# Written to ~/.bashrc so terminal installs are recorded in workspace/startup.sh
+# and replayed on next boot — packages survive Space restarts.
+if [ ! -f "$STARTUP_FILE" ]; then
+  touch "$STARTUP_FILE"
+  chmod +x "$STARTUP_FILE"
+  echo "Created workspace/startup.sh"
+fi
+cat > "$HOME/.bashrc" << 'BASHRC'
+export PATH="/opt/hermes/.venv/bin:/opt/data/.local/bin:$PATH"
+export DEBIAN_FRONTEND="${DEBIAN_FRONTEND:-noninteractive}"
+if [ -z "${PS1:-}" ] || [ "$PS1" = "$ " ]; then
+  export PS1="\u@\h:\w\$ "
+fi
+
+HERMES_HOME="${HERMES_HOME:-/opt/data}"
+STARTUP_FILE="$HERMES_HOME/workspace/startup.sh"
+
+_hm_append() {
+  [ "${HUGGINGMES_CAPTURE_DISABLE:-0}" = "1" ] && return 0
+  local line="$*"
+  mkdir -p "$(dirname "$STARTUP_FILE")"
+  touch "$STARTUP_FILE"
+  chmod +x "$STARTUP_FILE" 2>/dev/null || true
+  grep -qxF "$line" "$STARTUP_FILE" 2>/dev/null || echo "$line" >> "$STARTUP_FILE"
+}
+_hm_quote_args() {
+  local quoted=()
+  local arg
+  for arg in "$@"; do
+    printf -v arg '%q' "$arg"
+    quoted+=("$arg")
+  done
+  printf '%s' "${quoted[*]}"
+}
+_hm_append_cmd() {
+  local cmd="$1"
+  shift
+  local args
+  args=$(_hm_quote_args "$@")
+  if [ -n "$args" ]; then
+    _hm_append "$cmd $args"
+  else
+    _hm_append "$cmd"
+  fi
+}
+_hm_args_without_flags() {
+  local out=()
+  for arg in "$@"; do
+    case "$arg" in
+      ''|-|--*|-*) ;;
+      *) out+=("$arg") ;;
+    esac
+  done
+  printf '%s\n' "${out[@]}"
+}
+_hm_has_install_targets() {
+  local item
+  while IFS= read -r item; do
+    [ -n "$item" ] && return 0
+  done <<EOF
+$(_hm_args_without_flags "$@")
+EOF
+  return 1
+}
+_hm_has_arg() {
+  local needle="$1"
+  shift
+  for arg in "$@"; do
+    [ "$arg" = "$needle" ] && return 0
+  done
+  return 1
+}
+_hm_can_sudo_apt() {
+  command -v sudo >/dev/null 2>&1 && sudo -n apt-get --version >/dev/null 2>&1
+}
+_hm_apt_install() {
+  if [ "$(id -u)" -eq 0 ]; then
+    command apt-get update && command apt-get install -y "$@"
+  elif _hm_can_sudo_apt; then
+    sudo apt-get update && sudo apt-get install -y "$@"
+  else
+    echo "Error: apt install needs root." >&2
+    return 1
+  fi
+}
+apt-get() {
+  case "${1:-}" in
+    install)
+      shift
+      _hm_apt_install "$@"
+      local rc=$?
+      if [ $rc -eq 0 ]; then
+        _hm_has_install_targets "$@" && _hm_append_cmd "sudo apt-get update && sudo apt-get install -y" "$@"
+      fi
+      return $rc
+      ;;
+    update)
+      if [ "$(id -u)" -eq 0 ]; then command apt-get "$@"
+      elif _hm_can_sudo_apt; then sudo apt-get "$@"
+      else command apt-get "$@"; fi
+      return $?
+      ;;
+    *) command apt-get "$@"; return $? ;;
+  esac
+}
+apt() {
+  case "${1:-}" in
+    install)
+      shift
+      _hm_apt_install "$@"
+      local rc=$?
+      if [ $rc -eq 0 ]; then
+        _hm_has_install_targets "$@" && _hm_append_cmd "sudo apt-get update && sudo apt-get install -y" "$@"
+      fi
+      return $rc
+      ;;
+    update)
+      if [ "$(id -u)" -eq 0 ]; then command apt "$@"
+      elif _hm_can_sudo_apt; then sudo apt "$@"
+      else command apt "$@"; fi
+      return $?
+      ;;
+    *) command apt "$@"; return $? ;;
+  esac
+}
+pip() {
+  command pip "$@"
+  local rc=$?
+  if [ $rc -eq 0 ] && [ "${1:-}" = "install" ] \
+      && ! _hm_has_arg -r "${@:2}" && ! _hm_has_arg --requirement "${@:2}" \
+      && _hm_has_install_targets "${@:2}"; then
+    _hm_append_cmd "pip install" "${@:2}"
+  fi
+  return $rc
+}
+pip3() {
+  command pip3 "$@"
+  local rc=$?
+  if [ $rc -eq 0 ] && [ "${1:-}" = "install" ] \
+      && ! _hm_has_arg -r "${@:2}" && ! _hm_has_arg --requirement "${@:2}" \
+      && _hm_has_install_targets "${@:2}"; then
+    _hm_append_cmd "pip install" "${@:2}"
+  fi
+  return $rc
+}
+uv() {
+  command uv "$@"
+  local rc=$?
+  if [ $rc -eq 0 ] && [ "${1:-}" = "pip" ] && [ "${2:-}" = "install" ] \
+      && ! _hm_has_arg -r "${@:3}" && ! _hm_has_arg --requirements "${@:3}" \
+      && _hm_has_install_targets "${@:3}"; then
+    _hm_append_cmd "uv pip install" "${@:3}"
+  fi
+  return $rc
+}
+npm() {
+  command npm "$@"
+  local rc=$?
+  if [ $rc -eq 0 ] && { [ "${1:-}" = "install" ] || [ "${1:-}" = "i" ]; } && { [ "${2:-}" = "-g" ] || [ "${2:-}" = "--global" ]; } && _hm_has_install_targets "${@:3}"; then
+    _hm_append_cmd "npm install -g" "${@:3}"
+  fi
+  return $rc
+}
+hermes() {
+  command hermes "$@"
+  local rc=$?
+  if [ $rc -eq 0 ] && [ "${1:-}" = "plugins" ] && [ "${2:-}" = "install" ] && _hm_has_install_targets "${@:3}"; then
+    _hm_append_cmd "hermes plugins install" "${@:3}"
+  fi
+  return $rc
+}
+BASHRC
+cat > "$HOME/.profile" << 'PROFILE'
+[ -n "${BASH_VERSION:-}" ] && [ -f ~/.bashrc ] && . ~/.bashrc
+PROFILE
+echo "Shell capture wrappers ready."
+
+# ── Optional package installs from HF Variables/Secrets ──
+HM_STARTUP_FAILURES=0
+
+if [ -n "${HUGGINGMES_APT_PACKAGES:-}" ]; then
+  echo "Installing apt packages from HUGGINGMES_APT_PACKAGES..."
+  read -r -a HM_APT_PACKAGES <<< "$HUGGINGMES_APT_PACKAGES"
+  if command -v sudo >/dev/null 2>&1; then
+    if sudo apt-get update && sudo apt-get install -y "${HM_APT_PACKAGES[@]}"; then
+      echo "HUGGINGMES_APT_PACKAGES install complete."
+    else
+      HM_STARTUP_FAILURES=$((HM_STARTUP_FAILURES + 1))
+      echo "ERROR: HUGGINGMES_APT_PACKAGES install failed: ${HUGGINGMES_APT_PACKAGES}" >&2
+    fi
+  elif [ "$(id -u)" -eq 0 ]; then
+    if apt-get update && apt-get install -y "${HM_APT_PACKAGES[@]}"; then
+      echo "HUGGINGMES_APT_PACKAGES install complete."
+    else
+      HM_STARTUP_FAILURES=$((HM_STARTUP_FAILURES + 1))
+      echo "ERROR: HUGGINGMES_APT_PACKAGES install failed: ${HUGGINGMES_APT_PACKAGES}" >&2
+    fi
+  else
+    HM_STARTUP_FAILURES=$((HM_STARTUP_FAILURES + 1))
+    echo "ERROR: root/sudo unavailable; HUGGINGMES_APT_PACKAGES skipped" >&2
+  fi
+fi
+
+if [ -n "${HUGGINGMES_PIP_PACKAGES:-}" ]; then
+  echo "Installing Python packages from HUGGINGMES_PIP_PACKAGES..."
+  read -r -a HM_PIP_PACKAGES <<< "$HUGGINGMES_PIP_PACKAGES"
+  if /opt/hermes/.venv/bin/pip install "${HM_PIP_PACKAGES[@]}"; then
+    echo "HUGGINGMES_PIP_PACKAGES install complete."
+  else
+    HM_STARTUP_FAILURES=$((HM_STARTUP_FAILURES + 1))
+    echo "ERROR: HUGGINGMES_PIP_PACKAGES install failed: ${HUGGINGMES_PIP_PACKAGES}" >&2
+  fi
+fi
+
+if [ -n "${HUGGINGMES_NPM_PACKAGES:-}" ]; then
+  echo "Installing npm packages from HUGGINGMES_NPM_PACKAGES..."
+  read -r -a HM_NPM_PACKAGES <<< "$HUGGINGMES_NPM_PACKAGES"
+  if npm install -g "${HM_NPM_PACKAGES[@]}"; then
+    echo "HUGGINGMES_NPM_PACKAGES install complete."
+  else
+    HM_STARTUP_FAILURES=$((HM_STARTUP_FAILURES + 1))
+    echo "ERROR: HUGGINGMES_NPM_PACKAGES install failed: ${HUGGINGMES_NPM_PACKAGES}" >&2
+  fi
+fi
+
+# ── Arbitrary startup script (HUGGINGMES_RUN) ──
+# Supports plain bash or base64-encoded scripts (prefix with base64: or b64:).
+# Example: HUGGINGMES_RUN="pip install pandas && npm install -g typescript"
+# Example: HUGGINGMES_RUN="base64:$(base64 -w0 setup.sh)"
+hm_run_startup_auto() {
+  local payload="$1"
+  [ -n "$payload" ] || return 0
+  local script_file
+  script_file=$(mktemp "/tmp/huggingmes-startup.XXXXXX.sh")
+  {
+    echo 'export HUGGINGMES_CAPTURE_DISABLE=1'
+    echo '[ -f ~/.bashrc ] && . ~/.bashrc'
+    if [[ "$payload" == base64:* ]] || [[ "$payload" == b64:* ]]; then
+      printf '%s' "${payload#*:}" | base64 -d
+    else
+      printf '%s\n' "$payload"
+    fi
+  } > "$script_file"
+  chmod 700 "$script_file"
+  echo "[startup:HUGGINGMES_RUN] running script"
+  set +e
+  bash "$script_file"
+  local rc=$?
+  set -e
+  rm -f "$script_file"
+  if [ $rc -eq 0 ]; then
+    echo "[startup:HUGGINGMES_RUN] ok"
+  else
+    HM_STARTUP_FAILURES=$((HM_STARTUP_FAILURES + 1))
+    echo "ERROR: HUGGINGMES_RUN script failed (exit ${rc})" >&2
+  fi
+}
+
+if [ -n "${HUGGINGMES_RUN:-}" ]; then
+  hm_run_startup_auto "$HUGGINGMES_RUN"
+fi
+
+# ── Run workspace startup script ──
+# Replays install commands recorded by the shell wrappers from previous sessions.
+if [ -s "$STARTUP_FILE" ]; then
+  echo "Running workspace/startup.sh..."
+  set +e
+  HUGGINGMES_CAPTURE_DISABLE=1 bash -l "$STARTUP_FILE"
+  set -e
+  echo "Workspace startup script complete."
+fi
+
+if [ "$HM_STARTUP_FAILURES" -gt 0 ]; then
+  echo "Warning: ${HM_STARTUP_FAILURES} startup step(s) failed. Check logs above." >&2
+fi
+
 # ── Start background services ──
 node "$APP_DIR/health-server.js" &
 HEALTH_PID=$!
@@ -383,47 +710,101 @@ urllib.request.urlopen(req, timeout=10).read()
 PY
 fi
 
-echo "Launching Hermes dashboard on 127.0.0.1:${DASHBOARD_PORT}..."
-(hermes dashboard --host 127.0.0.1 --insecure 2>&1 | tee -a "$HERMES_HOME/logs/dashboard.log") &
-DASHBOARD_PID=$!
+# ── Launch dashboard once (restarts if it dies) ──
+start_dashboard_once() {
+  if [ -n "${DASHBOARD_PID:-}" ] && kill -0 "$DASHBOARD_PID" 2>/dev/null; then
+    return 0
+  fi
+  echo "Launching Hermes dashboard on 127.0.0.1:${DASHBOARD_PORT}..."
+  (hermes dashboard --host 127.0.0.1 --insecure 2>&1 | tee -a "$HERMES_HOME/logs/dashboard.log") &
+  DASHBOARD_PID=$!
+}
+
+# ── Start sync loop once — survives gateway restarts ──
+start_background_sync_once() {
+  [ -n "${HF_TOKEN:-}" ] || return 0
+  if [ -n "${SYNC_LOOP_PID:-}" ] && kill -0 "$SYNC_LOOP_PID" 2>/dev/null; then
+    return 0
+  fi
+  python3 -u "$APP_DIR/hermes-sync.py" loop &
+  SYNC_LOOP_PID=$!
+}
 
-# ── Launch gateway ──
-echo "Launching Hermes gateway..."
-(hermes gateway run 2>&1 | tee -a "$HERMES_HOME/logs/gateway.log") &
-GATEWAY_PID=$!
+start_dashboard_once
+start_jupyter
 
+# ── Gateway restart loop ──
+GATEWAY_RESTART_DELAY="${GATEWAY_RESTART_DELAY:-5}"
+GATEWAY_MAX_RESTARTS="${GATEWAY_MAX_RESTARTS:-0}"
+GATEWAY_RESTART_COUNT=0
 GATEWAY_READY_TIMEOUT="${GATEWAY_READY_TIMEOUT:-120}"
-ready=false
-for ((i=0; i<GATEWAY_READY_TIMEOUT; i++)); do
-  if (echo > "/dev/tcp/127.0.0.1/${GATEWAY_API_PORT}") 2>/dev/null; then
-    ready=true
-    break
+
+while true; do
+  # Monitor health-server — restart if it died unexpectedly
+  if [ -n "${HEALTH_PID:-}" ] && ! kill -0 "$HEALTH_PID" 2>/dev/null; then
+    echo "Warning: health-server exited (PID $HEALTH_PID dead); restarting..."
+    node "$APP_DIR/health-server.js" &
+    HEALTH_PID=$!
+    echo "Health server restarted (PID: $HEALTH_PID)"
   fi
-  if ! kill -0 "$GATEWAY_PID" 2>/dev/null; then
-    break
+
+  # Monitor Hermes dashboard — restart if it died unexpectedly
+  if [ -n "${DASHBOARD_PID:-}" ] && ! kill -0 "$DASHBOARD_PID" 2>/dev/null; then
+    echo "Warning: Hermes dashboard exited; restarting..."
+    start_dashboard_once
   fi
-  sleep 1
-done
 
-if [ "$ready" != "true" ]; then
-  echo ""
-  echo "Hermes gateway failed to expose the API health port. Last 40 log lines:"
-  echo "----------------------------------------"
-  tail -40 "$HERMES_HOME/logs/gateway.log" || true
-  exit 1
-fi
+  # Monitor JupyterLab — restart if it died unexpectedly
+  if [ "${DEV_MODE:-true}" != "false" ] && [ -n "${JUPYTER_PID:-}" ] && ! kill -0 "$JUPYTER_PID" 2>/dev/null; then
+    echo "Warning: JupyterLab exited (PID $JUPYTER_PID dead); restarting..."
+    unset JUPYTER_PID
+    start_jupyter
+  fi
 
-if [ -n "${HF_TOKEN:-}" ]; then
-  python3 -u "$APP_DIR/hermes-sync.py" loop &
-fi
+  echo "Launching Hermes gateway..."
+  (hermes gateway run 2>&1 | tee -a "$HERMES_HOME/logs/gateway.log") &
+  GATEWAY_PID=$!
+
+  ready=false
+  for ((i=0; i<GATEWAY_READY_TIMEOUT; i++)); do
+    if (echo > "/dev/tcp/127.0.0.1/${GATEWAY_API_PORT}") 2>/dev/null; then
+      ready=true
+      break
+    fi
+    if ! kill -0 "$GATEWAY_PID" 2>/dev/null; then
+      break
+    fi
+    sleep 1
+  done
+
+  if [ "$ready" != "true" ]; then
+    echo ""
+    echo "Hermes gateway failed to expose the API health port. Last 40 log lines:"
+    echo "----------------------------------------"
+    tail -40 "$HERMES_HOME/logs/gateway.log" || true
+    exit 1
+  fi
 
-start_jupyter
+  # Start sync loop (only once — shared across all gateway restarts)
+  start_background_sync_once
 
-wait "$GATEWAY_PID"
+  set +e
+  wait "$GATEWAY_PID"
+  GATEWAY_EXIT_CODE=$?
+  set -e
 
-# Gateway exited (e.g. user restarted from Hermes UI). Sync before container dies.
-# SIGTERM path is handled by graceful_shutdown trap above; this covers natural exit.
-if [ -n "${HF_TOKEN:-}" ]; then
-  echo "Gateway exited — syncing state before shutdown..."
-  python3 "$APP_DIR/hermes-sync.py" sync-once || echo "Warning: final sync failed."
-fi
+  # Sync state before restart
+  if [ -n "${HF_TOKEN:-}" ]; then
+    echo "Gateway exited — syncing state before restart..."
+    python3 "$APP_DIR/hermes-sync.py" sync-once || echo "Warning: sync failed."
+  fi
+
+  GATEWAY_RESTART_COUNT=$((GATEWAY_RESTART_COUNT + 1))
+  if [ "$GATEWAY_MAX_RESTARTS" != "0" ] && [ "$GATEWAY_RESTART_COUNT" -ge "$GATEWAY_MAX_RESTARTS" ]; then
+    echo "Gateway exited (code ${GATEWAY_EXIT_CODE}); restart limit (${GATEWAY_MAX_RESTARTS}) reached."
+    exit "$GATEWAY_EXIT_CODE"
+  fi
+
+  echo "Gateway exited (code ${GATEWAY_EXIT_CODE}); restarting in ${GATEWAY_RESTART_DELAY}s..."
+  sleep "$GATEWAY_RESTART_DELAY"
+done