Stabilize RAGForge v1.4.1 evaluation quotas and runtime API view

README.md

@@ -10,25 +10,21 @@ pinned: false
 
 # RAGForge
 
-**RAGForge v1.4 - a production-style, portfolio-ready agentic RAG system for Hugging Face Spaces.**
+**RAGForge v1.4.1 - a production-style, portfolio-ready agentic RAG system for Hugging Face Spaces.**
 
 RAGForge combines hybrid document retrieval, source-level/hierarchical retrieval, semantic query planning, corrective RAG, Self-RAG-style verification, Text2SQL and an “Ask-the-Web” research path in one CPU-friendly application. The default LLM is **Google Gemini 3.5 Flash-Lite**; the UI also exposes Gemini 3.1 Flash-Lite and stronger Flash models.
 
 
-## What is new in v1.4
+## What is new in v1.4.1
 
-- **Visible query state** - clicking Ask immediately changes the button to `Processing...`, disables repeat submissions, and shows a dedicated status line until the answer arrives.
-- **Visible evaluation state** - Quick/Standard/Deep runs disable the evaluation button and show an explicit running message, preventing accidental duplicate benchmark runs.
-- **Correct source-level retrieval metrics** - repeated chunks from the same file are deduplicated before source AP/MRR/nDCG are computed, so AP@5 is mathematically bounded by 1.0.
-- **Richer retrieval diagnostics** - evaluation now reports source Hit@1, Recall@5, MRR, AP@5, nDCG@5 and duplicate-source rate.
-- **Cache-bypassed benchmark timing** - evaluation disables response caching and chat-history mutation so Standard and Deep latency reflect actual pipeline execution instead of cached answers.
-- **Quality-gated grading** - a strong weighted average can no longer hide a weak subsystem such as Text2SQL, routing or citation quality; critical failures cap the letter grade.
-- **Calibrated Deep judge** - Gemini judge citation scores are bounded by deterministic citation validity/coverage, so an uncited answer cannot receive perfect citation-support credit.
-- **Improved structured-data routing** - the semantic planner now treats direct table lookups, filtering, sorting, min/max and aggregation as SQL-capable tasks while keeping task type separate from route.
-- **Expanded transparent benchmark** - the v1.4 demo benchmark adds multi-source QA and more structured-data planner/SQL cases.
-- **Evaluation diagnostics** - the report identifies weak subsystems and highlights quality/latency tradeoffs such as a reranker that adds latency without improving source ranking.
-- **Interactive Architecture + API tab** - live workspace/model snapshot, LangGraph node reference, endpoint table, copy-ready curl examples, Swagger/OpenAPI links and evaluation architecture are now visible in the UI.
-- **v1.1/v1.2 foundations retained** - semantic planning, source-profile/hierarchical/global retrieval, corrective RAG, conditional web use, browser session continuity, lazy demo recovery and explicit abstention remain in place.
+- **Quota-safe evaluation** - Standard/Deep default to a conservative 12 Gemini requests/minute and share a rolling request ledger with recent interactive calls from the same running Space.
+- **429-aware retry behavior** - surfaced Gemini rate-limit errors honor provider retry guidance before bounded retries instead of immediately creating another burst. Structured-output fallbacks no longer issue a second API call on transient 429/5xx failures.
+- **Fewer benchmark model calls** - Text2SQL execution now uses one Gemini call per case because SQL routing is already evaluated separately; Deep judging uses a representative labeled sample rather than re-judging every answer.
+- **Pacing-aware latency** - pipeline/planner latency excludes deliberate quota-wait time, while wall latency and pacing wait are reported separately.
+- **Evaluation request telemetry** - score card and raw report show target RPM, Gemini requests issued, deliberate pacing wait and surfaced 429 retries.
+- **Architecture + API runtime fix** - `Refresh runtime view` now returns the live workspace snapshot and copy-ready curl examples instead of failing in the UI callback.
+- **Score card wording cleanup** - visible evaluation copy consistently uses `score card`.
+- **v1.4 foundations retained** - bounded source metrics, cache-bypassed benchmarking, quality gates, citation-aware Deep judging, improved table routing and interactive architecture/API documentation remain in place.
 
 ## The retrieval philosophy
 
@@ -234,7 +230,8 @@ curl -X POST http://localhost:7860/api/v1/evaluate/demo \
   -d '{
     "session_id": "SESSION_ID",
     "level": "Standard",
-    "model": "gemini-3.5-flash-lite"
+    "model": "gemini-3.5-flash-lite",
+    "target_rpm": 12
   }'
 ```
 
@@ -261,15 +258,17 @@ The built-in **Evaluation** tab is now a layered benchmark instead of a single s
 - web-use precision/recall and unnecessary-web rate
 - corpus-overview source coverage and no-unnecessary-web regressions
 - explicit empty-workspace abstention correctness
-- Text2SQL route + answer checks
-- latency p50/p95, correction rate, runtime web-use rate and estimated LLM calls
+- Text2SQL read-only SQL generation/execution checks; SQL routing is measured separately in the planner suite
+- service latency p50/p95, pacing/wall time, correction rate, runtime web-use rate, request count and estimated LLM calls
 - retrieval ablation comparing Hybrid RRF with Hybrid + local cross-encoder reranking
 
 ### Deep LLM-as-judge metrics
 
-Deep mode adds Gemini scores for **faithfulness, answer relevance, completeness and citation support**. These judge scores are kept separate from deterministic metrics because an LLM judge is probabilistic and should not be treated as ground truth. The metric families mirror common RAG evaluation practice: retrieval quality is evaluated separately from generation faithfulness/relevance.
+Deep mode adds Gemini scores for **faithfulness, answer relevance, completeness and citation support** on a representative labeled sample, reducing free-tier request pressure while retaining diverse judge coverage. These judge scores are kept separate from deterministic metrics because an LLM judge is probabilistic and should not be treated as ground truth. The metric families mirror common RAG evaluation practice: retrieval quality is evaluated separately from generation faithfulness/relevance.
 
-The UI exposes **Quick**, **Standard** and **Deep** modes and renders a scorecard plus per-layer tables, with the full report still available as JSON. The benchmark is intentionally small and corpus-specific; it is a regression/architecture-validation suite, not a claim of general RAG benchmark performance.
+The UI exposes **Quick**, **Standard** and **Deep** modes and renders a score card plus per-layer tables, with the full report still available as JSON. The benchmark is intentionally small and corpus-specific; it is a regression/architecture-validation suite, not a claim of general RAG benchmark performance.
+
+Evaluation defaults to **quota-safe pacing at 12 RPM**. The active Gemini limit is project/model specific, so use the value shown for your project in Google AI Studio and set the evaluation target below it. A Standard run normally uses fewer model calls than v1.4 because Text2SQL no longer duplicates routing and answer-generation work; Deep additionally judges only a representative subset.
 
 ## Model and dependency note
 
@@ -302,6 +301,7 @@ This public-demo build intentionally uses per-session ephemeral storage, embedde
 - `docs/UX_LIFECYCLE.md` - browser/session lifecycle, lazy demo initialization and indexing UX
 - `docs/MIGRATION_1.3.md` - v1.2 to v1.3 upgrade notes
 - `docs/MIGRATION_1.4.md` - v1.3 to v1.4 upgrade notes
+- `docs/MIGRATION_1.4.1.md` - quota-safe evaluation and runtime-view stabilization patch
 - `docs/ARCHITECTURE_API.md` - live architecture/API surface and endpoint examples
 - `docs/MIGRATION_1.2.md` - v1.1 to v1.2 upgrade notes
 - `docs/architecture.mmd` - Mermaid architecture source

SECURITY.md

@@ -15,7 +15,7 @@ RAGForge is a hardened **portfolio/demo** application, not a compliance-certifie
 | Cross-user retrieval leakage | per-session workspace, vector index, DuckDB database, history, cache namespace, and session TTL |
 | Browser session identifier | only an opaque workspace ID is stored in `gr.BrowserState`; document text, embeddings, API keys and SQL data remain server-side |
 | Concurrent index corruption | per-workspace re-entrant lock serializes ingestion/query mutations; shared cache has its own lock |
-| API quota abuse | per-IP sliding-window query limiter in UI and REST query endpoint; optional REST Bearer token |
+| API quota abuse | per-IP sliding-window query limiter in UI/REST; optional REST Bearer token; evaluation uses a separate rolling per-key/per-model Gemini request budget with configurable RPM pacing and bounded 429 backoff |
 | Secret leakage | `.env` ignored; secrets are expected through Hugging Face Space Secrets or user-entered key |
 | Unbounded context | chunk/session limits, top-k limits, compact/truncated corpus manifest, source truncation before generation |
 | Accidental external-data leakage | semantic planner separates corpus/external/mixed scope; web fallback requires both permission and semantic relevance; corpus-only retrieval failure can abstain rather than automatically search the web |
@@ -27,7 +27,7 @@ RAGForge is a hardened **portfolio/demo** application, not a compliance-certifie
 - Browser persistence does not make the corpus durable. A container restart invalidates the server-side workspace even if the browser still has the old opaque ID; demo mode can rebuild bundled files, but custom uploads must be re-indexed.
 - The demo uses in-process sessions and one application process. Multi-replica deployments need durable tenant/session state and tenant filters enforced at the storage layer.
 - Basic prompt-injection detection is heuristic. Source-profile excerpts are derived from untrusted files and therefore remain an indirect-injection surface even with filtering/system instructions. Treat these controls as defense-in-depth, not a proof of safety.
-- Free Gemini API tiers may have different data-use terms from paid tiers. Do not put confidential documents into a public demo or a provider tier whose privacy terms do not meet your requirements.
+- Free Gemini API tiers may have different data-use terms from paid tiers. Do not put confidential documents into a public demo or a provider tier whose privacy terms do not meet your requirements. Active provider quotas can also change or vary by project/model; quota-safe pacing reduces burst failures but cannot create quota that the provider has not granted.
 
 ## Reporting
 

docs/ARCHITECTURE_API.md

@@ -1,4 +1,4 @@
-# Architecture and API - v1.4
+# Architecture and API - v1.4.1
 
 ## Runtime architecture
 
@@ -35,7 +35,7 @@ The Architecture + API tab exposes the responsibilities of each graph node in a
 - table count,
 - configured generation/embedding/reranker/search models.
 
-It also generates curl examples using the current browser workspace ID.
+It also generates curl examples using the current browser workspace ID. v1.4.1 fixes the runtime callback so the live snapshot, JSON payload and curl examples are returned together.
 
 ## REST surface
 
@@ -79,10 +79,15 @@ curl -X POST http://localhost:7860/api/v1/evaluate/demo \
   -d '{
     "session_id": "SESSION_ID",
     "level": "Standard",
-    "model": "gemini-3.5-flash-lite"
+    "model": "gemini-3.5-flash-lite",
+    "target_rpm": 12
   }'
 ```
 
 ## Storage lifecycle
 
 Standard Hugging Face Space disk is ephemeral for this deployment design. Browser state stores only the opaque workspace ID. A normal refresh can reconnect while the process lives; a container restart removes in-memory indexes and custom uploads must be re-indexed. Bundled demo data can be lazily rebuilt.
+
+## Evaluation quota controls
+
+`POST /api/v1/evaluate/demo` accepts `target_rpm`. The UI defaults to 12 RPM for quota-safe portfolio/free-tier runs. The benchmark uses one shared rolling request budget across planner, generation, Text2SQL and Deep-judge calls, and the raw report exposes request/pacing telemetry.

docs/EVALUATION.md

@@ -1,8 +1,8 @@
-# Evaluation architecture - v1.4
+# Evaluation architecture - v1.4.1
 
 RAGForge evaluates retrieval, orchestration, generation, structured-data behavior and runtime efficiency separately. The benchmark is intentionally small and transparent; it is a regression suite for the bundled demo corpus, not a claim about general RAG performance.
 
-## Why v1.4 changed the evaluator
+## Why v1.4/v1.4.1 changed the evaluator
 
 The v1.3 benchmark surfaced four evaluator/system issues during real Hugging Face runs:
 
@@ -13,6 +13,8 @@ The v1.3 benchmark surfaced four evaluator/system issues during real Hugging Fac
 
 v1.4 fixes all four.
 
+v1.4.1 then hardens the benchmark for free-tier API quotas and fixes a runtime-view packaging defect found during a deployed Hugging Face test. It adds rolling RPM pacing, provider-aware 429 backoff, request telemetry, lower-call Text2SQL evaluation and sampled Deep judging.
+
 ## Benchmark data
 
 The labels live in `evals/demo_benchmark.json` and include:
@@ -41,7 +43,7 @@ This is the recommended default for portfolio demonstrations because the core me
 
 ### Deep
 
-Runs Standard and additionally asks Gemini to evaluate selected generated answers for:
+Runs Standard and additionally asks Gemini to evaluate a representative labeled subset of generated answers for:
 
 - faithfulness,
 - answer relevance,
@@ -49,7 +51,7 @@ Runs Standard and additionally asks Gemini to evaluate selected generated answer
 - citation support,
 - overall quality and pass/fail.
 
-The Deep judge is auxiliary. Its citation score is conservatively bounded by deterministic citation validity and coverage, so an answer with no citations cannot receive perfect citation-support credit.
+The Deep judge is auxiliary. v1.4.1 samples cases across ordinary QA, policy/operations, NIST, cross-document synthesis and corpus overview instead of judging every answer. Its citation score is conservatively bounded by deterministic citation validity and coverage, so an answer with no citations cannot receive perfect citation-support credit.
 
 ## Cache and history policy
 
@@ -141,12 +143,14 @@ Task labels are intentionally separate from route. For example, a direct value l
 
 ## Text2SQL evaluation
 
-Cases verify both:
+The planner suite already evaluates whether Auto mode chooses `route=sql` and `retrieval_strategy=table`. The Text2SQL component suite therefore avoids duplicating that model call. For each SQL benchmark case it:
 
-- whether Auto mode routes the question to SQL,
-- whether the resulting answer contains the expected computed/filtered result.
+1. gives the DuckDB table schema and user question to Gemini;
+2. validates the generated statement as a single read-only `SELECT`/CTE;
+3. executes it in DuckDB;
+4. checks the computed result against transparent benchmark terms.
 
-The v1.4 benchmark covers min/max-style selection and direct table field lookup.
+This reduces the component test from roughly three model calls per case to one while preserving separate coverage for routing and SQL correctness.
 
 ## Lifecycle abstention
 
@@ -156,15 +160,26 @@ Explicit empty `Documents` and `Data (SQL)` requests must terminate through the
 
 The report includes:
 
-- pipeline latency p50/p95,
-- planner latency p50/p95,
+- pipeline service latency p50/p95,
+- planner service latency p50/p95,
+- per-case wall latency and deliberate pacing wait,
 - total evaluation wall time,
-- mean estimated LLM calls,
-- correction rate,
-- runtime web-use rate,
-- Deep judge mean latency when applicable.
+- actual Gemini requests issued through RAGForge's gateway,
+- configured evaluation RPM target,
+- surfaced 429 retry count and provider-directed retry wait,
+- mean estimated pipeline LLM calls,
+- correction rate and runtime web-use rate,
+- Deep judge latency when applicable.
+
+Service latency subtracts deliberate quota-pacing sleep so the benchmark does not make the normal interactive pipeline look slower merely because the evaluation is being rate-limited. `llm_calls_estimate` remains graph-derived and is useful for pipeline comparisons; `gemini_requests` is the gateway-level count for the evaluation run and can be higher when a provider retry occurs.
+
+## Quota-safe evaluation
 
-`llm_calls_estimate` is derived from executed LangGraph nodes. It is useful for relative comparisons but is not a provider billing record.
+Standard and Deep default to a conservative target of **12 Gemini requests per minute**. The UI exposes this target because active rate limits vary by project/model. The process-local ledger also records recent interactive requests, so starting an evaluation immediately after manual testing does not pretend the previous minute was empty.
+
+When quota-safe pacing is enabled, requests are smoothed across the minute rather than sent in a burst. If Gemini still surfaces a 429, RAGForge parses provider retry guidance when available, waits for that interval (plus small jitter) and performs only bounded retries. Structured-output code does not immediately fall back to a second plain-JSON request on transient 429/5xx errors.
+
+Set the API/GUI target below the active RPM shown for your project. `target_rpm=0` disables deliberate pacing, but bounded provider-aware retries remain.
 
 ## Quality-gated grade
 
@@ -199,6 +214,8 @@ The benchmark can be run programmatically:
 POST /api/v1/evaluate/demo
 ```
 
+The request body accepts `target_rpm` (default `12`; use `0` to disable deliberate pacing).
+
 Benchmark metadata is available without running the benchmark:
 
 ```text

docs/FEATURE_MATRIX.md

@@ -38,7 +38,8 @@
 | Caching | TTL result cache keyed by session/config/corpus version | latency/quota reduction without stale cross-corpus answers |
 | Rate limiting | sliding-window per IP | protects a public shared model key |
 | Observability | Prometheus + semantic plan/evidence/correction/node trace + node time/estimated LLM calls/web/correction flags | makes agent decisions and efficiency inspectable |
-| Evaluation | transparent v1.4 benchmark + bounded source Hit@1/Recall/MRR/AP/nDCG + duplicate-source rate + citations + planner/web policy + Text2SQL + abstention + cache-bypassed latency + calibrated Deep judge | separates retrieval, orchestration and generation failures without mathematically invalid source metrics or cache-distorted latency |
+| Evaluation | transparent v1.4.1 benchmark + bounded source Hit@1/Recall/MRR/AP/nDCG + duplicate-source rate + citations + planner/web policy + one-call Text2SQL component checks + abstention + cache-bypassed/pacing-aware latency + sampled calibrated Deep judge | separates retrieval, orchestration and generation failures while keeping free-tier benchmark request pressure controlled |
+| Evaluation quota control | shared rolling Gemini request ledger, configurable target RPM, provider retry-delay handling and request/pacing telemetry | prevents Standard/Deep benchmark bursts from repeatedly exhausting low free-tier RPM quotas |
 | Quality gates | subsystem thresholds cap the letter grade | prevents a strong weighted average from hiding weak Text2SQL/routing/citation behavior |
 | Evaluation diagnostics | structured warnings/recommendations for citations, planner taxonomy, Text2SQL and reranker tradeoffs | turns benchmark output into actionable engineering feedback |
 | API | FastAPI session/status/ingest/query/evaluate/benchmark-info/health/metrics + Swagger/OpenAPI | usable beyond the UI and introspectable from the Architecture + API tab |

docs/MIGRATION_1.4.1.md

@@ -0,0 +1,31 @@
+# Migration to v1.4.1
+
+v1.4.1 is a stabilization patch over v1.4. It does not change the core retrieval architecture or demo corpus.
+
+## Fixes
+
+- fixes `Architecture + API -> Refresh runtime view`, which could raise `NameError: name 'f' is not defined` because the generated curl-example string was truncated in the v1.4 packaged UI source;
+- changes visible evaluation wording from `scorecard` to `score card`;
+- adds quota-safe Gemini evaluation pacing, defaulting to 12 RPM;
+- records recent interactive requests in the same process-local per-key/per-model request ledger used by evaluation pacing;
+- honors surfaced Gemini 429 retry guidance before bounded retries;
+- prevents structured-output fallback code from immediately issuing another API request after transient 429/5xx failures;
+- reports deliberate pacing wait separately from service/pipeline latency;
+- reduces Text2SQL evaluation from roughly three Gemini calls per case to one by separating planner-routing evaluation from SQL generation/execution evaluation;
+- reduces Deep judge calls to a representative labeled benchmark subset.
+
+## Recommended free-tier setting
+
+Use the active RPM displayed for your API project in Google AI Studio as the source of truth. The UI defaults to 12 RPM, which leaves headroom when a project currently has a 15 RPM limit. Lower the target if the same API key is also receiving traffic outside this Space.
+
+## Upgrade
+
+Apply the v1.4.1 patch over a clean v1.4 tree, then commit normally. The patch does not contain the bundled NIST PDF.
+
+After deployment:
+
+1. run Quick with quota-safe pacing enabled;
+2. run Standard and confirm request/pacing telemetry appears in the score card;
+3. run Deep immediately afterward to test rolling-window continuity;
+4. open `Architecture + API` and click `Refresh runtime view`;
+5. confirm the runtime JSON and curl examples populate without a traceback.

docs/RESUME_BULLETS.md

@@ -10,3 +10,5 @@
 - Hardened public RAG lifecycle UX with browser-persistent workspace IDs, lazy demo re-indexing after ephemeral Space restarts, explicit empty-corpus/insufficient-evidence abstention, staged ingestion progress, and inspectable workspace/evidence traces.
 
 - Instrumented LangGraph traces with node latency, correction/web flags and estimated LLM-call counts; added visible run-state/duplicate-click protection for chat and evaluation, and an interactive Architecture + API view with live workspace metadata, endpoint reference and copy-ready curl examples.
+
+- Added **quota-aware evaluation infrastructure** with a rolling per-model Gemini request budget, provider-guided 429 backoff, pacing-vs-service latency separation, request telemetry, sampled LLM judging, and one-call Text2SQL component checks for reliable free-tier benchmark runs.

docs/SOURCES.md

@@ -50,3 +50,7 @@ No tutorial source code is copied into RAGForge. The requested projects were use
   https://docs.langchain.com/langsmith/evaluate-rag-tutorial
 - Gradio event progress controls - `show_progress=hidden` suppresses the automatic overlay when explicit progress is used  
   https://www.gradio.app/docs/gradio/on
+- Gemini API rate limits - active limits are project/model dependent and visible in Google AI Studio  
+  https://ai.google.dev/gemini-api/docs/rate-limits
+- Gemini troubleshooting - bounded exponential backoff for 429/5xx and retry guidance  
+  https://ai.google.dev/gemini-api/docs/troubleshooting

evals/README.md

@@ -4,12 +4,20 @@
 
 - focused QA labels - expected answer terms and relevant source files,
 - corpus-overview behavior - breadth, source coverage and unnecessary web use,
-- semantic planner behavior - expected route, task, retrieval strategy and whether web access is appropriate.
+- semantic planner behavior - expected route, task, retrieval strategy and whether web access is appropriate,
+- Text2SQL component behavior - validated read-only SQL generation/execution and expected result terms,
+- lifecycle abstention - explicit missing-resource cases that should use zero model calls.
 
 The benchmark is not meant to claim general RAG performance. It is a regression and architecture-validation suite for this demo corpus.
 
 ## Standard vs Deep evaluation
 
-**Standard** uses deterministic labels wherever possible: answer-key terms, source Recall@K, source MRR, citation validity/coverage, route/task/strategy accuracy, web-use precision/recall, overview source coverage and latency/trace efficiency.
+**Standard** uses deterministic labels wherever possible: answer-key terms, source Hit@1/Recall@K/MRR/AP/nDCG, citation validity/coverage, route/task/strategy accuracy, web-use precision/recall, overview source coverage, Text2SQL result checks and latency/trace efficiency.
 
-**Deep** adds an auxiliary Gemini judge for faithfulness, answer relevance, completeness and citation support. Judge scores are reported separately from deterministic metrics because LLM-as-judge evaluation is itself probabilistic.
+**Deep** adds an auxiliary Gemini judge for a representative labeled subset spanning ordinary QA, NIST, cross-document synthesis and corpus overview. Judge scores are reported separately from deterministic metrics because LLM-as-judge evaluation is itself probabilistic.
+
+## Quota-safe execution
+
+v1.4.1 defaults to a 12 RPM evaluation request budget. The process-local request ledger accounts for recent interactive requests made with the same key/model, and surfaced 429 responses honor provider retry guidance before bounded retries. Deliberate pacing time is reported separately from service latency.
+
+The Text2SQL component test uses one model call per case because SQL routing is already evaluated independently in the semantic-planner suite. This avoids duplicating route and answer-generation calls solely for the benchmark.

evals/demo_benchmark.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.4",
+  "version": "1.4.1",
   "description": "Transparent multi-layer benchmark for the bundled RAGForge demo corpus. Labels are source-level, auditable, and include focused QA, cross-document retrieval, planner policy, corpus overview, Text2SQL and lifecycle abstention.",
   "qa_cases": [
     {
@@ -12,7 +12,8 @@
       "reference_answer": "A Sev-1 incident should be acknowledged within 5 minutes.",
       "relevant_sources": [
         "acme_cloud_runbook.md"
-      ]
+      ],
+      "deep_judge": true
     },
     {
       "id": "qa_acme_availability",
@@ -49,7 +50,8 @@
       "reference_answer": "Refunds typically appear 5-10 business days after merchant confirmation.",
       "relevant_sources": [
         "orbitpay_policy.txt"
-      ]
+      ],
+      "deep_judge": true
     },
     {
       "id": "qa_atlas_hybrid_search",
@@ -99,7 +101,8 @@
       "reference_answer": "The four AI RMF core functions are Govern, Map, Measure, and Manage.",
       "relevant_sources": [
         "NIST_AI_RMF_1.0.pdf"
-      ]
+      ],
+      "deep_judge": true
     },
     {
       "id": "qa_cross_doc_escalation",
@@ -112,7 +115,8 @@
       "relevant_sources": [
         "acme_cloud_runbook.md",
         "orbitpay_policy.txt"
-      ]
+      ],
+      "deep_judge": true
     }
   ],
   "overview_cases": [
@@ -123,7 +127,8 @@
       "expected_task": "overview",
       "expected_strategy": "global",
       "web_expected": false,
-      "min_source_coverage": 0.8
+      "min_source_coverage": 0.8,
+      "deep_judge": true
     },
     {
       "id": "overview_documents",

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ragforge"
-version = "1.4.0"
+version = "1.4.1"
 description = "Production-style agentic RAG demo for Hugging Face Spaces"
 requires-python = ">=3.11"
 dependencies = []

src/ragforge/__init__.py

@@ -1,3 +1,3 @@
 """RAGForge: production-style agentic retrieval augmented generation demo."""
 
-__version__ = "1.4.0"
+__version__ = "1.4.1"

src/ragforge/api.py

@@ -29,7 +29,7 @@ def _auth(authorization: Annotated[str | None, Header()] = None) -> None:
 
 
 def create_api() -> FastAPI:
-    app = FastAPI(title="RAGForge API", version="1.4.0")
+    app = FastAPI(title="RAGForge API", version="1.4.1")
 
     @app.get("/api/health")
     def health():
@@ -48,7 +48,9 @@ def create_api() -> FastAPI:
                 "semantic-query-planning", "source-profile-index", "hierarchical-retrieval",
                 "source-balanced-global-retrieval", "hybrid-search", "reranking", "hyde",
                 "corrective-rag", "conditional-web", "self-rag", "text2sql", "ask-the-web",
-                "citations", "guardrails", "layered-evaluation", "retrieval-ablation", "quality-gated-evaluation", "cache-bypassed-benchmarking", "workspace-preflight", "browser-session-continuity", "lazy-demo-recovery", "explicit-abstention"
+                "citations", "guardrails", "layered-evaluation", "retrieval-ablation", "quality-gated-evaluation",
+                "cache-bypassed-benchmarking", "quota-aware-evaluation", "retry-after-backoff",
+                "workspace-preflight", "browser-session-continuity", "lazy-demo-recovery", "explicit-abstention"
             ],
         }
 
@@ -146,7 +148,13 @@ def create_api() -> FastAPI:
                 raise ValueError(
                     "The bundled demo benchmark requires the five bundled demo sources to be indexed in this session."
                 )
-            report = run_demo_eval(ws, api_key=None, model=payload.model, level=payload.level)
+            report = run_demo_eval(
+                ws,
+                api_key=None,
+                model=payload.model,
+                level=payload.level,
+                target_rpm=payload.target_rpm,
+            )
             REQUESTS.labels("evaluate_demo", "ok").inc()
             return report
         except RateLimitExceeded as exc:

src/ragforge/evaluation.py

@@ -8,7 +8,7 @@ from pathlib import Path
 from typing import Any, Callable
 
 from .eval_metrics import answer_key_match, citation_metrics, mean, percentile, safe_div, source_metrics
-from .llm import GeminiGateway
+from .llm import GeminiGateway, RequestPacer
 from .pipeline import RAGEngine
 from .schemas import PipelineConfig
 from .workspace import Workspace
@@ -36,8 +36,12 @@ def demo_benchmark_metadata() -> dict[str, Any]:
         "levels": {
             "Quick": "Small deployment smoke test",
             "Standard": "Full deterministic benchmark plus retrieval ablation",
-            "Deep": "Standard plus calibrated Gemini judge",
+            "Deep": "Standard plus calibrated Gemini judge on a representative labeled sample",
         },
+        "default_target_rpm": 12,
+        "deep_judge_cases": sum(
+            1 for case in benchmark.get("qa_cases", []) + benchmark.get("overview_cases", []) if case.get("deep_judge")
+        ),
         "cache_policy": "Response cache bypassed during benchmark execution",
     }
 
@@ -118,9 +122,13 @@ def _planner_eval(
     total = max(1, len(cases))
     for idx, case in enumerate(cases, start=1):
         progress(start + span * (idx - 1) / total, f"Planner case {idx}/{len(cases)}")
+        wait_before = gateway.request_pacer.total_sleep_seconds() if gateway.request_pacer else 0.0
         began = time.perf_counter()
         plan = gateway.analyze_query(case["question"], manifest, history=None, profile="Balanced")
-        latency = (time.perf_counter() - began) * 1000
+        wall_latency = (time.perf_counter() - began) * 1000
+        wait_after = gateway.request_pacer.total_sleep_seconds() if gateway.request_pacer else wait_before
+        pacing_wait = max(0.0, wait_after - wait_before) * 1000
+        latency = max(0.0, wall_latency - pacing_wait)
         planned_web = plan.web_relevance != "irrelevant" or plan.route in {"web", "hybrid"}
         rows.append(
             {
@@ -138,6 +146,8 @@ def _planner_eval(
                 "expected_web": bool(case["web_expected"]),
                 "planned_web": planned_web,
                 "latency_ms": round(latency, 1),
+                "wall_latency_ms": round(wall_latency, 1),
+                "pacing_wait_ms": round(pacing_wait, 1),
             }
         )
     return rows
@@ -150,6 +160,7 @@ def _judge_row(
     sources: list[dict[str, Any]],
     citations: dict[str, float | int],
 ) -> dict[str, Any]:
+    wait_before = judge.request_pacer.total_sleep_seconds() if judge.request_pacer else 0.0
     began = time.perf_counter()
     judgement = judge.evaluate_rag_answer(
         case["question"],
@@ -159,7 +170,10 @@ def _judge_row(
         citation_validity=float(citations["citation_validity"]),
         citation_coverage=float(citations["citation_coverage"]),
     )
-    judge_latency = (time.perf_counter() - began) * 1000
+    judge_wall_latency = (time.perf_counter() - began) * 1000
+    wait_after = judge.request_pacer.total_sleep_seconds() if judge.request_pacer else wait_before
+    judge_pacing_wait = max(0.0, wait_after - wait_before) * 1000
+    judge_latency = max(0.0, judge_wall_latency - judge_pacing_wait)
     return {
         "judge_faithfulness": round(judgement.faithfulness, 3),
         "judge_answer_relevance": round(judgement.answer_relevance, 3),
@@ -169,6 +183,8 @@ def _judge_row(
         "judge_pass": judgement.pass_,
         "judge_reason": judgement.reason,
         "judge_latency_ms": round(judge_latency, 1),
+        "judge_wall_latency_ms": round(judge_wall_latency, 1),
+        "judge_pacing_wait_ms": round(judge_pacing_wait, 1),
     }
 
 
@@ -178,12 +194,13 @@ def _qa_eval(
     api_key: str | None,
     model: str,
     deep_judge: bool,
+    request_pacer: RequestPacer,
     progress: Callable[[float, str], None],
     start: float,
     span: float,
 ) -> list[dict[str, Any]]:
-    engine = RAGEngine(workspace)
-    judge = GeminiGateway(api_key, model) if deep_judge else None
+    engine = RAGEngine(workspace, request_pacer=request_pacer)
+    judge = GeminiGateway(api_key, model, request_pacer=request_pacer) if deep_judge else None
     cfg = PipelineConfig(
         mode="Documents",
         profile="Fast",
@@ -196,9 +213,12 @@ def _qa_eval(
     total = max(1, len(cases))
     for idx, case in enumerate(cases, start=1):
         progress(start + span * (idx - 1) / total, f"Document QA case {idx}/{len(cases)}")
+        wait_before = request_pacer.total_sleep_seconds()
         began = time.perf_counter()
         result = engine.ask(case["question"], cfg, api_key, use_cache=False, record_history=False)
-        latency = (time.perf_counter() - began) * 1000
+        wall_latency = (time.perf_counter() - began) * 1000
+        pacing_wait = max(0.0, request_pacer.total_sleep_seconds() - wait_before) * 1000
+        latency = max(0.0, wall_latency - pacing_wait)
         returned_sources = _document_sources(result.sources, 5)
         retrieval = source_metrics(returned_sources, case.get("relevant_sources", []))
         citations = citation_metrics(result.answer, result.sources)
@@ -213,9 +233,11 @@ def _qa_eval(
             "citation_coverage": round(float(citations["citation_coverage"]), 3),
             "confidence": round(result.confidence, 3),
             "latency_ms": round(latency, 1),
+            "wall_latency_ms": round(wall_latency, 1),
+            "pacing_wait_ms": round(pacing_wait, 1),
             **efficiency,
         }
-        if judge:
+        if judge and bool(case.get("deep_judge", False)):
             row.update(_judge_row(judge, case, result.answer, result.sources, citations))
         rows.append(row)
     return rows
@@ -227,12 +249,13 @@ def _overview_eval(
     api_key: str | None,
     model: str,
     deep_judge: bool,
+    request_pacer: RequestPacer,
     progress: Callable[[float, str], None],
     start: float,
     span: float,
 ) -> list[dict[str, Any]]:
-    engine = RAGEngine(workspace)
-    judge = GeminiGateway(api_key, model) if deep_judge else None
+    engine = RAGEngine(workspace, request_pacer=request_pacer)
+    judge = GeminiGateway(api_key, model, request_pacer=request_pacer) if deep_judge else None
     cfg = PipelineConfig(
         mode="Auto",
         profile="Balanced",
@@ -245,9 +268,12 @@ def _overview_eval(
     total = max(1, len(cases))
     for idx, case in enumerate(cases, start=1):
         progress(start + span * (idx - 1) / total, f"Corpus overview case {idx}/{len(cases)}")
+        wait_before = request_pacer.total_sleep_seconds()
         began = time.perf_counter()
         result = engine.ask(case["question"], cfg, api_key, use_cache=False, record_history=False)
-        latency = (time.perf_counter() - began) * 1000
+        wall_latency = (time.perf_counter() - began) * 1000
+        pacing_wait = max(0.0, request_pacer.total_sleep_seconds() - wait_before) * 1000
+        latency = max(0.0, wall_latency - pacing_wait)
         plan = result.trace.get("query_plan", {})
         evidence = result.trace.get("evidence", {})
         efficiency = _trace_efficiency(result.trace)
@@ -273,10 +299,12 @@ def _overview_eval(
             "citation_validity": round(float(citations["citation_validity"]), 3),
             "citation_coverage": round(float(citations["citation_coverage"]), 3),
             "latency_ms": round(latency, 1),
+            "wall_latency_ms": round(wall_latency, 1),
+            "pacing_wait_ms": round(pacing_wait, 1),
             "pass": passed,
             **efficiency,
         }
-        if judge:
+        if judge and bool(case.get("deep_judge", False)):
             row.update(_judge_row(judge, case, result.answer, result.sources, citations))
         rows.append(row)
     return rows
@@ -287,33 +315,55 @@ def _sql_eval(
     cases: list[dict[str, Any]],
     api_key: str | None,
     model: str,
+    request_pacer: RequestPacer,
     progress: Callable[[float, str], None],
     start: float,
     span: float,
 ) -> list[dict[str, Any]]:
-    engine = RAGEngine(workspace)
-    cfg = PipelineConfig(mode="Auto", profile="Balanced", model=model, allow_web_fallback=False)
+    """Evaluate Text2SQL generation/execution with one model call per case.
+
+    SQL routing is already measured in the semantic-planner benchmark. Keeping
+    this component test route-independent avoids spending two extra Gemini
+    calls per case just to duplicate planner and answer-generation coverage.
+    """
+    gateway = GeminiGateway(api_key, model, request_pacer=request_pacer)
     rows: list[dict[str, Any]] = []
     total = max(1, len(cases))
     for idx, case in enumerate(cases, start=1):
         progress(start + span * (idx - 1) / total, f"Text2SQL case {idx}/{len(cases)}")
+        wait_before = request_pacer.total_sleep_seconds()
         began = time.perf_counter()
-        result = engine.ask(case["question"], cfg, api_key, use_cache=False, record_history=False)
-        latency = (time.perf_counter() - began) * 1000
-        plan = result.trace.get("query_plan", {})
-        efficiency = _trace_efficiency(result.trace)
+        try:
+            sql, result = workspace.sql.benchmark_query(case["question"], gateway)
+            preview = result.head(200)
+            result_text = preview.to_markdown(index=False) if len(preview) else "(no rows)"
+            matched = answer_key_match(result_text, case)
+            error = ""
+            readonly_validated = True
+        except Exception as exc:
+            sql = ""
+            result = None
+            result_text = ""
+            matched = False
+            error = f"{type(exc).__name__}: {exc}"
+            readonly_validated = False
+        wall_latency = (time.perf_counter() - began) * 1000
+        pacing_wait = max(0.0, request_pacer.total_sleep_seconds() - wait_before) * 1000
+        latency = max(0.0, wall_latency - pacing_wait)
         rows.append(
             {
                 "id": case["id"],
                 "question": case["question"],
-                "expected_route": case.get("expected_route"),
-                "route": plan.get("route"),
-                "route_correct": plan.get("route") == case.get("expected_route"),
-                "task": plan.get("task_type"),
-                "strategy": plan.get("retrieval_strategy"),
-                "answer_key_match": answer_key_match(result.answer, case),
+                "component": "Text2SQL",
+                "answer_key_match": matched,
+                "readonly_validated": readonly_validated,
+                "sql": sql,
+                "rows": int(len(result)) if result is not None else 0,
                 "latency_ms": round(latency, 1),
-                **efficiency,
+                "wall_latency_ms": round(wall_latency, 1),
+                "pacing_wait_ms": round(pacing_wait, 1),
+                "llm_calls_estimate": 1,
+                "error": error,
             }
         )
     return rows
@@ -417,6 +467,19 @@ def _diagnostics(
 ) -> list[dict[str, str]]:
     findings: list[dict[str, str]] = []
 
+    if int(summary.get("rate_limit_retries", 0) or 0) > 0:
+        findings.append(
+            {
+                "severity": "warning",
+                "area": "gemini quota",
+                "finding": (
+                    f"Gemini surfaced {int(summary.get('rate_limit_retries', 0))} rate-limit retry event(s); "
+                    f"provider-directed retry wait was {float(summary.get('rate_limit_sleep_ms', 0.0)) / 1000:.1f}s."
+                ),
+                "recommendation": "Keep quota-safe pacing enabled or lower the target RPM below the active project limit.",
+            }
+        )
+
     if summary.get("citation_coverage", 1.0) < 0.90:
         findings.append(
             {
@@ -437,7 +500,7 @@ def _diagnostics(
             }
         )
     if summary.get("text2sql_pass_rate", 1.0) < 0.90:
-        failed = [row["id"] for row in sql_rows if not (row.get("route_correct") and row.get("answer_key_match"))]
+        failed = [row["id"] for row in sql_rows if not row.get("answer_key_match")]
         findings.append(
             {
                 "severity": "warning",
@@ -481,6 +544,7 @@ def run_demo_eval(
     model: str,
     level: str = "Standard",
     progress_callback: Callable[[float, str], None] | None = None,
+    target_rpm: int = 12,
 ) -> dict[str, Any]:
     """Run the bundled benchmark with response caching disabled.
 
@@ -493,6 +557,7 @@ def run_demo_eval(
     benchmark = _load_benchmark()
     level = level if level in {"Quick", "Standard", "Deep"} else "Standard"
     deep_judge = level == "Deep"
+    request_pacer = RequestPacer(target_rpm=max(0, int(target_rpm)))
 
     qa_cases = benchmark["qa_cases"] if level != "Quick" else benchmark["qa_cases"][:3]
     planner_cases = benchmark["planner_cases"] if level != "Quick" else benchmark["planner_cases"][:5]
@@ -500,11 +565,19 @@ def run_demo_eval(
     sql_cases = benchmark.get("sql_cases", []) if level != "Quick" else benchmark.get("sql_cases", [])[:1]
 
     progress(0.01, "Preparing evaluation")
-    qa_rows = _qa_eval(workspace, qa_cases, api_key, model, deep_judge, progress, 0.03, 0.29)
-    gateway = GeminiGateway(api_key, model)
+    qa_rows = _qa_eval(
+        workspace, qa_cases, api_key, model, deep_judge, request_pacer, progress, 0.03, 0.29
+    )
+    gateway = GeminiGateway(api_key, model, request_pacer=request_pacer)
     planner_rows = _planner_eval(workspace, planner_cases, gateway, progress, 0.34, 0.22)
-    overview_rows = _overview_eval(workspace, overview_cases, api_key, model, deep_judge, progress, 0.58, 0.16)
-    sql_rows = _sql_eval(workspace, sql_cases, api_key, model, progress, 0.75, 0.10) if sql_cases else []
+    overview_rows = _overview_eval(
+        workspace, overview_cases, api_key, model, deep_judge, request_pacer, progress, 0.58, 0.16
+    )
+    sql_rows = (
+        _sql_eval(workspace, sql_cases, api_key, model, request_pacer, progress, 0.75, 0.10)
+        if sql_cases
+        else []
+    )
     progress(0.86, "Checking abstention and retrieval ablations")
     abstention_rows = _abstention_eval()
     ablation_rows = _retrieval_ablation(workspace, qa_cases) if level != "Quick" else []
@@ -524,9 +597,7 @@ def run_demo_eval(
     citation_coverage = mean([float(row["citation_coverage"]) for row in qa_rows + overview_rows])
     overview_pass = mean([float(row["pass"]) for row in overview_rows])
     abstention_accuracy = mean([float(row["pass"]) for row in abstention_rows])
-    sql_accuracy = (
-        mean([float(row["route_correct"] and row["answer_key_match"]) for row in sql_rows]) if sql_rows else 1.0
-    )
+    sql_accuracy = mean([float(row["answer_key_match"]) for row in sql_rows]) if sql_rows else 1.0
 
     deterministic_score = (
         0.20 * answer_accuracy
@@ -556,6 +627,8 @@ def run_demo_eval(
             "judge_latency_mean_ms": mean([float(row.get("judge_latency_ms", 0.0)) for row in judge_rows]),
         }
 
+    pacing_stats = request_pacer.stats()
+
     metrics_for_gate = {
         "planner_route_accuracy": planner_metrics["planner_route_accuracy"],
         "web_use_precision": planner_metrics["web_use_precision"],
@@ -597,6 +670,12 @@ def run_demo_eval(
         "runtime_web_use_rate": round(mean([float(row.get("web_used", False)) for row in all_runtime_rows]), 3),
         "cache_bypassed": True,
         "evaluation_wall_ms": 0.0,
+        "evaluation_target_rpm": int(pacing_stats["target_rpm"]),
+        "gemini_requests": int(pacing_stats["gemini_requests"]),
+        "pacing_sleep_ms": float(pacing_stats["pacing_sleep_ms"]),
+        "rate_limit_retries": int(pacing_stats["rate_limit_retries"]),
+        "rate_limit_sleep_ms": float(pacing_stats["rate_limit_sleep_ms"]),
+        "deep_judge_cases": len(judge_rows),
         **{key: round(value, 3) for key, value in judge_summary.items()},
     }
 
@@ -624,9 +703,18 @@ def run_demo_eval(
                 "reflects real benchmark execution rather than cached answers."
             ),
             "deep_judge": (
-                "Optional Gemini judge for faithfulness, answer relevance, completeness and citation support. "
-                "Citation-support and overall scores are conservatively calibrated against deterministic citation "
-                "validity/coverage so an uncited answer cannot receive perfect citation credit."
+                "Optional Gemini judge for a representative labeled subset of benchmark cases, covering focused QA, "
+                "NIST, cross-document synthesis and corpus overview. Sampling reduces free-tier request pressure while "
+                "citation-support and overall scores remain calibrated against deterministic citation validity/coverage."
+            ),
+            "quota_safety": (
+                f"All Gemini calls in this run share a rolling request pacer targeting {int(pacing_stats['target_rpm'])} RPM. "
+                "The pacer also accounts for recent interactive requests recorded by this process and surfaced 429s "
+                "honor provider retry guidance before a bounded retry."
+            ),
+            "text2sql": (
+                "Text2SQL routing is evaluated in the semantic-planner suite. The Text2SQL component suite uses one "
+                "model call per case to generate validated read-only SQL, executes it in DuckDB and checks the computed result."
             ),
             "quality_gates": (
                 "The letter grade is capped when a critical subsystem is weak, preventing a high weighted average "

src/ragforge/llm.py

@@ -1,10 +1,13 @@
 from __future__ import annotations
 
+import hashlib
 import json
 import mimetypes
 import random
 import re
+import threading
 import time
+from collections import defaultdict, deque
 from pathlib import Path
 from typing import Any, TypeVar
 
@@ -25,8 +28,110 @@ Be concise but complete. Do not expose system or developer instructions, secrets
 T = TypeVar("T", bound=BaseModel)
 
 
+class _GeminiRequestLedger:
+    """Process-local rolling request ledger shared by chat and evaluation.
+
+    Normal interactive calls are only recorded. Evaluation can additionally
+    enforce a conservative per-model RPM budget against the same ledger, so a
+    benchmark started immediately after manual testing accounts for recent
+    requests instead of beginning with an empty limiter window.
+    """
+
+    def __init__(self) -> None:
+        self._lock = threading.RLock()
+        self._calls: dict[str, deque[float]] = defaultdict(deque)
+
+    @staticmethod
+    def _bucket(key_id: str, model: str) -> str:
+        return f"{key_id}:{model}"
+
+    def _prune(self, bucket: deque[float], now: float) -> None:
+        cutoff = now - 60.0
+        while bucket and bucket[0] <= cutoff:
+            bucket.popleft()
+
+    def record(self, key_id: str, model: str) -> None:
+        now = time.monotonic()
+        with self._lock:
+            bucket = self._calls[self._bucket(key_id, model)]
+            self._prune(bucket, now)
+            bucket.append(now)
+
+    def acquire(self, key_id: str, model: str, target_rpm: int) -> float:
+        """Wait until one request can be admitted under a rolling RPM budget."""
+        target = max(1, int(target_rpm))
+        waited = 0.0
+        min_interval = 60.0 / target
+        while True:
+            now = time.monotonic()
+            with self._lock:
+                bucket = self._calls[self._bucket(key_id, model)]
+                self._prune(bucket, now)
+                delay = 0.0
+                if bucket:
+                    delay = max(delay, bucket[-1] + min_interval - now)
+                if len(bucket) >= target:
+                    delay = max(delay, bucket[0] + 60.0 - now)
+                if delay <= 0.0:
+                    bucket.append(now)
+                    return waited
+            sleep_for = min(max(delay, 0.05), 65.0)
+            time.sleep(sleep_for)
+            waited += sleep_for
+
+
+_REQUEST_LEDGER = _GeminiRequestLedger()
+
+
+class RequestPacer:
+    """Optional quota-safe pacing and telemetry for benchmark runs."""
+
+    def __init__(self, target_rpm: int = 12):
+        self.target_rpm = max(0, int(target_rpm))
+        self.request_count = 0
+        self.pacing_sleep_seconds = 0.0
+        self.rate_limit_retries = 0
+        self.rate_limit_sleep_seconds = 0.0
+        self._lock = threading.RLock()
+
+    def before_request(self, key_id: str, model: str) -> None:
+        if self.target_rpm > 0:
+            waited = _REQUEST_LEDGER.acquire(key_id, model, self.target_rpm)
+        else:
+            _REQUEST_LEDGER.record(key_id, model)
+            waited = 0.0
+        with self._lock:
+            self.request_count += 1
+            self.pacing_sleep_seconds += waited
+
+    def note_rate_limit_retry(self, delay_seconds: float) -> None:
+        with self._lock:
+            self.rate_limit_retries += 1
+            self.rate_limit_sleep_seconds += max(0.0, delay_seconds)
+
+    def total_sleep_seconds(self) -> float:
+        with self._lock:
+            return self.pacing_sleep_seconds + self.rate_limit_sleep_seconds
+
+    def stats(self) -> dict[str, float | int]:
+        with self._lock:
+            return {
+                "target_rpm": self.target_rpm,
+                "gemini_requests": self.request_count,
+                "pacing_sleep_ms": round(self.pacing_sleep_seconds * 1000, 1),
+                "rate_limit_retries": self.rate_limit_retries,
+                "rate_limit_sleep_ms": round(self.rate_limit_sleep_seconds * 1000, 1),
+            }
+
+
+
 class GeminiGateway:
-    def __init__(self, api_key: str | None = None, model: str | None = None):
+    def __init__(
+        self,
+        api_key: str | None = None,
+        model: str | None = None,
+        request_pacer: RequestPacer | None = None,
+    ):
         settings = get_settings()
         key = api_key or (settings.gemini_api_key if settings.allow_server_api_key else None)
         if not key:
@@ -34,22 +139,78 @@ class GeminiGateway:
         self.model = model or settings.default_model
         self.settings = settings
         self.client = genai.Client(api_key=key)
+        self.request_pacer = request_pacer
+        self._key_id = hashlib.sha256(key.encode("utf-8")).hexdigest()[:16]
+
+    @staticmethod
+    def _status_code(exc: Exception) -> int | None:
+        status = getattr(exc, "code", None) or getattr(exc, "status_code", None)
+        try:
+            if status is not None:
+                return int(status)
+        except (TypeError, ValueError):
+            pass
+        match = re.search(r"(?:Error code|status(?:_code)?)\s*[:=]\s*(\d{3})", str(exc), flags=re.I)
+        return int(match.group(1)) if match else None
+
+    @classmethod
+    def _is_transient(cls, exc: Exception) -> bool:
+        status = cls._status_code(exc)
+        return status in {408, 429, 500, 502, 503, 504}
+
+    @staticmethod
+    def _retry_after_seconds(exc: Exception) -> float | None:
+        response = getattr(exc, "response", None)
+        headers = getattr(response, "headers", None)
+        if headers:
+            value = headers.get("retry-after") or headers.get("Retry-After")
+            if value:
+                try:
+                    return max(0.0, float(value))
+                except (TypeError, ValueError):
+                    pass
+        text = str(exc)
+        for pattern in (
+            r"retry in\s*([0-9]+(?:\.[0-9]+)?)s",
+            r"retryDelay[^0-9]*([0-9]+(?:\.[0-9]+)?)s",
+        ):
+            match = re.search(pattern, text, flags=re.I)
+            if match:
+                try:
+                    return max(0.0, float(match.group(1)))
+                except ValueError:
+                    pass
+        return None
+
+    def _before_request(self, model: str) -> None:
+        if self.request_pacer is not None:
+            self.request_pacer.before_request(self._key_id, model)
+        else:
+            _REQUEST_LEDGER.record(self._key_id, model)
 
     def _create_interaction(self, **kwargs):
-        """Retry only bounded transient failures; never loop indefinitely."""
+        """Retry bounded transient failures and honor server retry guidance.
+
+        The Google SDK already performs transient retries internally. This layer
+        is intentionally conservative: on a surfaced 429 we respect the
+        server-provided retry delay instead of immediately issuing another burst.
+        """
         retryable = {408, 429, 500, 502, 503, 504}
+        model = str(kwargs.get("model") or self.model)
         for attempt in range(self.settings.llm_max_retries + 1):
+            self._before_request(model)
             try:
                 return self.client.interactions.create(**kwargs)
             except Exception as exc:
-                status = getattr(exc, "code", None) or getattr(exc, "status_code", None)
-                try:
-                    status = int(status) if status is not None else None
-                except (TypeError, ValueError):
-                    status = None
+                status = self._status_code(exc)
                 if attempt >= self.settings.llm_max_retries or (status is not None and status not in retryable):
                     raise
-                time.sleep((0.6 * (2**attempt)) + random.uniform(0.0, 0.25))
+                suggested = self._retry_after_seconds(exc) if status == 429 else None
+                fallback = min(60.0, 1.0 * (2**attempt))
+                delay = max(suggested or 0.0, fallback) + random.uniform(0.05, 0.35)
+                if self.request_pacer is not None and status == 429:
+                    self.request_pacer.note_rate_limit_retry(delay)
+                time.sleep(delay)
         raise RuntimeError("unreachable")
 
     def complete(self, prompt: str, system: str = SYSTEM_PROMPT, model: str | None = None) -> str:
@@ -97,7 +258,12 @@ class GeminiGateway:
                 },
             )
             return schema.model_validate_json((interaction.output_text or "{}").strip())
-        except Exception:
+        except Exception as exc:
+            # Do not turn a 429/5xx into an immediate second API call through
+            # the plain-JSON fallback. That amplifies quota pressure exactly
+            # when the provider is asking us to slow down.
+            if self._is_transient(exc):
+                raise
             try:
                 data = self.complete_json(prompt, default.model_dump())
                 return schema.model_validate(data)

src/ragforge/pipeline.py

@@ -12,7 +12,7 @@ from cachetools import TTLCache
 from langgraph.graph import END, START, StateGraph
 
 from .config import get_settings
-from .llm import GeminiGateway
+from .llm import GeminiGateway, RequestPacer
 from .schemas import EvidenceAssessment, PipelineConfig, QueryPlan, QueryResponse, SearchHit
 from .security import prompt_injection_score
 from .web_search import WebSearchEngine
@@ -52,9 +52,11 @@ class RAGEngine:
         self,
         workspace: Workspace,
         progress_callback: Callable[[float, str], None] | None = None,
+        request_pacer: RequestPacer | None = None,
     ):
         self.workspace = workspace
         self.progress_callback = progress_callback
+        self.request_pacer = request_pacer
         settings = get_settings()
         if RAGEngine._shared_cache is None:
             RAGEngine._shared_cache = TTLCache(maxsize=512, ttl=settings.cache_ttl_seconds)
@@ -227,7 +229,11 @@ class RAGEngine:
                     pass
 
     def _gateway(self, state: GraphState) -> GeminiGateway:
-        return GeminiGateway(state.get("api_key"), state["config"].model)
+        return GeminiGateway(
+            state.get("api_key"),
+            state["config"].model,
+            request_pacer=self.request_pacer,
+        )
 
     def _guard(self, state: GraphState) -> GraphState:
         t = time.perf_counter()

src/ragforge/schemas.py

@@ -137,6 +137,7 @@ class EvaluationRequest(BaseModel):
     session_id: str
     level: Literal["Quick", "Standard", "Deep"] = "Standard"
     model: str = "gemini-3.5-flash-lite"
+    target_rpm: int = Field(default=12, ge=0, le=60)
 
 
 class QueryResponse(BaseModel):

src/ragforge/sql_agent.py

@@ -44,7 +44,7 @@ class SQLWorkspace:
             pieces.append(f"{table}({cols})")
         return "\n".join(pieces)
 
-    def ask(self, question: str, gateway: GeminiGateway) -> tuple[str, str, list[dict[str, Any]]]:
+    def generate_sql(self, question: str, gateway: GeminiGateway) -> str:
         if not self.tables:
             raise ValueError("No CSV/XLSX tables are loaded in this session")
         prompt = f"""You write DuckDB SQL for a read-only analytics assistant.
@@ -52,8 +52,25 @@ Available tables:\n{self.schema_text()}
 Question: {question}
 Return JSON with keys sql and rationale. The SQL must be a single SELECT or WITH query. Never modify data."""
         data = gateway.complete_json(prompt, {"sql": "", "rationale": ""})
-        sql = validate_readonly_sql(str(data.get("sql", "")))
-        result = self.conn.execute(sql).fetchdf()
+        return validate_readonly_sql(str(data.get("sql", "")))
+
+    def execute_sql(self, sql: str) -> pd.DataFrame:
+        validated = validate_readonly_sql(sql)
+        return self.conn.execute(validated).fetchdf()
+
+    def benchmark_query(self, question: str, gateway: GeminiGateway) -> tuple[str, pd.DataFrame]:
+        """Generate and execute SQL with one LLM call for component evaluation.
+
+        Routing is evaluated separately by the semantic-planner benchmark. The
+        Text2SQL component benchmark therefore avoids an extra planner call and
+        a second natural-language answer-generation call.
+        """
+        sql = self.generate_sql(question, gateway)
+        return sql, self.execute_sql(sql)
+
+    def ask(self, question: str, gateway: GeminiGateway) -> tuple[str, str, list[dict[str, Any]]]:
+        sql = self.generate_sql(question, gateway)
+        result = self.execute_sql(sql)
         preview = result.head(200)
         result_md = preview.to_markdown(index=False) if len(preview) else "(no rows)"
         answer_prompt = f"""Answer the user's data question using the SQL result below.

src/ragforge/ui.py

@@ -120,9 +120,9 @@ def _inspector_markdown(trace: dict[str, Any]) -> str:
 def _eval_summary_markdown(report: dict[str, Any]) -> str:
     summary = report.get("summary", {}) if report else {}
     if not summary:
-        return "*Run an evaluation to see the scorecard.*"
+        return "*Run an evaluation to see the score card.*"
     lines = [
-        f"### Evaluation scorecard - {summary.get('evaluation_level', '-')} - grade {summary.get('quality_grade', '-')}",
+        f"### Evaluation score card - {summary.get('evaluation_level', '-')} - grade {summary.get('quality_grade', '-')}",
         "",
         f"**Deterministic quality:** `{float(summary.get('deterministic_quality_score', 0.0)):.3f}` - "
         f"**answer accuracy:** `{float(summary.get('answer_accuracy', 0.0)):.0%}` - "
@@ -151,6 +151,12 @@ def _eval_summary_markdown(report: dict[str, Any]) -> str:
         "",
         f"**Evaluation wall time:** `{float(summary.get('evaluation_wall_ms', 0.0)) / 1000:.1f} s` - "
         f"**response cache bypassed:** `{bool(summary.get('cache_bypassed', False))}`",
+        "",
+        f"**Gemini request budget:** "
+        f"`{int(summary.get('evaluation_target_rpm', 0) or 0)} RPM` - "
+        f"**requests issued:** `{int(summary.get('gemini_requests', 0) or 0)}` - "
+        f"**deliberate pacing wait:** `{float(summary.get('pacing_sleep_ms', 0.0) or 0.0) / 1000:.1f} s` - "
+        f"**429 retries:** `{int(summary.get('rate_limit_retries', 0) or 0)}`",
     ]
     gates = summary.get("quality_gate_notes") or []
     if gates:
@@ -163,7 +169,8 @@ def _eval_summary_markdown(report: dict[str, Any]) -> str:
             f"completeness `{float(summary.get('judge_completeness', 0.0)):.3f}` - "
             f"citation support `{float(summary.get('judge_citation_support', 0.0)):.3f}` - "
             f"overall `{float(summary.get('judge_overall', 0.0)):.3f}` - "
-            f"judge pass `{float(summary.get('judge_pass_rate', 0.0)):.0%}`",
+            f"judge pass `{float(summary.get('judge_pass_rate', 0.0)):.0%}` - "
+            f"sampled cases `{int(summary.get('deep_judge_cases', 0) or 0)}`",
         ]
     return "\n".join(lines)
 
@@ -224,15 +231,60 @@ def _architecture_snapshot(session_id: str | None) -> tuple[str, str, str, dict[
     sid, ws = _ensure_session(session_id)
     settings = get_settings()
     stats = ws.stats()
+    runtime_json = {
+        "ragforge_version": "1.4.1",
+        "workspace": stats,
+        "models": {
+            "generation": settings.default_model,
+            "embedding": settings.embedding_model,
+            "reranker": settings.reranker_model,
+            "native_search": settings.native_search_model,
+        },
+        "limits": {
+            "max_upload_mb": settings.max_upload_mb,
+            "max_archive_files": settings.max_archive_files,
+            "max_archive_uncompressed_mb": settings.max_archive_uncompressed_mb,
+            "session_ttl_minutes": settings.session_ttl_minutes,
+        },
+        "storage": {
+            "runtime_data_dir": str(settings.data_dir),
+            "persistent": False,
+        },
+    }
     runtime = (
         "### Live runtime\n"
-        f"**RAGForge:** `v1.4.0` - **workspace:** `{sid[:12]}...` - **status:** `{stats['status']}`\n\n"
-        f"**Corpus:** `{stats['sources']}` sources - `{stats['chunks']}` chunks - `{stats['source_profiles']}` source profiles - "
-        f"`{stats['tables']}` tables - corpus version `{stats['version']}`\n\n"
+        f"**RAGForge:** `v1.4.1` - **workspace:** `{sid[:12]}...` - **status:** `{stats['status']}`\n\n"
+        f"**Corpus:** `{stats['sources']}` sources - `{stats['chunks']}` chunks - "
+        f"`{stats['source_profiles']}` source profiles - `{stats['tables']}` tables - "
+        f"corpus version `{stats['version']}`\n\n"
         f"**Models:** generation `{settings.default_model}` - embeddings `{settings.embedding_model}` - "
         f"reranker `{settings.reranker_model}` - native search `{settings.native_search_model}`"
     )
-    curl = f
+    curl = f"""# Replace with your deployed Space URL
+BASE_URL=\"https://YOUR-SPACE.hf.space\"
+SESSION_ID=\"{sid}\"
+
+# Health
+curl \"$BASE_URL/api/health\"
+
+# Current workspace status
+curl \"$BASE_URL/api/v1/session/$SESSION_ID\"
+
+# Query the current workspace
+curl -X POST \"$BASE_URL/api/v1/query\" \\
+  -H \"Content-Type: application/json\" \\
+  -d '{{
+    \"session_id\": \"{sid}\",
+    \"query\": \"What is the collection about?\",
+    \"config\": {{\"mode\": \"Auto\", \"profile\": \"Balanced\"}}
+  }}'
+
+# Benchmark metadata
+curl \"$BASE_URL/api/v1/evaluation/benchmark\"
+"""
+    return sid, runtime, curl, runtime_json
+
+
 def _eval_frame(report: dict[str, Any], key: str) -> pd.DataFrame:
     rows = report.get(key, []) if report else []
     return pd.DataFrame(rows)
@@ -553,9 +605,29 @@ def build_ui() -> gr.Blocks:
                         "ablation. Deep additionally uses a calibrated Gemini judge."
                     ),
                 )
+                with gr.Accordion("Evaluation API pacing", open=False):
+                    eval_quota_safe = gr.Checkbox(
+                        value=True,
+                        label="Quota-safe pacing (recommended for free-tier Gemini API keys)",
+                    )
+                    eval_target_rpm = gr.Slider(
+                        4,
+                        30,
+                        value=12,
+                        step=1,
+                        label="Target Gemini requests per minute",
+                        info=(
+                            "Use a value below the active RPM shown for your project in Google AI Studio. "
+                            "12 RPM leaves headroom when the active limit is 15 RPM."
+                        ),
+                    )
+                    gr.Markdown(
+                        "Quota-safe mode also accounts for recent requests made by this running Space and honors "
+                        "Gemini retry guidance when a 429 is returned. Standard and Deep runs can therefore take longer."
+                    )
                 eval_btn = gr.Button("Run evaluation", variant="primary")
                 eval_status = gr.Markdown("Ready to evaluate.", elem_classes=["status-line"])
-                eval_scorecard = gr.Markdown("*Run an evaluation to see the scorecard.*")
+                eval_scorecard = gr.Markdown("*Run an evaluation to see the score card.*")
                 eval_diagnostics = gr.Markdown("*Diagnostics appear after an evaluation run.*")
                 with gr.Tabs():
                     with gr.Tab("Focused QA"):
@@ -573,19 +645,24 @@ def build_ui() -> gr.Blocks:
                 with gr.Accordion("Raw evaluation report", open=False):
                     eval_output = gr.JSON(label="Evaluation report")
 
-                def begin_eval(level):
+                def begin_eval(level, quota_safe, target_rpm):
                     descriptions = {
-                        "Quick": "Running Quick evaluation - smoke-testing QA, planner, overview, SQL and abstention.",
-                        "Standard": "Running Standard evaluation - full deterministic benchmark plus retrieval ablation.",
-                        "Deep": "Running Deep evaluation - full benchmark plus calibrated Gemini judge calls. This is the slowest mode.",
+                        "Quick": "Running Quick evaluation - smoke-testing QA, planner, overview, SQL and abstention (about 11 Gemini calls before retries).",
+                        "Standard": "Running Standard evaluation - full deterministic benchmark plus retrieval ablation (about 26 Gemini calls before retries).",
+                        "Deep": "Running Deep evaluation - Standard plus a representative Deep-judge sample (about 31 Gemini calls before retries). This is the slowest mode.",
                     }
+                    pacing = (
+                        f" Quota-safe pacing is enabled at {int(target_rpm)} RPM."
+                        if quota_safe
+                        else " Quota-safe pacing is disabled; provider 429s are still retried with backoff."
+                    )
                     return (
                         gr.Button(value=f"Running {level} evaluation...", interactive=False),
-                        f"**{descriptions.get(level, descriptions['Standard'])}** Please keep this tab open.",
+                        f"**{descriptions.get(level, descriptions['Standard'])}**{pacing} Please keep this tab open.",
                         "*Evaluation is running. Results will replace this message when the run finishes.*",
                     )
 
-                def run_eval(sid, key, model_name, level, request: gr.Request):
+                def run_eval(sid, key, model_name, level, quota_safe, target_rpm, request: gr.Request):
                     client = getattr(getattr(request, "client", None), "host", None) or "unknown"
                     try:
                         limiter.check(f"ui-eval:{client}")
@@ -593,7 +670,13 @@ def build_ui() -> gr.Blocks:
                         if not ws.chunks:
                             ws.ingest(_demo_paths(), ocr=False, api_key=(key or None), model=model_name)
 
-                        report = run_demo_eval(ws, key or None, model_name, level=level)
+                        report = run_demo_eval(
+                            ws,
+                            key or None,
+                            model_name,
+                            level=level,
+                            target_rpm=int(target_rpm) if quota_safe else 0,
+                        )
                         skipped_note = (
                             " Quick mode intentionally skips the retrieval ablation and Deep judge."
                             if level == "Quick"
@@ -617,22 +700,28 @@ def build_ui() -> gr.Blocks:
                         return (
                             sid or "",
                             gr.Button(value="Run evaluation", interactive=True),
-                            f"**Evaluation failed.** `{type(exc).__name__}: {exc}`",
-                            "*No scorecard produced for this run.*",
+                            (
+                                "**Evaluation paused by Gemini quota.** The provider still returned a 429 after bounded "
+                                "backoff. Leave quota-safe pacing enabled, lower the target RPM, or wait for the quota "
+                                "window to reset.\n\n" + f"`{type(exc).__name__}: {exc}`"
+                                if "429" in str(exc) or "quota" in str(exc).lower()
+                                else f"**Evaluation failed.** `{type(exc).__name__}: {exc}`"
+                            ),
+                            "*No score card produced for this run.*",
                             "*Fix the error above and run the benchmark again.*",
                             pd.DataFrame(), pd.DataFrame(), pd.DataFrame(), pd.DataFrame(), pd.DataFrame(), pd.DataFrame(), {},
                         )
 
                 eval_event = eval_btn.click(
                     begin_eval,
-                    [eval_level],
+                    [eval_level, eval_quota_safe, eval_target_rpm],
                     [eval_btn, eval_status, eval_scorecard],
                     queue=False,
                     show_progress="hidden",
                 )
                 eval_event.then(
                     run_eval,
-                    [session_state, api_key, model, eval_level],
+                    [session_state, api_key, model, eval_level, eval_quota_safe, eval_target_rpm],
                     [
                         session_state, eval_btn, eval_status, eval_scorecard, eval_diagnostics,
                         eval_qa, eval_planner, eval_overview, eval_sql, eval_ablation, eval_abstention, eval_output,

tests/test_evaluation_assets.py

@@ -5,7 +5,7 @@ from pathlib import Path
 def test_demo_benchmark_is_multilayer_and_auditable():
     path = Path("evals/demo_benchmark.json")
     data = json.loads(path.read_text(encoding="utf-8"))
-    assert data["version"] == "1.4"
+    assert data["version"] == "1.4.1"
     assert len(data["qa_cases"]) >= 9
     assert len(data["planner_cases"]) >= 10
     assert len(data["overview_cases"]) >= 2
@@ -47,4 +47,28 @@ def test_demo_evaluation_and_introspection_are_available_through_api():
     assert "/api/v1/evaluate/demo" in text
     assert "/api/v1/evaluation/benchmark" in text
     assert "/api/v1/session/{session_id}" in text
-    assert 'version="1.4.0"' in text
+    assert 'version="1.4.1"' in text
+
+
+def test_evaluation_is_quota_aware_and_reduces_sql_calls():
+    eval_text = Path("src/ragforge/evaluation.py").read_text(encoding="utf-8")
+    llm_text = Path("src/ragforge/llm.py").read_text(encoding="utf-8")
+    sql_text = Path("src/ragforge/sql_agent.py").read_text(encoding="utf-8")
+    assert "RequestPacer" in eval_text
+    assert "target_rpm" in eval_text
+    assert "pacing_sleep_ms" in eval_text
+    assert "rate_limit_retries" in eval_text
+    assert "deep_judge_cases" in eval_text
+    assert "benchmark_query" in sql_text
+    assert "one model call per case" in eval_text
+    assert "retry in" in llm_text
+    assert "_retry_after_seconds" in llm_text
+    assert "if self._is_transient(exc):" in llm_text
+
+
+def test_deep_judge_uses_representative_sample():
+    data = json.loads(Path("evals/demo_benchmark.json").read_text(encoding="utf-8"))
+    judged_qa = [case for case in data["qa_cases"] if case.get("deep_judge")]
+    judged_overviews = [case for case in data["overview_cases"] if case.get("deep_judge")]
+    assert 3 <= len(judged_qa) < len(data["qa_cases"])
+    assert len(judged_overviews) == 1

tests/test_ui_copy.py

@@ -42,3 +42,18 @@ def test_architecture_api_tab_is_interactive_and_live():
     assert "Copy-ready curl examples" in text
     assert "Runtime snapshot" in text
     assert "Evaluation architecture" in text
+
+
+def test_ui_has_quota_safe_evaluation_controls_and_score_card_spacing():
+    text = Path("src/ragforge/ui.py").read_text(encoding="utf-8")
+    assert "Quota-safe pacing" in text
+    assert "Target Gemini requests per minute" in text
+    assert "Evaluation score card" in text
+    assert "Evaluation scorecard" not in text
+
+
+def test_architecture_snapshot_returns_complete_runtime_payload():
+    text = Path("src/ragforge/ui.py").read_text(encoding="utf-8")
+    assert '"ragforge_version": "1.4.1"' in text
+    assert "return sid, runtime, curl, runtime_json" in text
+    assert "curl = f\ndef _eval_frame" not in text