docs/ARCHITECTURE.md

Architecture

The project is intentionally small at first. The PRD describes a large workbench; this repo starts with the smallest version that can grow into it.

High-Level Flow

app.py
  loads config/models.yaml
  configures lightweight logging
  builds Gradio tabs
  passes model catalog to UI modules

ui/*
  defines each Gradio tab
  calls service classes
  emits local app events for inference, datasets, and field notes
  uses shared progress settings for callback loading indicators

agent/*
  holds deterministic local agent planning and trace export helpers

models/*
  holds model catalog, local backend config, and inference services

datasets/*
  stores dataset, synthetic data, and correction-loop helpers

mcp_tools/*
  holds local tool functions, VINDEX call planning, and Gradio-native MCP bridge metadata

config/*
  holds model and training settings

training/*
  holds non-executing training, LoRA request, evaluation, and export planning helpers

tracking/*
  holds local JSONL tracing and optional Trackio integration

deployment/*
  holds Hugging Face Space deployment planning and validation helpers

plant/*
  holds the first reference domain app built from the template
  can run standalone with python -m plant.app --no-model
  keeps heavy model dependencies optional

core/*
  shared app state, event, logging, and registry helpers

Files And Classes

app.py

Builds and launches the Gradio app.

• build_app() creates the Gradio Blocks app.
• Loads the model catalog from config/models.yaml.
• Registers the current UI tabs.
• APP_CSS defines compact responsive layout rules for app width, mobile padding, scrollable tabs, and button touch targets.

plant/app.py

Standalone Plant Discovery reference app built from the template.

• build_app(no_model=True) creates a Gradio app without loading model weights.
• Loads plant/models.yaml.
• Builds a local species index.
• Reuses datasets.field_notes.FieldNoteStore for corrections.
• Uses DemoPlantVisionService for screenshots/tests or PlantVisionService for OpenBMB MiniCPM-V zero-shot and fine-tuned adapter inference.

plant/plant_service.py

Domain service and schema for Plant Discovery.

• PlantID is the structured output schema.
• DemoPlantVisionService provides deterministic no-model results.
• PlantVisionService lazy-loads optional MiniCPM-V dependencies only during identification.
• PlantVisionService.from_config(..., "plant_vlm_finetuned") can load a PEFT adapter after a real adapter repo is configured.
• extract_json_object() and parse_plant_response() make model JSON output testable.

plant/training.py

Non-executing training planner for Plant Discovery.

• build_plant_training_plan() returns SWIFT and LLaMA-Factory command previews.
• plant_training_dependency_report() reports optional training dependency availability.
• write_llamafactory_dataset_info() writes a dataset-info preview for LLaMA-Factory workflows.
• Training is never started by the Gradio UI or script.

plant/plant_loader.py

Domain data and export helpers for Plant Discovery.

• PlantRecord normalizes plant examples into training rows.
• LocalFolderLoader maps species folders to image metadata.
• SpeciesIndexBuilder builds a no-network species index with demo fallback.
• FieldNotesPlantExporter exports corrected field notes to plant training JSONL.

plant/plant_tab.py

Focused Gradio UI for Plant Discovery.

• Identify tab uploads images and renders a safe escaped result card.
• Field Guide tab searches the species index.
• Corrections tab saves and exports training-ready corrections.
• Stats tab summarizes species and correction counts.
• Training is represented as a non-executing plan, not a subprocess.

plant/plant_tools.py

Optional local/MCP tools for Plant Discovery.

• Pure functions can be tested without an MCP server.
• build_mcp_server() imports mcp only when explicitly requested.
• Tools expose identify, species search, correction save/export, stats, and training plan.

models/model_catalog.py

Reads model configuration and turns it into typed Python objects.

• ModelInfo describes one configured model.
• load_model_catalog(path) reads YAML and returns all configured models.
• model_choices(catalog, model_type) filters models for a UI dropdown.
• model_summary(model) returns display metadata for the Gradio JSON panel.
• backend_capabilities maps each model to supported local backend capabilities.

models/placeholder_service.py

Deterministic placeholder model service used before real inference is wired.

• PlaceholderModelService.chat() returns a deterministic text response.
• PlaceholderModelService.vision_chat() returns a deterministic image/prompt response.

This file should be replaced or complemented by real services such as:

• ollama_service.py
• llama_cpp_service.py
• openai_compatible_service.py
• sglang_runner.py
• minicpm_vision.py
• transformers_text.py
• sglang_service.py

models/base.py

Defines service contracts and backend status records.

• BackendStatus describes whether a backend is available.
• TextModelService is the text chat protocol.
• VisionModelService is the vision chat protocol.

models/ollama_service.py

Ollama-backed local inference client.

• Checks whether ollama is installed and reachable.
• Sends text and vision chat requests to http://127.0.0.1:11434/api/chat.
• Lists locally available Ollama models through /api/tags.
• Builds explicit ollama pull <model> commands for the Status tab.
• Does not pull or download models automatically.

models/llama_cpp_service.py

llama.cpp HTTP client for local GGUF inference.

• Checks whether llama-server is installed and reachable.
• Builds explicit llama-server -m <model.gguf> commands.
• Supports --mmproj <mmproj.gguf> command metadata for multimodal models.
• Sends text chat requests to /v1/chat/completions.
• Does not download GGUF files or start background servers automatically.

models/local_backend_config.py

User-local backend settings stored under ignored data/local_backends.yaml.

• LocalBackendConfig stores llama.cpp server URL, OpenAI-compatible base URL, optional served model name, GGUF path, mmproj path, context length, and GPU layers.
• save_local_backend_config() writes local-only settings without touching tracked model config.
• build_llama_server_command() returns the explicit command the user can run.
• local_backend_summary() reports file status and confirms no startup downloads or automatic model loads.

models/openai_compatible_service.py

Local OpenAI-compatible chat client for LM Studio, vLLM-style servers, or similar local endpoints.

• Checks /v1/models for reachability.
• Sends text chat requests to /v1/chat/completions.
• Supports an optional served-model-name override for tools such as LM Studio.
• Returns visible unavailable/request-failed messages instead of crashing the Gradio callback.
• Does not call cloud APIs or download model weights.

models/llama_cpp_python_service.py

Optional direct Python binding backend for GGUF inference.

• Checks whether llama_cpp is importable.
• Requires an explicit local GGUF path.
• Does not download model files.
• Provides text chat through Llama.create_chat_completion().
• Vision support remains routed through llama-server until mmproj/image serialization is wired.

models/minicpm_vision.py

Optional MiniCPM vision backend.

• Checks whether the transformers package is available.
• Lazy-loads AutoProcessor and AutoModelForImageTextToText only when selected.
• Formats image/text messages for image-text-to-text generation.
• Maps thinking mode into the prompt template.
• Provides a video support plan for future local frame sampling.

models/sglang_runner.py

SGLang local server planner and OpenAI-compatible chat client.

• Builds an explicit python -m sglang.launch_server command.
• Includes MiniCPM tool parser configuration.
• Checks /health, sends chat requests to /v1/chat/completions, and can request /shutdown.
• Does not install SGLang, start a process, download model weights, or load a model on app startup.

models/vllm_runner.py

vLLM local server planner and OpenAI-compatible chat client.

• Builds explicit vllm serve <model> command plans.
• Checks /health, parses Prometheus-style /metrics, and sends chat requests to /v1/chat/completions.
• Logs parsed benchmark metrics through TrackingClient.
• Does not install vLLM, start a process, download model weights, or load a model on app startup.

models/transformers_text.py

Optional Transformers text backend.

• Checks whether the transformers package is installed.
• Lazy-loads AutoTokenizer and AutoModelForCausalLM only when the backend is selected.
• Reads trust_remote_code, device map, dtype, max token, and temperature settings from explicit config.
• Provides a simple token-list streaming helper for future Gradio streaming wiring.
• Does not download model weights on startup.

models/service_factory.py

Creates the selected backend service for the UI.

• TEXT_SERVICE_REGISTRY registers available text backend factories.
• VISION_SERVICE_REGISTRY registers available vision backend factories.
• create_text_service() chooses placeholder, llama.cpp, llama-cpp-python, Ollama, OpenAI-compatible, SGLang, or Transformers text service.
• create_vision_service() chooses placeholder, llama.cpp, llama-cpp-python, Ollama, or Transformers MiniCPM vision service.
• backend_statuses() reports current backend availability.
• llama.cpp, llama-cpp-python, and OpenAI-compatible services read ignored local backend settings when selected.

ui/chat_tab.py

Builds the text chat tab.

• Shows text models from the catalog.
• Displays selected model metadata.
• Calls the selected backend service.
• Emits inference request and response events.

ui/vision_tab.py

Builds the vision tab.

• Shows vision models from the catalog.
• Accepts an image and prompt.
• Calls the selected backend service.
• Emits inference request and response events.

ui/dataset_tab.py

Local dataset preview surface.

• Previews local CSV, JSONL, and NDJSON files.
• Previews Hugging Face datasets when the optional external datasets package is installed.
• Shows source, row count, columns, and sample rows.
• Calculates basic local dataset statistics.
• Emits dataset loaded events.

Future behavior:

• Serve dataset tools through the selected MCP path.

ui/train_tab.py

Training planning and local evaluation surface.

• Builds a LoRA dry-run training plan without launching training.
• Builds a non-executing LoRA trainer request with dependency status.
• Shows SWIFT/LLaMA-Factory vision fine-tuning plan.
• Shows checkpoint output path, validation status, and hardware notes.
• Runs local base-vs-tuned evaluation from newline-separated response text.
• Shows exact-match summary and a qualitative eval table.
• Logs tuned evaluation reports to data/eval_results.jsonl.

Future behavior:

• Start LoRA training.
• Show loss and metrics.
• Write Trackio traces.

ui/vllm_tab.py

vLLM local serving planner.

• Builds explicit vllm serve command plans.
• Checks local vLLM /health.
• Fetches and parses /metrics.
• Logs vLLM benchmark metrics through local JSONL/Trackio fallback tracking.
• Does not install vLLM, start a process, download models, or load weights on startup.

ui/export_tab.py

GGUF export planning surface.

• Selects a configured model and quantization.
• Shows official GGUF download command plans when the model has GGUF metadata.
• Shows local HF-to-GGUF conversion and llama.cpp quantization command plans.
• Lists files already present under the selected export directory.
• Exposes existing exported files through a Gradio download output.
• Does not execute downloads, conversion, or quantization.

Future behavior:

• Execute downloads and conversions after explicit user action.

ui/notes_tab.py

Field notes implementation.

• Saves prompt, model response, correction, and tags to data/field_notes.csv.
• Captures optional image path, video path, and a use-for-training flag.
• Exports corrected notes to JSONL.
• Exports local Hugging Face Dataset-style files under data/hf_field_notes.
• Imports uncertain OCR predictions for human correction.
• Exports corrected OCR rows to JSONL.
• Emits field note saved events.

Future behavior:

• Push corrected notes to a remote Hugging Face Dataset after login.
• Feed notes into fine-tuning.

ui/traces_tab.py

Local trace and tracking preview.

• Shows manual trace event previews.
• Shows recent local app events.
• Shows JSONL trace rows and tracking status.
• Exports local traces to exports/traces.jsonl.
• Calls Trackio only when the optional package is installed and enabled.

ui/agent_tab.py

Local non-autonomous agent mode.

• Drafts a research-plan-implement-verify trace.
• Saves agent traces to data/agent_traces.jsonl.
• Exports trace JSONL and local HF Dataset-style trace files.
• Does not execute shell commands, commit, push, deploy, download models, or call external services.

ui/status_tab.py

Shows configured models and backend metadata.

• Helps verify model-size compliance and backend status.
• Provides local llama.cpp settings, GGUF/mmproj file pickers, and command generation.
• Provides LM Studio/OpenAI-compatible base URL, optional model-name storage, and reachability check.
• Provides SGLang command planning, health check, and shutdown request controls.

datasets/field_notes.py

Field note data model and CSV store.

• FieldNote captures prompt, response, correction, tags, and timestamp.
• FieldNote also captures optional image/video paths and a training inclusion flag.
• FieldNoteStore.save() persists notes to data/field_notes.csv.
• FieldNoteStore.list_notes() filters by correction, tag, and training inclusion.
• FieldNoteStore.export_jsonl() writes training-ready JSONL.
• FieldNoteStore.export_hf_dataset() writes local HF Dataset-style files.
• SQLiteFieldNoteStore stores and lists notes in SQLite for larger correction loops.

datasets/loader.py

Dataset preview and statistics helpers.

• preview_local_dataset() previews CSV, JSONL, and NDJSON files.
• dataset_statistics() reports row count, column count, names, and non-empty counts.
• preview_huggingface_dataset() optionally uses the external Hugging Face datasets package.

datasets/synthetic.py

Deterministic local synthetic data helpers.

• generate_synthetic_examples() creates local prompt/response/correction examples.
• validate_synthetic_example() checks schema requirements.
• quality_filter_examples() removes incomplete or low-value examples.
• augment_examples() creates deterministic variants for workflow testing.
• export_synthetic_jsonl() writes JSONL without external services.

datasets/ocr.py

Local OCR correction helpers.

• OCRPrediction stores source path, predicted text, confidence, and optional page.
• load_ocr_predictions() loads local .csv, .jsonl, and .ndjson prediction files.
• uncertain_predictions() filters rows at or below a confidence threshold or with empty text.
• import_uncertain_predictions() creates Field Notes correction tasks for uncertain rows.
• export_corrected_ocr_notes() writes corrected OCR examples to JSONL for evaluation or training.
• ocr_import_summary() previews uncertain rows for the Field Notes tab.

mcp_tools/tools.py

Local MCP-style tools.

• dataset_stats_tool() returns local dataset statistics.
• hf_dataset_preview_tool() previews Hugging Face datasets when optional dependencies exist.
• safe_calculator_tool() evaluates numeric arithmetic only.
• model_inference_tool() routes text prompts through the selected model service.
• tool_registry() returns the local tool map for a future MCP endpoint.

mcp_tools/vindex_tool.py

Non-executing VINDEX integration boundary.

• Defines the eight VINDEX PRD methods and their local FastAPI paths.
• build_vindex_call_plan() validates method names and builds endpoint/payload plans.
• Caps star_spread.n_neighbors at 5 and calibrated_edit.causal_window at 3 based on the PRD safety notes.
• vindex_dependency_report() checks whether the optional vindex package or local health endpoint is available.
• vindex_verification_report() combines dependency status with a safe call plan and keeps execution disabled until the local VINDEX install is verified.

mcp_tools/bridge.py

Gradio-native MCP bridge metadata and local invocation helper.

• MCP_PATH documents /gradio_api/mcp/sse.
• mcp_manifest() returns the selected mode, path, and tool definitions.
• invoke_mcp_tool() verifies local tool invocation by name.

agent/runner.py

Deterministic local agent trace runner.

• AGENT_SYSTEM_PROMPT defines the agent behavior contract.
• run_agent_loop() produces research, plan, implement, and verify trace steps.
• run_paper_to_code_loop() produces paper-to-code research, plan, implement, and verify trace steps.
• default_safety_gates() lists the non-autonomous safety requirements.
• save_agent_trace() appends traces to JSONL.
• export_agent_traces() exports trace JSONL.
• export_agent_traces_hf_dataset() writes local HF Dataset-style trace files.
• The runner can call safe local tools, but it is not autonomous.

core/file_exports.py

Shared export helper.

• copy_text_file_or_empty() copies a text artifact to an export path or creates an empty one.

training/export.py

Non-executing GGUF export planning.

• detect_llama_cpp_tools() checks llama-server, llama-cli, and llama-quantize.
• build_export_plan() creates explicit download, conversion, and quantization command plans.
• list_exported_files() lists generated/local export files.
• ExportPlan.as_dict() marks that commands are not executed and no startup downloads occur.

training/evaluation.py

Local deterministic evaluation helpers.

• default_prompt_cases() returns a small built-in prompt test set.
• load_prompt_cases() loads prompt/expected pairs from JSONL.
• evaluate_responses() computes exact-match rows and a qualitative table.
• perplexity_from_losses() computes perplexity from explicit negative log likelihood values.
• compare_base_vs_tuned() reports exact-match delta.
• log_eval_report() appends JSONL evaluation results.

training/lora_trainer.py

Non-executing LoRA trainer request builder.

• lora_dependency_report() reports PEFT, TRL, Transformers, and Torch availability.
• build_lora_training_request() combines the training plan with dependency status and a command preview.
• vision_finetuning_plan() documents SWIFT/LLaMA-Factory as the future MiniCPM-V fine-tuning path.
• Keeps execute_training false until dependencies, hardware, and dataset schema are approved.

training/reward_eval.py

Deterministic local reward-style evaluation helpers.

• RewardEvaluator.evaluate() scores supplied responses with transparent lexical heuristics.
• best_of_n() selects the highest-scoring candidate without model calls.
• create_dpo_pairs() creates chosen/rejected pairs for DPO-style datasets.
• eval_lora_vs_base() compares base and LoRA response rewards.

training/planner.py

Non-executing LoRA training planner.

• load_training_config() reads LoRA and training settings from config/training.yaml.
• build_training_plan() creates a dry-run plan with checkpoint output path.
• validate_training_plan() checks dataset existence and numeric training settings.
• training_hardware_notes() documents practical local hardware expectations.

tracking/trackio_client.py

Tracking client with JSONL fallback.

• load_tracking_config() reads Trackio settings from config/training.yaml.
• TrackingClient.init() starts Trackio only when enabled and installed.
• TrackingClient.log() always writes local JSONL and optionally forwards to Trackio.
• TrackingClient.finish() closes optional Trackio state.
• export_traces() copies local traces to exports/traces.jsonl.
• read_trace_rows() returns recent local trace rows for the UI.

core/events.py

Small event bus reserved for future cross-module events.

• EventType names app events.
• UI_ERROR records visible tab-level failures.
• Event carries event data.
• EventBus registers handlers and emits events.

core/app_state.py

Shared local app state.

• AppState.emit() records events, logs them, and dispatches them through EventBus.
• AppState.emit() also writes trace events through TrackingClient.
• AppState.recent_events() returns local trace previews for the Traces tab.
• emit_inference_response() records shared response metadata.

core/tab_feedback.py

Formats tab status text and emits ui_error events for visible tab-level failures.

ui/progress.py

Defines the shared Gradio progress mode used by tab button callbacks.

core/app_logging.py

Lightweight logging setup.

• configure_app_logging() configures compact process logging once.

core/registry.py

Generic registry helper.

• Registry.register(name, item) stores a service.
• Registry.get(name) retrieves a service.
• Registry.list() lists registered services.

Current Design Rule

The app must not download model weights on startup. Model loading should happen only after the user chooses a backend/model and clicks an explicit action.