diff --git a/.scratch/arch-improvements/issues/01-appcontext-expand.md b/.scratch/arch-improvements/issues/01-appcontext-expand.md deleted file mode 100644 index 6eebcc9fd455f7824c3fd8e1b789d56f74683079..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/01-appcontext-expand.md +++ /dev/null @@ -1,16 +0,0 @@ -# 01 — Introduce `AppContext` dataclass (Expand) - -**Blocked by:** None — can start immediately -**Status:** ready-for-agent - -## What to build - -Introduce a single `AppContext` dataclass that packages all runtime state currently scattered as module-level globals in `functions.py` (`_bm25_index`, `_vector_index`, `_embedding_provider`, `_hybrid_search`, `_stream_broadcaster`, and the `kv` reference). Construct one `AppContext` instance inside `create_app()` and thread it through to every function that needs it. The existing globals stay in place as a compatibility shim — no callers are changed in this ticket. Zero behaviour change; this is the expand step that makes the god-module split (tickets 05–08) possible. - -## Acceptance criteria - -- [ ] `AppContext` dataclass exists in a new `context.py` module (or equivalent) with typed fields for `kv`, `bm25`, `vector`, `embedder`, and `broadcast` -- [ ] `create_app()` constructs one `AppContext` and stores it on the Flask app (e.g. `app.ctx`) -- [ ] The five existing `set_*` functions in `functions.py` remain working — globals are still set alongside the new `AppContext` -- [ ] All existing tests pass with no changes -- [ ] No route handler or worker is changed in this ticket diff --git a/.scratch/arch-improvements/issues/02-privacy-module.md b/.scratch/arch-improvements/issues/02-privacy-module.md deleted file mode 100644 index 7d648fc1c4729c288362066d0af03b06ec54c021..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/02-privacy-module.md +++ /dev/null @@ -1,17 +0,0 @@ -# 02 — Extract `privacy.py` deep module - -**Blocked by:** None — can start immediately -**Status:** ready-for-agent - -## What to build - -Extract `strip_private_data()` from `functions.py` into a dedicated `privacy.py` module. The new module exposes a single `scrub(text: str, patterns=DEFAULT_PATTERNS) -> str` interface. The regex pattern list is promoted to a public, documented constant (`DEFAULT_PATTERNS`) and is extensible via an env var (`AGENTCACHE_REDACT_PATTERNS`). `functions.py` calls `privacy.scrub()` internally — all existing callers continue working without change. First unit tests for Privacy Scrub are added in this ticket, targeting `scrub()` directly. - -## Acceptance criteria - -- [ ] `privacy.py` exists with a `scrub(text, patterns=DEFAULT_PATTERNS)` function -- [ ] `DEFAULT_PATTERNS` is a documented list of regex strings — visible and auditable -- [ ] An env var `AGENTCACHE_REDACT_PATTERNS` (comma-separated regex strings) appends additional patterns at startup -- [ ] `functions.py` no longer contains the scrubbing regex list — it imports and calls `privacy.scrub()` -- [ ] Unit tests cover: API key pattern, bearer token pattern, custom pattern via argument, no false positives on safe text -- [ ] All existing tests pass diff --git a/.scratch/arch-improvements/issues/03-auth-middleware.md b/.scratch/arch-improvements/issues/03-auth-middleware.md deleted file mode 100644 index 7ecaa9eee2b89704f28befe99380e80d8d31a0b3..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/03-auth-middleware.md +++ /dev/null @@ -1,16 +0,0 @@ -# 03 — Centralise auth middleware - -**Blocked by:** None — can start immediately -**Status:** ready-for-agent - -## What to build - -Replace the copy-pasted `_check_auth()` function that currently lives independently in every route blueprint (`mcp.py`, `observations.py`, `memories.py`, `health.py`, and others) with a single `require_auth` decorator defined in a new `auth.py` module. Every route that previously called `_check_auth()` at the top of its handler switches to the `@require_auth` decorator. The auth logic (timing-safe `hmac.compare_digest` Bearer token check) is identical — this is purely a deduplication. RBAC can be added in one place in a future ticket. - -## Acceptance criteria - -- [ ] `auth.py` exists with a `require_auth(f)` decorator that performs the timing-safe Bearer token check -- [ ] Every route blueprint imports and uses `@require_auth` — no blueprint defines its own `_check_auth` -- [ ] A request with no secret configured passes through (existing behaviour preserved) -- [ ] A request with a wrong token still gets a `401` response -- [ ] All existing tests pass diff --git a/.scratch/arch-improvements/issues/04-mcp-tool-registry.md b/.scratch/arch-improvements/issues/04-mcp-tool-registry.md deleted file mode 100644 index c7594145cd87c8c21746ec8ff3cc87e3888ff9e7..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/04-mcp-tool-registry.md +++ /dev/null @@ -1,17 +0,0 @@ -# 04 — MCP tool registry (replace elif chain) - -**Blocked by:** 03 — centralise auth middleware -**Status:** ready-for-agent - -## What to build - -Replace the 300-line `if/elif` dispatch chain in `routes/mcp.py` with a `@register("tool_name")` decorator registry. Each of the 30+ MCP tool handlers becomes a standalone function decorated with `@register`. The dispatcher becomes a two-liner: look up the tool name in the registry dict and call the handler. Auth is handled by the shared `@require_auth` decorator from ticket 03. The tool schema list (`get_mcp_tools_schemas`) stays as-is. No change to the MCP wire format or agent behaviour. - -## Acceptance criteria - -- [ ] A `_tools: dict[str, Callable]` registry exists and a `@register(name)` decorator populates it -- [ ] Every MCP tool handler is a standalone function — not an inline block inside a giant if/elif -- [ ] The POST `/mcp/tools` handler body is ≤ 20 lines (lookup + call + error handling) -- [ ] Each tool handler function is independently importable and callable in a test without starting Flask -- [ ] Adding a new tool requires only adding a new decorated function — no editing of existing dispatch code -- [ ] All existing MCP tool calls produce the same responses as before diff --git a/.scratch/arch-improvements/issues/05-split-observations-memories.md b/.scratch/arch-improvements/issues/05-split-observations-memories.md deleted file mode 100644 index c7f32851085b5d15916fd48ad0cfe60bb39cffb9..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/05-split-observations-memories.md +++ /dev/null @@ -1,23 +0,0 @@ -# 05 — Split god module batch 1: `observations.py` + `memories.py` - -**Blocked by:** 01 — AppContext dataclass (Expand) -**Status:** ready-for-agent - -## What to build - -Move the first two major domain areas out of `functions.py` into focused modules, using the domain vocabulary from `UBIQUITOUS_LANGUAGE.md`. - -**`observations.py`** receives: `folder_observe()`, `observe()` (legacy), `dedup_folder_observations()`, `build_synthetic_compression()`, `infer_type()`, `extract_files()`, `extract_image()`, image store helpers (`save_image_to_disk`, `delete_image`, `touch_image`, `is_managed_image_path`), and `normalize_folder_path()` / `validate_agent_id()`. - -**`memories.py`** receives: `remember()`, `forget()`, memory versioning logic, and `jaccard_similarity()`. - -`functions.py` re-exports everything from these two modules so all existing callers continue to work unchanged. CI must stay green. This is the first batch of the expand phase of the god-module split. - -## Acceptance criteria - -- [ ] `observations.py` and `memories.py` exist as standalone modules -- [ ] All moved functions accept `AppContext` (from ticket 01) instead of reaching into module-level globals, while also accepting the legacy `kv: StateKV` signature for backward compat -- [ ] `functions.py` re-exports every moved symbol — no call site outside `functions.py` needs to change -- [ ] Existing route blueprints and workers continue to import from `functions` without modification -- [ ] All existing tests pass -- [ ] No logic is changed — this is a pure relocation diff --git a/.scratch/arch-improvements/issues/06-split-indexing-retrieval.md b/.scratch/arch-improvements/issues/06-split-indexing-retrieval.md deleted file mode 100644 index 1ab5637d82c11c22f1634c4fa37676bfffe90386..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/06-split-indexing-retrieval.md +++ /dev/null @@ -1,24 +0,0 @@ -# 06 — Split god module batch 2: `indexing.py` + `retrieval.py` - -**Blocked by:** 05 — split batch 1 (observations + memories) -**Status:** ready-for-agent - -## What to build - -Move the second major domain area out of `functions.py`. - -**`indexing.py`** receives: `IndexPersistence`, `rebuild_index()`, `backfill_obs_lookup_if_needed()`, `vector_index_add_guarded()`, `clip_embed_input()`, and all index-related helpers. This module owns the Index Rebuild lifecycle (see `UBIQUITOUS_LANGUAGE.md`). - -**`retrieval.py`** receives: `folder_search()`, `folder_timeline()`, `compile_context()`, and `export_data()`. This module owns the Recall, Smart Search, Folder Search, Timeline, and Context Compilation operations. - -`functions.py` continues to re-export everything. CI stays green. - -## Acceptance criteria - -- [ ] `indexing.py` and `retrieval.py` exist as standalone modules -- [ ] All moved functions use `AppContext` from ticket 01 rather than reaching into globals directly -- [ ] `functions.py` re-exports every moved symbol unchanged -- [ ] `IndexPersistence` is importable from `indexing` in `workers.py` without changing any other worker code -- [ ] `folder_search()` and `compile_context()` are callable from a test by constructing an `AppContext` with an in-memory KV and a fresh `SearchIndex` — no Flask app required -- [ ] All existing tests pass -- [ ] No logic is changed diff --git a/.scratch/arch-improvements/issues/07-split-consolidation-slots-lessons.md b/.scratch/arch-improvements/issues/07-split-consolidation-slots-lessons.md deleted file mode 100644 index d66956b6d2654228a8dd25e2ee935f3c67f7c5cd..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/07-split-consolidation-slots-lessons.md +++ /dev/null @@ -1,25 +0,0 @@ -# 07 — Split god module batch 3: `consolidation.py` + `slots.py` + `lessons.py` - -**Blocked by:** 06 — split batch 2 (indexing + retrieval) -**Status:** ready-for-agent - -## What to build - -Move the final major domain areas out of `functions.py`. - -**`consolidation.py`** receives: `consolidate()`, `auto_forget()`, `folder_graph_build()`, and all LLM-call helpers used exclusively by the Consolidation pipeline. This module owns the Consolidation lifecycle (Working Memory → Episodic → Semantic → Procedural). - -**`slots.py`** receives: all Slot CRUD functions (`get_slots`, `set_slot`, `delete_slot`, `reflect_slot`, `append_slot`, etc.). - -**`lessons.py`** receives: all Lesson CRUD functions (`get_lessons`, `save_lesson`, `search_lessons`, `strengthen_lesson`, lesson decay logic). - -`functions.py` re-exports everything. After this ticket, `functions.py` is a pure shim — no domain logic remains in it. - -## Acceptance criteria - -- [ ] `consolidation.py`, `slots.py`, and `lessons.py` exist as standalone modules -- [ ] `functions.py` contains no domain logic — only re-export statements -- [ ] All moved functions use `AppContext` from ticket 01 -- [ ] `auto_forget()` is callable from a test without importing `functions` directly -- [ ] All existing tests pass -- [ ] No logic is changed diff --git a/.scratch/arch-improvements/issues/08-delete-functions-shim.md b/.scratch/arch-improvements/issues/08-delete-functions-shim.md deleted file mode 100644 index c6192c743fdf477e42701847362a9d6612033847..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/08-delete-functions-shim.md +++ /dev/null @@ -1,23 +0,0 @@ -# 08 — Contract: delete `functions.py` shim, migrate all callers to direct imports - -**Blocked by:** 05, 06, 07 — all three split batches must be complete -**Status:** ready-for-agent - -## What to build - -Delete `functions.py` entirely. Update every import across `routes/`, `workers.py`, `app.py`, and the test suite to point directly to the focused module that owns the symbol. This is the contract step of the expand–contract sequence. After this ticket the domain vocabulary from `UBIQUITOUS_LANGUAGE.md` is directly reflected in the module names — there is no god module. - -Import mapping (non-exhaustive): -- `from .functions import folder_observe` → `from .observations import folder_observe` -- `from .functions import remember, forget` → `from .memories import remember, forget` -- `from .functions import folder_search, compile_context` → `from .retrieval import folder_search, compile_context` -- `from .functions import IndexPersistence, rebuild_index` → `from .indexing import IndexPersistence, rebuild_index` -- `from .functions import consolidate, auto_forget` → `from .consolidation import consolidate, auto_forget` - -## Acceptance criteria - -- [ ] `functions.py` does not exist in the repository -- [ ] `rg "from .functions import\|from agentcache.functions import\|import functions"` returns zero results in `src/` -- [ ] All existing tests pass with imports updated -- [ ] No test imports from `functions` — each test imports from the specific module it exercises -- [ ] CI is green diff --git a/.scratch/arch-improvements/issues/09-typed-observations-table-expand.md b/.scratch/arch-improvements/issues/09-typed-observations-table-expand.md deleted file mode 100644 index 57a029bf83fab266482a61f73f7a41cff8b84102..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/09-typed-observations-table-expand.md +++ /dev/null @@ -1,19 +0,0 @@ -# 09 — Add typed `observations` table to SQLite (Expand) - -**Blocked by:** 05 — observations module must exist to own this migration -**Status:** ready-for-agent - -## What to build - -Add a typed `observations` table to the SQLite schema with columns for `folder`, `agent`, `timestamp`, `type`, `importance`, and `text`. When a new Observation is ingested via `folder_observe()`, write it to both the new typed table and the existing `kv_store` scope (dual-write). Reads still use `kv_store` — zero behaviour change for any query path. This is the expand step that makes SQL-level filtering possible without breaking anything. - -The new table enables: `WHERE folder = ? AND agent = ? AND timestamp > ?` and `ORDER BY importance DESC LIMIT N` entirely at the SQLite layer — no Python-side filtering needed. - -## Acceptance criteria - -- [ ] `observations(id TEXT PK, folder TEXT, agent TEXT, timestamp TEXT, type TEXT, importance INTEGER, text TEXT)` table exists in the DB schema with indexes on `(folder, agent, timestamp)` and `(importance)` -- [ ] `folder_observe()` dual-writes: one row to `observations`, one entry to `kv_store` (existing path) -- [ ] The DB migration runs automatically on startup if the table does not exist (no manual step) -- [ ] All reads continue to use `kv_store` — no query is changed in this ticket -- [ ] All existing tests pass -- [ ] A DB integrity test verifies that for each `kv_store` Observation entry written, a matching row exists in `observations` diff --git a/.scratch/arch-improvements/issues/10-typed-observations-reads-contract.md b/.scratch/arch-improvements/issues/10-typed-observations-reads-contract.md deleted file mode 100644 index 2e7f155389e2b5748d4566d25445eb9d315b8093..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/10-typed-observations-reads-contract.md +++ /dev/null @@ -1,18 +0,0 @@ -# 10 — Migrate Observation reads to typed table (Contract) - -**Blocked by:** 09 — typed observations table (Expand) must be complete -**Status:** ready-for-agent - -## What to build - -Switch `folder_search`, `folder_timeline`, `compile_context`, and `auto_forget` to read Observations from the typed `observations` table using SQL-level filtering, replacing the current pattern of `kv.list(scope)` + Python-side filtering. Stop dual-writing new Observations to `kv_store` — the typed table is now the single source of truth. A one-time migration backfills any existing `kv_store` Observation entries into the typed table for users upgrading from a previous version. - -## Acceptance criteria - -- [ ] `folder_timeline()` issues a single `SELECT … WHERE folder=? AND agent=? ORDER BY timestamp DESC LIMIT ?` — no Python-side filtering loop -- [ ] `folder_search()` hydrates candidates from the typed table rather than `kv.list(scope)` for the Observation load step -- [ ] `compile_context()` fetches recent Observations using `ORDER BY importance DESC, timestamp DESC LIMIT ?` at the SQL layer -- [ ] New `folder_observe()` calls write only to the typed table (no `kv_store` Observation scope write) -- [ ] A backfill function runs once on startup to migrate existing `kv_store` Observations into the typed table -- [ ] A benchmark test (or manual note in the PR) shows query latency improvement at ≥ 10k Observations -- [ ] All existing tests pass diff --git a/.scratch/arch-improvements/issues/11-basevectorindex-protocol.md b/.scratch/arch-improvements/issues/11-basevectorindex-protocol.md deleted file mode 100644 index c4b86dcf40b51fac7034d8ed3f9394abef1cc4e5..0000000000000000000000000000000000000000 --- a/.scratch/arch-improvements/issues/11-basevectorindex-protocol.md +++ /dev/null @@ -1,17 +0,0 @@ -# 11 — `BaseVectorIndex` protocol + `InMemoryVectorIndex` adapter - -**Blocked by:** 06 — indexing module must exist before its interface is formalised -**Status:** ready-for-agent - -## What to build - -Define a `BaseVectorIndex` Protocol (or ABC) in `indexing.py` with three methods: `add(obs_id, session_id, embedding)`, `remove(obs_id)`, and `search(query_embedding, limit) -> list`. Rename the current flat-list implementation to `InMemoryVectorIndex` and have it implement `BaseVectorIndex`. Update `AppContext.vector` to be typed as `BaseVectorIndex`. Behaviour is completely unchanged — this ticket only cuts the adapter seam so future backends (HNSW, Qdrant, Chroma) can be swapped in without touching `retrieval.py` or `consolidation.py`. - -## Acceptance criteria - -- [ ] `BaseVectorIndex` Protocol exists in `indexing.py` with `add`, `remove`, and `search` as the only required methods -- [ ] `InMemoryVectorIndex` implements `BaseVectorIndex` and passes a `isinstance(idx, BaseVectorIndex)` check -- [ ] `AppContext.vector` is typed as `BaseVectorIndex | None` -- [ ] A test constructs a minimal stub that implements `BaseVectorIndex` and passes it to `folder_search()` via `AppContext` — confirming the seam is real and injectable -- [ ] `VectorIndex` (old name) remains as an alias for `InMemoryVectorIndex` for one release to avoid breaking any external imports -- [ ] All existing tests pass with no behaviour change diff --git a/src/agentcache.egg-info/PKG-INFO b/src/agentcache.egg-info/PKG-INFO index 533ac5fa157ae60c2b7be3e1f8c6fea82548d1c5..c6651cc724bf570ef3036a182899ae592498231c 100644 --- a/src/agentcache.egg-info/PKG-INFO +++ b/src/agentcache.egg-info/PKG-INFO @@ -1,4 +1,4 @@ -Metadata-Version: 2.1 +Metadata-Version: 2.4 Name: agentcache Version: 0.9.8 Summary: A Python REST + WebSocket + MCP cache server for AI agents, backed by SQLite @@ -35,486 +35,4 @@ Requires-Dist: ruff>=0.3.0; extra == "dev" Requires-Dist: twine>=5.0.0; extra == "dev" Provides-Extra: local-embeddings Requires-Dist: sentence-transformers>=2.7.0; extra == "local-embeddings" - ---- -title: AgentCache Python -emoji: 🧠 -colorFrom: blue -colorTo: indigo -sdk: docker -pinned: false ---- - -

agentcache-python

- -

- Persistent memory for AI coding agents — pure Python, zero external databases.
- Works with Claude Code, Cursor, Cline, Windsurf, Gemini CLI, and any MCP client. -

- -

- Python 3.10+ - SQLite WAL - Flask 3.0 - MCP Compatible - HuggingFace Space -

- -

- Quick Start • - Features • - MCP • - API • - Config • - Deploy • - Viewer • - Architecture -

- ---- - -## What Is This? - -**agentcache-python** is a Python reimplementation of the [agentcache](https://github.com/rohitg00/agentcache) persistent memory server. It exposes a REST API, WebSocket stream, and MCP tools endpoint that AI coding agents use to store and retrieve session observations, long-term memories, lessons, and pinned memory slots. - -Key differences from the Node.js original: - -- **No Node.js or iii-engine** — runs with plain `python src/app.py` -- **SQLite instead of Dolt** — single file, WAL mode, instant startup -- **HuggingFace Space ready** — deploys in one click, data synced to an HF dataset repo -- **Same REST + MCP wire format** — drop-in for any agent already wired to agentcache - -Your agent captures every tool call, stores them as observations, compresses them into searchable memory, and injects the right context at the start of every new session — automatically. - ---- - -## Quick Start - -### Run locally - -```bash -# Clone -git clone https://github.com/Yashwant00CR7/agentcache.git -cd agentcache - -# Install dependencies (no build step) -pip install -r requirements.txt - -# Start the server -python src/app.py -``` - -Server starts on **http://localhost:3111**. Open the viewer at http://localhost:3111/viewer. - -### Verify it works - -```bash -# Health check -curl http://localhost:3111/agentcache/livez -# {"status": "ok"} - -# Save a memory -curl -X POST http://localhost:3111/agentcache/remember \ - -H "Content-Type: application/json" \ - -d '{"content": "JWT auth uses jose middleware in src/middleware/auth.ts", "concepts": ["auth", "jwt"]}' - -# Recall it -curl -X POST http://localhost:3111/agentcache/search \ - -H "Content-Type: application/json" \ - -d '{"query": "authentication middleware", "limit": 5}' -``` - ---- - -## Features - -| Feature | Status | Notes | -|---------|--------|-------| -| REST API — sessions, memories, observations | ✅ | Full surface | -| WebSocket live stream | ✅ | `/stream/mem-live/viewer` | -| MCP tools endpoint | ✅ | 31 tools | -| Built-in HTML viewer | ✅ | Real-time dashboard at `/viewer` | -| BM25 keyword search | ✅ | Always on, no API key needed | -| Hybrid BM25 + vector search | ✅ | Requires `GEMINI_API_KEY` | -| 4-tier memory consolidation | ⚙️ | `CONSOLIDATION_ENABLED=true` + LLM key | -| Knowledge graph extraction | ⚙️ | `GRAPH_EXTRACTION_ENABLED=true` + LLM key | -| LLM observation compression | ⚙️ | `AGENTCACHE_AUTO_COMPRESS=true` + LLM key | -| Lessons with confidence decay | ✅ | Fingerprinted, auto-strengthen on repeat | -| Memory slots (pinned context) | ✅ | CRUD + auto-reflect | -| Session replay | ✅ | Full timeline in viewer | -| Audit log | ✅ | Tracks every write with agent_id + timestamp | -| HuggingFace Space deploy | ✅ | One-click, data synced to dataset repo | -| Privacy filtering | ✅ | Strips API keys, tokens before storage | - -### 4-Tier Memory Model - -Inspired by how human memory works — raw experience → compressed episodes → extracted facts → learned patterns. - -| Tier | What | When | -|------|------|------| -| **Working** | Raw observations from tool use | Every tool call | -| **Episodic** | Compressed session summaries | Session end | -| **Semantic** | Extracted facts and patterns | Consolidation | -| **Procedural** | Workflows and decision patterns | Consolidation | - ---- - -## MCP Integration - -Wire agentcache-python into your agent's MCP config. It speaks the same MCP protocol as the Node.js original. - -### Most agents (Cursor, Claude Desktop, Cline, Windsurf) - -```json -{ - "mcpServers": { - "agentcache": { - "command": "npx", - "args": ["-y", "@agentcache/mcp"], - "env": { - "AGENTCACHE_URL": "http://localhost:3111" - } - } - } -} -``` - -### Claude Code - -Paste this prompt and your agent will wire everything: - -``` -Start agentcache-python: run `python src/app.py` from the agentcache-python directory. -Then add this MCP server to ~/.claude.json under mcpServers: -{ - "agentcache": { - "command": "npx", - "args": ["-y", "@agentcache/mcp"], - "env": { "AGENTCACHE_URL": "http://localhost:3111" } - } -} -Verify with: curl http://localhost:3111/agentcache/livez -Open the viewer at: http://localhost:3111/viewer -``` - -### Available MCP Tools (31) - -| Tool | Description | -|------|-------------| -| `memory_save` | Save a long-term insight, decision, or pattern | -| `memory_recall` | Search past observations by keyword | -| `memory_smart_search` | Hybrid BM25 + vector semantic search | -| `memory_sessions` | List recent sessions | -| `memory_sessions_list` | Retrieve all memory sessions | -| `memory_timeline` | Chronological observations for a session | -| `memory_observations` | Observations for a session | -| `memory_profile` | Per-project concept + file profile | -| `memory_lessons` | List active lessons with confidence scores | -| `memory_lesson_save` | Save a lesson (duplicate saves strengthen it) | -| `memory_lesson_recall` | Search lessons by query | -| `memory_lesson_search` | Search lessons by keywords | -| `memory_consolidate` | Run 4-tier memory consolidation | -| `memory_reflect` | Reflect on session, update context | -| `memory_diagnose` | Health check across all subsystems | -| `memory_forget` | Delete memory, session, or observations | -| `memory_export` | Export all memory data as JSON | -| `agent_observe` | Log agent execution observation | -| `agent_remember` | Save agent cache to long-term storage | -| `memory_antigravity_sync` | Sync Antigravity transcripts to memory | -| `memory_antigravity_sync_all` | Master sync: transcript + crystallize + reflect | -| `memory_slot_list` | List all pinned memory slots | -| `memory_slot_get` | Retrieve a specific pinned memory slot | -| `memory_slot_create` | Create/overwrite a pinned memory slot | -| `memory_slot_append` | Append text content to a pinned memory slot | -| `memory_slot_replace` | Replace pinned memory slot content | -| `memory_slot_delete` | Delete a pinned memory slot | -| `memory_action_create` | Create a new work item / action | -| `memory_action_update` | Update fields of an existing action | -| `memory_frontier` | Get active and pending actions sorted by priority | -| `memory_crystallize` | Crystallize/summarize observations in a session | - ---- - -## API Reference - -Base URL: `http://localhost:3111/agentcache` - -### Health - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/livez` | Liveness probe — no auth required | - -### Sessions - -| Method | Path | Description | -|--------|------|-------------| -| `POST` | `/session/start` | Start a new session | -| `POST` | `/session/end` | End a session | -| `POST` | `/session/commit` | Commit session with summary | -| `GET` | `/sessions` | List all sessions | - -### Observations - -| Method | Path | Description | -|--------|------|-------------| -| `POST` | `/observe` | Ingest a hook event observation | -| `POST` | `/agent/observe` | Simplified observe for direct agent use | -| `GET` | `/observations` | List observations (`?session_id=`) | -| `POST` | `/timeline` | Chronological observation window | - -### Memories - -| Method | Path | Description | -|--------|------|-------------| -| `POST` | `/remember` | Save long-term memory | -| `POST` | `/agent/remember` | Simplified remember | -| `POST` | `/forget` | Delete memory / session / observations | -| `POST` | `/search` | BM25 + vector search | -| `POST` | `/context` | Compile context for a session + project | -| `GET` | `/memories` | List memories (`?latest=true&limit=N`) | -| `POST` | `/evolve` | Create a new memory version | - -### Lessons - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/lessons` | List lessons | -| `POST` | `/lessons` | Create lesson | -| `POST` | `/lessons/search` | Search lessons | -| `POST` | `/lessons/strengthen` | Reinforce an existing lesson | - -### Slots - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/slots` | List all pinned slots | -| `POST` | `/slot` | Create or update a slot | -| `GET` | `/slot` | Get slot by name | -| `DELETE` | `/slot` | Delete a slot | -| `POST` | `/slot/reflect` | Auto-populate from session observations | - -### Graph + Profile - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/relations` | Knowledge graph edges | -| `POST` | `/relations` | Add a relation | -| `GET` | `/profile` | Project profile (top concepts, files) | - -### Actions - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/actions` | List actions | -| `POST` | `/actions` | Create an action | -| `PATCH` | `/actions/` | Update action status / fields | -| `GET` | `/frontier` | Pending actions sorted by priority | -| `GET` | `/insights` | List insights | - -### Replay - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/replay/sessions` | Sessions list for replay tab | -| `GET` | `/replay/load` | Full session + observations (`?sessionId=`) | - -### MCP - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/mcp/tools` | MCP tool schema list | -| `POST` | `/mcp/tools` | MCP tool call dispatch | - ---- - -## Configuration - -Create `~/.agentcache/.env` (no `export` prefix needed): - -```env -# Server port -III_REST_PORT=3111 - -# Vector search — enables Gemini 768-dim embeddings -GEMINI_API_KEY=your-gemini-key - -# LLM for compression / consolidation / graph extraction -# Any one of these enables LLM features: -ANTHROPIC_API_KEY=your-anthropic-key -# OPENAI_API_KEY=your-openai-key -# GEMINI_API_KEY=your-key (same key as above works for both) - -# LLM-powered features (disabled by default — spend tokens) -CONSOLIDATION_ENABLED=true -GRAPH_EXTRACTION_ENABLED=true -AGENTCACHE_AUTO_COMPRESS=true - -# Context injection limits -TOKEN_BUDGET=2000 -MAX_OBS_PER_SESSION=500 - -# Auth — set to require Bearer token on all endpoints -AGENTCACHE_SECRET=your-secret - -# Agent scope isolation -AGENT_ID=my-agent -AGENTCACHE_AGENT_SCOPE=isolated # only see this agent's data - -# HuggingFace sync -HF_TOKEN=your-hf-token -AGENTCACHE_DATASET_REPO=username/agentcache-data -``` - -### Full Variable Reference - -| Variable | Default | Purpose | -|----------|---------|---------| -| `III_REST_PORT` / `PORT` | `3111` | API server port | -| `GEMINI_API_KEY` / `GOOGLE_API_KEY` | — | Enables 768-dim vector search | -| `AGENTCACHE_SECRET` | — | Bearer token auth on all endpoints | -| `AGENT_ID` | — | Default agent ID for scope isolation | -| `AGENTCACHE_AGENT_SCOPE=isolated` | — | Filters data to current `AGENT_ID` | -| `MAX_OBS_PER_SESSION` | `500` | Hard cap on observations per session | -| `TOKEN_BUDGET` | `2000` | Max tokens in compiled context | -| `GRAPH_EXTRACTION_ENABLED` | `false` | Knowledge graph (needs LLM) | -| `CONSOLIDATION_ENABLED` | `false` | Memory consolidation (needs LLM) | -| `AGENTCACHE_AUTO_COMPRESS` | `false` | LLM observation compression | - ---- - -## Viewer - -Built-in dashboard at **http://localhost:3111/viewer**. - -| Tab | What You See | -|-----|-------------| -| **Dashboard** | Session stats, memory counts, recent activity | -| **Sessions** | Browse sessions, inspect observations | -| **Memories** | Search, filter, and read long-term memories | -| **Graph** | Project folder visualization — nodes = folders, edges = shared concepts or parent path | -| **Timeline** | Per-session chronological observation view | -| **Lessons** | Confidence-scored lessons with decay tracking | -| **Slots** | Pinned memory slots editor | -| **Replay** | Scrub through past sessions frame by frame | - ---- - -## Deploy to HuggingFace - -This project is designed to run as a HuggingFace Space. Data is stored in an HF dataset repo and restored on every boot — so no persistent disk is needed. - -### Setup - -1. Fork this repo as a HuggingFace Space (SDK: Docker) -2. Create a dataset repo (e.g. `your-username/agentcache-data`) -3. Add Space secrets in the HF dashboard: - - | Secret | Value | - |--------|-------| - | `HF_TOKEN` | Your HF write token | - | `AGENTCACHE_DATASET_REPO` | `your-username/agentcache-data` | - | `AGENTCACHE_SECRET` | A random secret (optional but recommended) | - | `GEMINI_API_KEY` | Gemini key (optional, enables vector search) | - -4. The Space boots, restores `agentcache.db` from the dataset repo, and starts the server - -### How sync works - -`sync.py` uses mtime fingerprinting — it only uploads when the database actually changed, so there are no unnecessary uploads during idle periods. - -```bash -# Manual backup -python sync.py - -# Environment for sync -HF_TOKEN=... -AGENTCACHE_DATASET_REPO=username/agentcache-data -``` - ---- - -## Architecture - -``` -agentcache-python/ -├── src/ -│ ├── app.py Flask server — all endpoints, WebSocket broadcaster -│ ├── db.py SQLite StateKV — WAL mode, audit_log table -│ ├── functions.py Core logic — observe, remember, search, context -│ ├── search.py BM25 + Gemini vector index + HybridSearch (RRF) -│ └── viewer/ -│ └── index.html Single-file HTML dashboard (no build step) -├── sync.py HuggingFace dataset backup/restore -├── Dockerfile HF Space container -├── start.sh Boot script (restore → start server → start sync) -└── requirements.txt 6 Python dependencies, no external DB required -``` - -### Database layout - -Two SQLite tables in `~/.agentcache/agentcache.db`: - -```sql --- All data lives here, namespaced by scope -kv_store ( - scope TEXT NOT NULL, -- e.g. "mem:sessions", "mem:obs:{session_id}" - key TEXT NOT NULL, - value TEXT NOT NULL, -- JSON-serialized - PRIMARY KEY (scope, key) -) - --- Audit trail replaces Dolt git versioning -audit_log ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - ts INTEGER NOT NULL, -- unix millis - agent_id TEXT NOT NULL, - message TEXT NOT NULL -) -``` - -### Search pipeline - -``` -Query - → BM25 (always) — Porter-stemmed keyword matching - → Vector (if Gemini key) — 768-dim cosine similarity - → RRF fusion — Reciprocal Rank Fusion (k=60) - → Session diversify — max 3 results per session - → Return top-K -``` - ---- - -## vs Original agentcache - -| | agentcache (Node.js) | agentcache-python | -|---|---|---| -| Runtime | Node.js 20+ | Python 3.10+ | -| Storage | Dolt SQL (git-versioned MySQL) | SQLite WAL (single file) | -| Engine dependency | iii-engine (separate binary) | None — just Flask | -| Embeddings | 6 providers + local `@xenova/transformers` | Gemini 768-dim | -| MCP tools | 53 | 31 | -| REST endpoints | 128 | ~50 | -| Deploy | npm, Docker, fly.io, Railway, Render | Docker, HuggingFace Spaces | -| Cold boot | ~7s (iii engine warm-up) | <2s | -| Database size | ~232MB (417 Dolt chunk files) | ~20MB (single `.db` file) | -| Setup | `npm install -g @agentcache/agentcache` | `pip install -r requirements.txt` | - -Choose the Python version for: simpler setup, HF Space deployment, single-file database, no Node.js, or Python ecosystem integration. - -Choose the Node.js version for: the full 53-tool MCP surface, iii-engine observability, production multi-agent deployments, or the full auto-hook suite. - ---- - -## Contributing - -See [CONTRIBUTING.md](CONTRIBUTING.md). Issues and PRs welcome. - -Priority areas: test coverage, additional embedding providers, more agent hook scripts. - ---- - -## License - -Apache-2.0 — see [LICENSE](LICENSE). +Dynamic: license-file diff --git a/src/agentcache.egg-info/SOURCES.txt b/src/agentcache.egg-info/SOURCES.txt index e7e9efe649956cad37f90b0dfb8f857e0fe1befc..3ccfeec03f3c80f284de95f5d0dc89d69067ab47 100644 --- a/src/agentcache.egg-info/SOURCES.txt +++ b/src/agentcache.egg-info/SOURCES.txt @@ -1,5 +1,4 @@ LICENSE -README.md pyproject.toml src/agentcache/__init__.py src/agentcache/app.py @@ -26,6 +25,9 @@ src/agentcache/cache/health.py src/agentcache/cache/observe.py src/agentcache/cache/remember.py src/agentcache/cache/timeline.py +src/agentcache/core/__init__.py +src/agentcache/core/kv_scopes.py +src/agentcache/core/search_service.py src/agentcache/routes/__init__.py src/agentcache/routes/graph.py src/agentcache/routes/health.py @@ -42,6 +44,8 @@ src/agentcache/viewer/favicon.svg src/agentcache/viewer/index.html tests/test_api.py tests/test_auth.py +tests/test_auto_forget.py +tests/test_cli_context.py tests/test_context.py tests/test_debounce.py tests/test_folder_graph_build.py @@ -56,4 +60,5 @@ tests/test_properties.py tests/test_remember.py tests/test_route_regressions.py tests/test_search.py +tests/test_security.py tests/test_timeline.py \ No newline at end of file diff --git a/src/agentcache/__init__.py b/src/agentcache/__init__.py index c932a8206065b071c3285a810a2f8d5c84db72d0..ac5f88a86f66467d53838e703057888448862b78 100644 --- a/src/agentcache/__init__.py +++ b/src/agentcache/__init__.py @@ -6,13 +6,10 @@ __version__ = "0.9.8" from .app import create_app from .connect import run_connect +from .core import KV, ObservationEvents, ObservationStore, SearchService from .db import StateKV -from .functions import ( +from .legacy import ( folder_graph_build, - folder_observe, - folder_search, - folder_timeline, - forget, health_check, remember, ) @@ -22,11 +19,11 @@ __all__ = [ "create_app", "StateKV", "run_connect", - "folder_observe", - "folder_search", - "folder_timeline", + "KV", + "ObservationStore", + "ObservationEvents", + "SearchService", "folder_graph_build", "remember", - "forget", "health_check", ] diff --git a/src/agentcache/app.py b/src/agentcache/app.py index fa29e33eec3f4be1e628472887cd517e092f5221..94bcf62b2864c1d65368a634f1a99f72602875b8 100644 --- a/src/agentcache/app.py +++ b/src/agentcache/app.py @@ -13,7 +13,7 @@ import sys from flask import Flask, request, send_from_directory from flask_sock import Sock -from . import functions +from . import legacy # Prevent double-import of app when run directly as __main__ if __name__ == "__main__": @@ -41,26 +41,30 @@ def _load_env() -> None: _load_env() -# Module-level globals — set once by create_app(), read by blueprints via `import app` +# Module-level singletons — set once by init_services(), read by blueprints and workers. kv = None embedding_provider = None -persistence = None +persistence = None # kept for backward compat with workers; use search_service instead +search_service = None # SearchService instance +observation_store = None # ObservationStore instance def init_services() -> tuple: - """Initialise database, embedding provider, and index persistence.""" - global kv, embedding_provider, persistence + """Initialise database, SearchService, ObservationStore, and legacy persistence shim.""" + global kv, embedding_provider, persistence, search_service, observation_store if kv is not None: return kv, embedding_provider, persistence - from . import functions from . import search as search_mod + from .core.observation_store import ObservationEvents, ObservationStore + from .core.search_service import SearchService from .db import StateKV + from .search import SearchIndex, VectorIndex # 1. DB kv = StateKV() - # 2. Embedding provider — auto-select by priority (D5.3): + # 2. Embedding provider — auto-select by priority: # GEMINI_API_KEY → OPENAI_API_KEY → AGENTCACHE_LOCAL_EMBEDDING_MODEL → BM25-only api_key = os.getenv("GEMINI_API_KEY") or os.getenv("GOOGLE_API_KEY") openai_key = os.getenv("OPENAI_API_KEY") @@ -71,7 +75,6 @@ def init_services() -> tuple: if api_key: try: embedding_provider = search_mod.GeminiEmbeddingProvider(api_key) - functions.set_embedding_provider(embedding_provider) print( f"[search] Embedding provider active: gemini ({embedding_provider.dimensions} dims)" ) @@ -80,7 +83,6 @@ def init_services() -> tuple: elif openai_key: try: embedding_provider = search_mod.OpenAIEmbeddingProvider(openai_key) - functions.set_embedding_provider(embedding_provider) print( f"[search] Embedding provider active: openai ({embedding_provider.dimensions} dims)" ) @@ -89,7 +91,6 @@ def init_services() -> tuple: elif local_model: try: embedding_provider = search_mod.SentenceTransformerProvider(local_model) - functions.set_embedding_provider(embedding_provider) print( f"[search] Embedding provider active: sentence-transformers/{local_model} ({embedding_provider.dimensions} dims)" ) @@ -100,22 +101,27 @@ def init_services() -> tuple: else: print("[search] No embedding API key found — running in BM25-only mode.") - # 3. Index persistence — use embedding_provider variable set above - has_vector = embedding_provider is not None - persistence = functions.IndexPersistence( - kv, - functions._bm25_index, - functions._vector_index if has_vector else None, + # 3. Construct SearchService and ObservationStore with injected dependencies. + bm25 = SearchIndex() + vector = VectorIndex() if embedding_provider is not None else None + search_service = SearchService(bm25, vector, embedding_provider, kv) + observation_store = ObservationStore( + kv, search_service=search_service, events=ObservationEvents() ) - functions.set_index_persistence(persistence) - loaded = persistence.load() + + # Load persisted indexes. + loaded = search_service.load_persisted() print( f"[persistence] Load results: BM25={loaded['bm25']}, Vector={loaded['vector']}" ) - # Backfill coordinate lookup index if missing/incomplete + # Keep persistence reference for workers backward compat. + persistence = search_service._persistence + + # Backfill coordinate lookup index if missing/incomplete. try: - functions.backfill_obs_lookup_if_needed(kv) + if observation_store is not None: + observation_store.backfill_lookup() except Exception as e: print(f"[db] Warning backfilling obs_lookup: {e}") @@ -136,6 +142,9 @@ def create_app() -> Flask: # 4. Flask app + blueprints flask_app = Flask(__name__) + flask_app.extensions["observation_store"] = observation_store + flask_app.extensions["search_service"] = search_service + from werkzeug.middleware.proxy_fix import ProxyFix flask_app.wsgi_app = ProxyFix( @@ -143,7 +152,9 @@ def create_app() -> Flask: ) from .routes import register_blueprints - register_blueprints(flask_app) + register_blueprints( + flask_app, observation_store=observation_store, search_service=search_service + ) # 5. WebSocket broadcaster sock = Sock(flask_app) @@ -176,7 +187,10 @@ def create_app() -> Flask: except Exception: _ws_clients.discard(ws) - functions.set_stream_broadcaster(_broadcast) + legacy.set_stream_broadcaster(_broadcast) + + if observation_store and observation_store.events: + observation_store.events.on_added.append(_broadcast) # 6. Viewer static routes from importlib.resources import files diff --git a/src/agentcache/cache/__init__.py b/src/agentcache/cache/__init__.py index f943ba2922bf4aac9b16cc84cbb410fa5ac794da..07e37a639f9bdeb6b19806795a4ca8ef406164fc 100644 --- a/src/agentcache/cache/__init__.py +++ b/src/agentcache/cache/__init__.py @@ -18,7 +18,8 @@ callers may import from this package (A2.2). # Each name is imported lazily via a try/except so missing items don't break # the package import on partially-initialised environments. # --------------------------------------------------------------------------- -from .. import functions as _fn # noqa: E402 +from .. import legacy as _fn # noqa: E402 +from ..core import KV from .context import context, export_data, rebuild_index from .graph import folder_graph_build from .health import auto_forget, health_check @@ -31,14 +32,10 @@ from .observe import ( from .remember import forget, jaccard_similarity, remember from .timeline import folder_search, folder_timeline -KV = _fn.KV generate_id = _fn.generate_id fingerprint_id = _fn.fingerprint_id normalize_folder_path = _fn.normalize_folder_path validate_agent_id = _fn.validate_agent_id -IndexPersistence = _fn.IndexPersistence -set_embedding_provider = _fn.set_embedding_provider -set_index_persistence = _fn.set_index_persistence set_stream_broadcaster = _fn.set_stream_broadcaster get_agent_id = _fn.get_agent_id record_audit = _fn.record_audit @@ -72,15 +69,12 @@ __all__ = [ # health.py "health_check", "auto_forget", - # functions.py shims (A2.2) + # legacy.py shims "KV", "generate_id", "fingerprint_id", "normalize_folder_path", "validate_agent_id", - "IndexPersistence", - "set_embedding_provider", - "set_index_persistence", "set_stream_broadcaster", "get_agent_id", "record_audit", diff --git a/src/agentcache/cache/context.py b/src/agentcache/cache/context.py index 323623ece93e6548b458fa85de258145aa0c178a..bc1b8f67938d4d3b629059db0a5804266900b239 100644 --- a/src/agentcache/cache/context.py +++ b/src/agentcache/cache/context.py @@ -11,7 +11,7 @@ from __future__ import annotations from typing import Any, Dict -from .. import functions as _fn +from .. import legacy as _fn from ..db import StateKV diff --git a/src/agentcache/cache/graph.py b/src/agentcache/cache/graph.py index 525b5aa3acc4c2459c5db1dd5745337a9efafa97..c948bf582cbb4f5f82002ba80073805839646570 100644 --- a/src/agentcache/cache/graph.py +++ b/src/agentcache/cache/graph.py @@ -9,7 +9,7 @@ from __future__ import annotations from typing import Any, Dict -from .. import functions as _fn +from .. import legacy as _fn from ..db import StateKV diff --git a/src/agentcache/cache/health.py b/src/agentcache/cache/health.py index 33b90f57e437917d1e758db0f4ea49c7c902da5b..94cf838e73270100a2d790c79378f6abcad26d72 100644 --- a/src/agentcache/cache/health.py +++ b/src/agentcache/cache/health.py @@ -10,7 +10,7 @@ from __future__ import annotations from typing import Any, Dict -from .. import functions as _fn +from .. import legacy as _fn from ..db import StateKV diff --git a/src/agentcache/cache/observe.py b/src/agentcache/cache/observe.py index 8eda41294eebe8f8383a5adb33c3d04dff372f65..872c342bc5f731a41497923f966a50e85fc24f4f 100644 --- a/src/agentcache/cache/observe.py +++ b/src/agentcache/cache/observe.py @@ -12,7 +12,7 @@ from __future__ import annotations from typing import Any, Dict -from .. import functions as _fn # access module-level globals (_bm25_index, etc.) +from .. import legacy as _fn # access module-level globals (_bm25_index, etc.) from ..db import StateKV # Re-export for backward compatibility diff --git a/src/agentcache/cache/remember.py b/src/agentcache/cache/remember.py index cfc7fd0acede8b16e6869382bd8ecfc1f269a31c..c30723c838f4b4906b9b928834356a2aaf94c19a 100644 --- a/src/agentcache/cache/remember.py +++ b/src/agentcache/cache/remember.py @@ -11,7 +11,7 @@ from __future__ import annotations from typing import Any, Dict -from .. import functions as _fn +from .. import legacy as _fn from ..db import StateKV diff --git a/src/agentcache/cache/timeline.py b/src/agentcache/cache/timeline.py index f2af4cc754599b1c84a0170e6350603e2aee9488..db9404841e3809427a6ce3b71f42d10081c3e48f 100644 --- a/src/agentcache/cache/timeline.py +++ b/src/agentcache/cache/timeline.py @@ -10,7 +10,7 @@ from __future__ import annotations from typing import Any, Dict, List, Optional -from .. import functions as _fn +from .. import legacy as _fn from ..db import StateKV diff --git a/src/agentcache/cli.py b/src/agentcache/cli.py index b8b3ae7b0d745af817750ab5bf21082c7ad78b35..d7f884615b4f52d9482dfaeede699aa4fad132b7 100644 --- a/src/agentcache/cli.py +++ b/src/agentcache/cli.py @@ -28,7 +28,7 @@ def cmd_serve(args) -> None: def cmd_migrate(args) -> None: """Run session → folder migration.""" from .db import StateKV - from .functions import migrate_sessions_to_folders + from .legacy import migrate_sessions_to_folders kv = StateKV() result = migrate_sessions_to_folders(kv, dry_run=args.dry_run) @@ -54,7 +54,7 @@ def cmd_migrate(args) -> None: def cmd_export(args) -> None: """Export all data as JSON.""" from .db import StateKV - from .functions import export_data + from .legacy import export_data kv = StateKV() data = export_data(kv, {}) @@ -113,7 +113,8 @@ def cmd_context(args) -> None: import time from .app import init_services - from .functions import KV, normalize_folder_path + from .core import KV + from .core.observation_store import normalize_folder_path # 1. Resolve folder and agent cwd = os.getcwd() diff --git a/src/agentcache/core/__init__.py b/src/agentcache/core/__init__.py index b12d4d43928b88b174673c47c709794f655484b8..f8c6e6f959f52d7d41f745ef55a0a4165639e952 100644 --- a/src/agentcache/core/__init__.py +++ b/src/agentcache/core/__init__.py @@ -10,5 +10,10 @@ from .kv_scopes import KV from .observation_store import ObservationEvents, ObservationStore from .search_service import IndexPersistence, SearchService -__all__ = ["KV", "SearchService", "IndexPersistence", "ObservationStore", "ObservationEvents"] - +__all__ = [ + "KV", + "SearchService", + "IndexPersistence", + "ObservationStore", + "ObservationEvents", +] diff --git a/src/agentcache/core/kv_scopes.py b/src/agentcache/core/kv_scopes.py index d06d856206cdd66150eba5e6d062bee14465143e..de133069d0663dc4e90cf053562d0e01cb3c514e 100644 --- a/src/agentcache/core/kv_scopes.py +++ b/src/agentcache/core/kv_scopes.py @@ -5,8 +5,6 @@ Single source of truth for every SQLite scope string used in the system. Import this module wherever a KV scope key is needed — routes, stores, workers. """ -from typing import Optional - class KV: # ---- Folder memory scopes ---- diff --git a/src/agentcache/core/observation_store.py b/src/agentcache/core/observation_store.py index 876bd82b580d85115a8792b27eeedee8191272e2..227faf179a02760d99dfff296ebd21d1ffb9d108 100644 --- a/src/agentcache/core/observation_store.py +++ b/src/agentcache/core/observation_store.py @@ -104,8 +104,7 @@ class ObservationStore: folder_path = normalize_folder_path(folder_path_raw) agent_id = validate_agent_id(agent_id_raw) - from ..legacy import strip_private_data, infer_type, extract_files - + from ..legacy import extract_files, infer_type, strip_private_data safe_text = strip_private_data(text_raw)[:4000] @@ -116,7 +115,11 @@ class ObservationStore: dedup_lock = self._get_dedup_lock(folder_path, agent_id) with dedup_lock: existing_dedup = self.kv.get(KV.obs_dedup(folder_path, agent_id), dedup_fp) - if existing_dedup and isinstance(existing_dedup, dict) and existing_dedup.get("obsId"): + if ( + existing_dedup + and isinstance(existing_dedup, dict) + and existing_dedup.get("obsId") + ): return {"observationId": existing_dedup["obsId"], "deduplicated": True} max_obs = int(os.getenv("MAX_OBS_PER_FOLDER", "2000")) @@ -267,7 +270,6 @@ class ObservationStore: total_kept = 0 for pair in pairs: - fp = pair["folderPath"] aid = pair["agentId"] all_obs = self.kv.list(KV.folder_obs(fp, aid)) @@ -292,7 +294,9 @@ class ObservationStore: duplicates.append(obs["id"]) if duplicates: - self.forget({"folderPath": fp, "agentId": aid, "observationIds": duplicates}) + self.forget( + {"folderPath": fp, "agentId": aid, "observationIds": duplicates} + ) total_removed += len(duplicates) total_kept += len(fingerprint_map) @@ -305,7 +309,6 @@ class ObservationStore: {"obsId": obs["id"], "timestamp": obs.get("timestamp", "")}, ) - return { "success": True, "deduplicated": total_removed, @@ -323,7 +326,6 @@ class ObservationStore: deleted = 0 deleted_mem_ids: List[str] = [] deleted_obs_ids: List[str] = [] - deleted_session = False if memory_id: mem = self.kv.get(KV.memories, memory_id) @@ -351,8 +353,6 @@ class ObservationStore: if "observationIds" in data and data["observationIds"] is not None: partial_deleted = 0 for oid in obs_ids: - - obs = self.kv.get(obs_scope, oid) existed = self.kv.delete(obs_scope, oid) if existed: @@ -385,7 +385,9 @@ class ObservationStore: try: cb(deleted_obs_ids) except Exception as ex: - print(f"[observation_store] Error in on_deleted callback: {ex}") + print( + f"[observation_store] Error in on_deleted callback: {ex}" + ) else: all_obs = self.kv.list(obs_scope) for obs in all_obs: @@ -411,7 +413,9 @@ class ObservationStore: try: cb(fp, aid) except Exception as ex: - print(f"[observation_store] Error in on_folder_deleted callback: {ex}") + print( + f"[observation_store] Error in on_folder_deleted callback: {ex}" + ) if session_id and obs_ids: for oid in obs_ids: @@ -454,7 +458,6 @@ class ObservationStore: deleted += 1 self.kv.delete(KV.sessions, session_id) self.kv.delete(KV.summaries, session_id) - deleted_session = True deleted += 2 if deleted > 0 and self.search_service: @@ -474,7 +477,9 @@ class ObservationStore: index_entries = self.kv.list(KV.folders) if folder_path is not None: - index_entries = [e for e in index_entries if e.get("folderPath") == folder_path] + index_entries = [ + e for e in index_entries if e.get("folderPath") == folder_path + ] if agent_id is not None: index_entries = [e for e in index_entries if e.get("agentId") == agent_id] @@ -568,5 +573,3 @@ class ObservationStore: self.search_service.schedule_persist() return total_indexed - - diff --git a/src/agentcache/core/search_service.py b/src/agentcache/core/search_service.py index a80da9da1171e166d2235222e3ee738222d41568..cde1f10051a6fe2c6c7750adc81e39df45a0daf3 100644 --- a/src/agentcache/core/search_service.py +++ b/src/agentcache/core/search_service.py @@ -8,7 +8,6 @@ MemoryStore, routes) call this service instead of touching index globals. from __future__ import annotations import json -import math import sqlite3 import threading import time @@ -24,7 +23,9 @@ class IndexPersistence: DEBOUNCE_SECONDS: float = 5.0 - def __init__(self, kv: Any, bm25: SearchIndex, vector: Optional[VectorIndex] = None): + def __init__( + self, kv: Any, bm25: SearchIndex, vector: Optional[VectorIndex] = None + ): self.kv = kv self.bm25 = bm25 self.vector = vector @@ -149,14 +150,20 @@ class IndexPersistence: cursor.close() except sqlite3.OperationalError as ex: err_msg = str(ex).lower() - if ("locked" in err_msg or "busy" in err_msg) and attempt < max_retries - 1: + if ( + "locked" in err_msg or "busy" in err_msg + ) and attempt < max_retries - 1: time.sleep(delay) delay *= 2 continue - print(f"[index persistence] error cleaning up obsolete shards: {ex}") + print( + f"[index persistence] error cleaning up obsolete shards: {ex}" + ) break except Exception as ex: - print(f"[index persistence] error cleaning up obsolete shards: {ex}") + print( + f"[index persistence] error cleaning up obsolete shards: {ex}" + ) break if ( @@ -372,7 +379,11 @@ class SearchService: results.append(result) seen_ids.add(obs_id) # Lazy backfill lookup - active_kv.set(KV.obs_lookup, obs_id, {"folderPath": fp, "agentId": aid}) + active_kv.set( + KV.obs_lookup, + obs_id, + {"folderPath": fp, "agentId": aid}, + ) found = True break if found: diff --git a/src/agentcache/functions.py b/src/agentcache/functions.py deleted file mode 100644 index 1387db676e72b73160d9bcf8a3c3b7a8fc1f2f56..0000000000000000000000000000000000000000 --- a/src/agentcache/functions.py +++ /dev/null @@ -1,4583 +0,0 @@ -import datetime -import hashlib -import json -import os -import re -import sqlite3 -import threading -import time -import uuid -from typing import Any, Dict, List, Optional, Set, Tuple - -from .db import StateKV -from .search import HybridSearch, SearchIndex, VectorIndex - -# ===================================================================== -# Global Variables / Module State -# ===================================================================== -_bm25_index = SearchIndex() -_vector_index = VectorIndex() -_embedding_provider = None -_hybrid_search = HybridSearch(_bm25_index, _vector_index, None, None) -_index_persistence = None -_stream_broadcaster = None # Callable: (payload) -> None -_dedup_locks: Dict[str, threading.Lock] = {} # per-(folder, agent) write locks -_dedup_locks_meta = threading.Lock() # protects _dedup_locks dict itself - - -# KV scope registry — folder-based memory model -class KV: - # ---- Folder memory scopes (new) ---- - - # Global index of all (folder_path, agent_id) pairs known to the system. - # Key = "{safe_folder_path}:{agent_id}", value = FolderIndexEntry dict. - folders = "mem:folders" - - # Lookup index for O(1) observation hydration. - # Scope = "mem:obs_lookup", Key = obs_id, Value = {"folderPath": folder_path, "agentId": agent_id} - obs_lookup = "mem:obs_lookup" - - @staticmethod - def folder_obs(folder_path: str, agent_id: str) -> str: - """Per-(folder, agent) observations scope. - Key = obs_id, value = FolderObservation dict. - """ - safe_path = folder_path.replace("\\", "/").strip("/") - safe_agent = agent_id.strip() - return f"mem:folder:{safe_path}:{safe_agent}" - - @staticmethod - def folder_meta(folder_path: str, agent_id: str) -> str: - """Per-(folder, agent) metadata scope. - Key = "meta", value = FolderMeta dict (obsCount, lastUpdated, summary). - """ - safe_path = folder_path.replace("\\", "/").strip("/") - safe_agent = agent_id.strip() - return f"mem:foldermeta:{safe_path}:{safe_agent}" - - @staticmethod - def obs_dedup(folder_path: str, agent_id: str) -> str: - """Deduplication index scope for (folder, agent) pairs. - Key = SHA-256 fingerprint hex of normalized text. - Value = {"obsId": str, "timestamp": str} - """ - safe_path = folder_path.replace("\\", "/").strip("/") - safe_agent = agent_id.strip() - return f"mem:obs_dedup:{safe_path}:{safe_agent}" - - # ---- Global / shared scopes (kept) ---- - - # Long-term memories — unchanged from previous implementation. - memories = "mem:memories" - - # BM25 index shards — unchanged. - bm25Index = "mem:index:bm25" - - # Audit log — unchanged. - audit = "mem:audit" - - # Graph edges — repurposed for folder graph edges. - relations = "mem:relations" - - # ---- Legacy scopes (read-only; kept for migration and backward compat) ---- - - # Legacy session store — read by migrate_sessions_to_folders() and legacy observe(). - sessions = "mem:sessions" - - @staticmethod - def observations(session_id: str) -> str: - """Legacy per-session observations scope. - Key = obs_id, value = raw/synthetic observation dict. - Read by migrate_sessions_to_folders() and legacy observe(). - """ - return f"mem:obs:{session_id}" - - # Lessons — confidence-scored learning entries. - lessons = "mem:lessons" - - # Legacy summary / profile / slot / image-ref scopes retained for legacy code paths. - summaries = "mem:summaries" - profiles = "mem:profiles" - slots = "mem:slots" - imageRefs = "mem:image-refs" - - # Global (cross-project) pinned slots. - globalSlots = "mem:global-slots" - - -def get_current_project(kv: StateKV) -> Optional[str]: - try: - sessions = kv.list(KV.sessions) - if not sessions: - return None - active_sessions = [s for s in sessions if s.get("status") == "active"] - if active_sessions: - active_sessions.sort(key=lambda s: s.get("updatedAt", ""), reverse=True) - return active_sessions[0].get("project") - sessions.sort(key=lambda s: s.get("updatedAt", ""), reverse=True) - return sessions[0].get("project") - except Exception: - return None - - -def project_slots_scope(kv: StateKV, project: Optional[str] = None) -> str: - if not project: - project = get_current_project(kv) - if not project: - return KV.slots - return f"mem:slots:{project}" - - -# ===================================================================== -# Core Helpers & Utilities -# ===================================================================== - - -def generate_id(prefix: str) -> str: - t = int(time.time() * 1000) - chars = "0123456789abcdefghijklmnopqrstuvwxyz" - ts_str = "" - while t > 0: - ts_str = chars[t % 36] + ts_str - t //= 36 - if not ts_str: - ts_str = "0" - rand = uuid.uuid4().hex[:12] - return f"{prefix}_{ts_str}_{rand}" - - -def fingerprint_id(prefix: str, content: str) -> str: - h = hashlib.sha256(content.strip().lower().encode("utf-8")).hexdigest() - return f"{prefix}_{h[:16]}" - - -# ---- Folder-path normalisation (REQ-002, REQ-063, REQ-064, REQ-066) ---- - -_MAX_PATH_LEN = 512 - - -def normalize_folder_path(path: str) -> str: - """Normalize a folder path for safe use in KV scope keys. - - Steps applied in order: - 1. Cap the raw input at 512 characters (REQ-066). - 2. Apply ``os.path.normpath`` to collapse redundant separators and - resolve any ``..`` components at the OS level. - 3. Convert all OS-native separators to forward slashes. - 4. Strip any remaining leading or trailing slashes. - - Raises: - ValueError: if *path* is empty (before or after normalization), or - if the normalized result still contains a ``..`` segment, - which would indicate an attempt at path traversal - (REQ-064). - - Returns: - A non-empty, forward-slash-separated string with no leading/trailing - slashes and no ``..`` segments — safe for use as a KV scope fragment. - - Property (REQ-074): idempotent — applying this function twice yields - the same result as applying it once. - """ - if not path: - raise ValueError("folder_path must not be empty") - - # 1. Length cap before any processing. - path = path[:_MAX_PATH_LEN] - - # Pre-normalisation traversal check: reject any path that contains a ".." - # component in the raw input before normpath has a chance to resolve it. - # This catches inputs like "/home/user/../../etc/passwd" which normpath - # would silently resolve to "etc/passwd" (REQ-064). - raw_parts = path.replace("\\", "/").split("/") - if any(part == ".." for part in raw_parts): - raise ValueError(f"folder_path contains path traversal segment '..': {path!r}") - - # 2. OS-level normalisation (resolves duplicate separators, etc.) - normalized = os.path.normpath(path) - - # 3. Unify separators to forward slash. - normalized = normalized.replace("\\", "/") - - # 4. Strip leading / trailing slashes. - normalized = normalized.strip("/") - - # Guard: also reject any ".." that somehow survives normalisation. - parts = normalized.split("/") - if any(part == ".." for part in parts): - raise ValueError(f"folder_path contains path traversal segment '..': {path!r}") - - if not normalized: - raise ValueError("folder_path is empty after normalization") - - return normalized - - -def validate_agent_id(agent_id: str) -> str: - """Validate and sanitize an agent_id before use in KV scope keys. - - Strips surrounding whitespace and caps at 512 characters (REQ-066). - - Raises: - ValueError: if *agent_id* is empty after stripping. - - Returns: - Sanitized agent_id string. - """ - if not agent_id: - raise ValueError("agent_id must not be empty") - - sanitized = agent_id.strip()[:_MAX_PATH_LEN] - - if not sanitized: - raise ValueError("agent_id is empty after stripping whitespace") - - return sanitized - - -def _get_dedup_lock(folder_path: str, agent_id: str) -> threading.Lock: - """Return a per-(folder_path, agent_id) Lock, creating it if necessary. - - Uses _dedup_locks_meta to protect concurrent creation of new lock entries. - """ - key = f"{folder_path}:{agent_id}" - with _dedup_locks_meta: - if key not in _dedup_locks: - _dedup_locks[key] = threading.Lock() - return _dedup_locks[key] - - -def auto_complete_old_active_sessions( - kv: StateKV, - current_session_id: str, - project: Optional[str] = None, - agent_id: Optional[str] = None, -) -> int: - sessions = kv.list(KV.sessions) - count = 0 - now = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - for s in sessions: - if s.get("id") != current_session_id and s.get("status") == "active": - if project and s.get("project") != project: - continue - if agent_id and s.get("agentId") != agent_id: - continue - s["status"] = "completed" - if "endedAt" not in s: - s["endedAt"] = now - s["updatedAt"] = now - kv.set(KV.sessions, s["id"], s) - count += 1 - if count > 0: - print(f"[session] Auto-completed {count} dangling active sessions.") - return count - - -def jaccard_similarity(a: str, b: str) -> float: - tokens_a = [t for t in a.split() if len(t) > 2] - tokens_b = [t for t in b.split() if len(t) > 2] - set_a = set(tokens_a) - set_b = set(tokens_b) - if not set_a and not set_b: - return 1.0 - if not set_a or not set_b: - return 0.0 - intersection = len(set_a.intersection(set_b)) - union = len(set_a.union(set_b)) - return intersection / union - - -# ===================================================================== -# Privacy & Data Scrubbing -# ===================================================================== - -PRIVATE_TAG_RE = re.compile(r"[\s\S]*?", re.IGNORECASE) - -SECRET_PATTERN_SOURCES = [ - re.compile( - r'(?:api[_-]?key|secret|token|password|credential|auth)[\s]*[=:]\s*["\']?[A-Za-z0-9_\-/.+]{20,}["\']?', - re.IGNORECASE, - ), - re.compile(r"Bearer\s+[A-Za-z0-9._\-+/=]{20,}", re.IGNORECASE), - re.compile(r"sk-proj-[A-Za-z0-9\-_]{20,}", re.IGNORECASE), - re.compile(r"(?:sk|pk|rk|ak)-[A-Za-z0-9][A-Za-z0-9\-_]{19,}", re.IGNORECASE), - re.compile(r"sk-ant-[A-Za-z0-9\-_]{20,}", re.IGNORECASE), - re.compile(r"gh[pus]_[A-Za-z0-9]{36,}", re.IGNORECASE), - re.compile(r"github_pat_[A-Za-z0-9_]{22,}", re.IGNORECASE), - re.compile(r"xoxb-[A-Za-z0-9\-]+", re.IGNORECASE), - re.compile(r"AKIA[0-9A-Z]{16}", re.IGNORECASE), - re.compile(r"AIza[A-Za-z0-9\-_]{35}", re.IGNORECASE), - re.compile( - r"eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}", re.IGNORECASE - ), - re.compile(r"npm_[A-Za-z0-9]{36}", re.IGNORECASE), - re.compile(r"glpat-[A-Za-z0-9\-_]{20,}", re.IGNORECASE), - re.compile(r"dop_v1_[A-Za-z0-9]{64}", re.IGNORECASE), -] - - -def strip_private_data(input_str: str) -> str: - result = PRIVATE_TAG_RE.sub("[REDACTED]", input_str) - for pattern in SECRET_PATTERN_SOURCES: - result = pattern.sub("[REDACTED_SECRET]", result) - return result - - -# ===================================================================== -# Audit Log System -# ===================================================================== - - -def record_audit( - kv: StateKV, - operation: str, - function_id: str, - target_ids: List[str], - details: Dict[str, Any] = {}, - quality_score: Optional[float] = None, - user_id: Optional[str] = None, -) -> Dict[str, Any]: - entry = { - "id": generate_id("aud"), - "timestamp": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - "operation": operation, - "userId": user_id, - "functionId": function_id, - "targetIds": target_ids, - "details": details, - "qualityScore": quality_score, - } - kv.set(KV.audit, entry["id"], entry) - return entry - - -def safe_audit( - kv: StateKV, - operation: str, - function_id: str, - target_ids: List[str], - details: Dict[str, Any] = {}, - quality_score: Optional[float] = None, - user_id: Optional[str] = None, -) -> None: - try: - record_audit( - kv, operation, function_id, target_ids, details, quality_score, user_id - ) - except Exception as e: - print(f"[audit] Failed to write audit: {e}") - - -def query_audit( - kv: StateKV, filter_opts: Optional[Dict[str, Any]] = None -) -> List[Dict[str, Any]]: - all_entries = kv.list(KV.audit) - entries = sorted(all_entries, key=lambda x: x.get("timestamp", ""), reverse=True) - if not filter_opts: - return entries[:100] - - op = filter_opts.get("operation") - if op: - entries = [e for e in entries if e.get("operation") == op] - - import dateutil.parser - - date_from = filter_opts.get("dateFrom") - if date_from: - try: - dt_from = dateutil.parser.parse(date_from).replace(tzinfo=None) - filtered_entries = [] - for e in entries: - ts = e.get("timestamp") - if ts: - try: - dt_ts = dateutil.parser.parse(ts).replace(tzinfo=None) - if dt_ts >= dt_from: - filtered_entries.append(e) - except Exception: - pass - entries = filtered_entries - except Exception: - pass - - date_to = filter_opts.get("dateTo") - if date_to: - try: - dt_to = dateutil.parser.parse(date_to).replace(tzinfo=None) - filtered_entries = [] - for e in entries: - ts = e.get("timestamp") - if ts: - try: - dt_ts = dateutil.parser.parse(ts).replace(tzinfo=None) - if dt_ts <= dt_to: - filtered_entries.append(e) - except Exception: - pass - entries = filtered_entries - except Exception: - pass - - limit = filter_opts.get("limit", 100) - return entries[:limit] - - -# ===================================================================== -# Image Store System -# ===================================================================== - -IMAGES_DIR = os.path.join(os.path.expanduser("~"), ".agentcache", "images") - - -def get_max_bytes() -> int: - return int( - os.getenv("AGENTCACHE_IMAGE_STORE_MAX_BYTES") - or os.getenv("AGENTMEMORY_IMAGE_STORE_MAX_BYTES") - or 500 * 1024 * 1024 - ) - - -def is_managed_image_path(file_path: str) -> bool: - if not file_path: - return False - resolved = os.path.abspath(file_path) - normalized_images_dir = os.path.abspath(IMAGES_DIR) - return ( - resolved.startswith(normalized_images_dir + os.sep) - or resolved == normalized_images_dir - ) - - -def save_image_to_disk(base64_data: str) -> Tuple[str, int]: - if not base64_data: - return "", 0 - - if not os.path.exists(IMAGES_DIR): - os.makedirs(IMAGES_DIR, exist_ok=True) - - clean_base64 = base64_data - ext = "png" - - if base64_data.startswith("data:image/"): - comma_idx = base64_data.find(",") - if comma_idx != -1: - meta = base64_data[:comma_idx] - if "jpeg" in meta or "jpg" in meta: - ext = "jpg" - elif "webp" in meta: - ext = "webp" - elif "gif" in meta: - ext = "gif" - clean_base64 = base64_data[comma_idx + 1 :] - elif base64_data.startswith("/9j/"): - ext = "jpg" - - h = hashlib.sha256(clean_base64.encode("utf-8")).hexdigest() - file_path = os.path.join(IMAGES_DIR, f"{h}.{ext}") - - if os.path.exists(file_path): - return file_path, 0 - - import base64 - - buffer = base64.b64decode(clean_base64) - with open(file_path, "wb") as f: - f.write(buffer) - - size = os.path.getsize(file_path) - return file_path, size - - -def delete_image(file_path: Optional[str]) -> int: - if not file_path or not is_managed_image_path(file_path): - return 0 - try: - if os.path.exists(file_path): - size = os.path.getsize(file_path) - os.remove(file_path) - return size - except Exception as e: - print(f"[agentcache] Failed to delete image context: {e}") - return 0 - - -def touch_image(file_path: str) -> None: - if not file_path or not is_managed_image_path(file_path): - return - try: - if os.path.exists(file_path): - os.utime(file_path, None) - except Exception: - pass - - -# ===================================================================== -# Index Persistence System (JSON Sharded) -# ===================================================================== - - -class IndexPersistence: - """Persist BM25 and vector indexes to the KV store with a debounce queue. - - A4.1: schedule_save() uses a threading.Timer that resets on each call and - fires the actual save() after DEBOUNCE_SECONDS of inactivity. This prevents - a persistence write on every single observation under high throughput. - - A4.2: save() skips writing an index that has not been dirtied since the last - save (relies on SearchIndex._dirty / VectorIndex._dirty flags). - """ - - DEBOUNCE_SECONDS: float = 5.0 - - def __init__(self, kv: StateKV, bm25: SearchIndex, vector: Optional[VectorIndex]): - self.kv = kv - self.bm25 = bm25 - self.vector = vector - self._timer: Optional[threading.Timer] = None - self._timer_lock = threading.Lock() - - def schedule_save(self) -> None: - """Schedule a debounced save — resets the 5-second timer on each call.""" - with self._timer_lock: - if self._timer is not None: - self._timer.cancel() - self._timer = threading.Timer(self.DEBOUNCE_SECONDS, self._fire_save) - self._timer.daemon = True - self._timer.start() - - def _fire_save(self) -> None: - """Called by the timer after DEBOUNCE_SECONDS of inactivity.""" - with self._timer_lock: - self._timer = None - self.save() - - def flush(self) -> None: - """Cancel any pending debounce timer and save immediately (used on shutdown).""" - with self._timer_lock: - if self._timer is not None: - self._timer.cancel() - self._timer = None - self.save() - - def save(self) -> None: - try: - # A4.2: skip save if neither index is dirty - bm25_dirty = getattr(self.bm25, "_dirty", True) - vector_dirty = self.vector and getattr(self.vector, "_dirty", True) - - if bm25_dirty: - self.save_sharded_index( - json.dumps(self.bm25.serialize_data()), - "data:manifest", - "data", - "mem:index:bm25:bm25:", - ) - self.bm25._dirty = False # A4.2 — reset after save - - if self.vector and vector_dirty: - self.save_sharded_index( - json.dumps(self.vector.serialize_data()), - "vectors:manifest", - "vectors", - "mem:index:bm25:vectors:", - ) - self.vector._dirty = False # A4.2 — reset after save - - if not bm25_dirty and not vector_dirty: - print("[index persistence] indexes not dirty — skipping save") - except Exception as e: - print(f"[index persistence] failed to save index: {e}") - - def save_sharded_index( - self, serialized: str, manifest_key: str, legacy_key: str, scope_prefix: str - ) -> None: - previous = self.kv.get(KV.bm25Index, manifest_key) - generation = generate_id("idx") - chunk_chars = 2000000 - shards = [] - chunks = [] - - offset = 0 - shard_idx = 0 - while offset < len(serialized): - scope = f"{scope_prefix}{generation}:{str(shard_idx).zfill(5)}" - chunk = serialized[offset : offset + chunk_chars] - shards.append({"scope": scope, "key": "data", "chars": len(chunk)}) - chunks.append(chunk) - offset += chunk_chars - shard_idx += 1 - - for shard, chunk in zip(shards, chunks): - self.kv.set(shard["scope"], shard["key"], chunk) - - next_manifest = { - "v": 1, - "generation": generation, - "shards": shards, - "chars": len(serialized), - } - - self.kv.set(KV.bm25Index, manifest_key, next_manifest) - self.kv.delete(KV.bm25Index, legacy_key) - - # Cleanup ALL obsolete shards starting with scope_prefix that are NOT in the current shards - with self.kv._lock: - max_retries = 5 - delay = 0.05 - for attempt in range(max_retries): - try: - conn = self.kv._get_conn() - cursor = conn.cursor() - try: - cursor.execute( - "SELECT DISTINCT scope FROM kv_store WHERE scope LIKE ?", - (scope_prefix + "%",), - ) - rows = cursor.fetchall() - current_scopes = {s["scope"] for s in shards} - to_delete = [] - for row in rows: - scope_name = row["scope"] - if scope_name not in current_scopes: - to_delete.append(scope_name) - - if to_delete: - for i in range(0, len(to_delete), 50): - chunk_delete = to_delete[i : i + 50] - format_strings = ",".join(["?"] * len(chunk_delete)) - cursor.execute( - f"DELETE FROM kv_store WHERE scope IN ({format_strings})", # nosec B608 - tuple(chunk_delete), - ) - conn.commit() - break # Success - finally: - cursor.close() - except sqlite3.OperationalError as ex: - err_msg = str(ex).lower() - if ( - "locked" in err_msg or "busy" in err_msg - ) and attempt < max_retries - 1: - time.sleep(delay) - delay *= 2 - continue - print( - f"[index persistence] error cleaning up obsolete shards: {ex}" - ) - break - except Exception as ex: - print( - f"[index persistence] error cleaning up obsolete shards: {ex}" - ) - break - - if ( - previous - and isinstance(previous, dict) - and previous.get("v") == 1 - and isinstance(previous.get("shards"), list) - ): - current_shards = {(s["scope"], s["key"]) for s in shards} - for old_shard in previous["shards"]: - if (old_shard["scope"], old_shard["key"]) not in current_shards: - self.kv.delete(old_shard["scope"], old_shard["key"]) - - def load(self) -> Dict[str, Any]: - bm25_data = self.load_sharded_data("data", "data:manifest") - bm25_loaded = False - if bm25_data: - try: - self.bm25.restore_from_data(json.loads(bm25_data)) - bm25_loaded = True - except Exception as e: - print(f"[index persistence] failed to restore BM25: {e}") - - vector_loaded = False - if self.vector: - vector_data = self.load_sharded_data("vectors", "vectors:manifest") - if vector_data: - try: - self.vector.restore_from_data(json.loads(vector_data)) - vector_loaded = True - except Exception as e: - print(f"[index persistence] failed to restore vectors: {e}") - - return {"bm25": bm25_loaded, "vector": vector_loaded} - - def load_sharded_data(self, legacy_key: str, manifest_key: str) -> Optional[str]: - manifest = self.kv.get(KV.bm25Index, manifest_key) - if manifest and isinstance(manifest, dict) and manifest.get("v") == 1: - shards = manifest.get("shards", []) - chunks = [] - for shard in shards: - chunk = self.kv.get(shard["scope"], shard["key"]) - if chunk is None: - return None - chunks.append(chunk) - return "".join(chunks) - - legacy = self.kv.get(KV.bm25Index, legacy_key) - if isinstance(legacy, str): - return legacy - return None - - -# ===================================================================== -# Vector Index / Embedding Helpers -# ===================================================================== - - -def clip_embed_input(text: str) -> str: - EMBED_MAX_CHARS = 16000 - if len(text) <= EMBED_MAX_CHARS: - return text - return text[:EMBED_MAX_CHARS] - - -def get_agent_id() -> Optional[str]: - return os.getenv("AGENT_ID") or None - - -def commit_if_enabled( - kv: StateKV, message: str, agent_id: Optional[str] -) -> Optional[str]: - return kv.commit_version(message, agent_id or "unknown-agent") - - -def is_agent_scope_isolated() -> bool: - return ( - os.getenv("AGENTCACHE_AGENT_SCOPE") or os.getenv("AGENTMEMORY_AGENT_SCOPE") - ) == "isolated" - - -def is_auto_compress_enabled() -> bool: - return ( - os.getenv("AGENTCACHE_AUTO_COMPRESS") or os.getenv("AGENTMEMORY_AUTO_COMPRESS") - ) == "true" - - -def is_slots_enabled() -> bool: - return (os.getenv("AGENTCACHE_SLOTS") or os.getenv("AGENTMEMORY_SLOTS")) == "true" - - -def is_reflect_enabled() -> bool: - return ( - os.getenv("AGENTCACHE_REFLECT") or os.getenv("AGENTMEMORY_REFLECT") - ) == "true" - - -def is_graph_extraction_enabled() -> bool: - return os.getenv("GRAPH_EXTRACTION_ENABLED") == "true" - - -def is_consolidation_enabled() -> bool: - val = os.getenv("CONSOLIDATION_ENABLED") - if val in ("false", "0"): - return False - if val in ("true", "1"): - return True - return bool(os.getenv("GEMINI_API_KEY") or os.getenv("GOOGLE_API_KEY")) - - -def vector_index_add_guarded( - obs_id: str, session_id: str, text: str, context: Dict[str, Any] -) -> bool: - vi = _vector_index - ep = _embedding_provider - if not vi or not ep: - return False - try: - clipped = clip_embed_input(text) - embedding = ep.embed(clipped) - if len(embedding) != ep.dimensions: - print( - f"[vector-index] Dimension mismatch: expected {ep.dimensions}, got {len(embedding)}" - ) - return False - vi.add(obs_id, session_id, embedding) - return True - except Exception as e: - print(f"[vector-index] Embed failed: {e}") - return False - - -# ===================================================================== -# Observation System (Observe, Synthetic Compression) -# ===================================================================== - - -def extract_image(d: Any) -> Optional[str]: - if not d: - return None - if isinstance(d, str): - if ( - d.startswith("data:image/") - or d.startswith("iVBORw0KGgo") - or d.startswith("/9j/") - ): - return d - return None - if isinstance(d, dict): - for k in ["image_data", "image_path", "imageBase64", "imagePath"]: - if isinstance(d.get(k), str): - return d[k] - for key, val in d.items(): - match = extract_image(val) - if match: - return match - return None - - -def infer_type(tool_name: Optional[str], hook_type: str) -> str: - if hook_type == "post_tool_failure": - return "error" - if hook_type == "prompt_submit": - return "conversation" - if hook_type in ("subagent_stop", "task_completed"): - return "subagent" - if hook_type == "notification": - return "notification" - - if not tool_name: - return "other" - - n = re.sub(r"([a-z])([A-Z])", r"\1_\2", tool_name) - n = re.sub(r"[-\s]+", "_", n).lower() - - def has_word(word: str) -> bool: - return ( - bool(re.search(rf"(^|_){word}(_|$)", n)) - or n == word - or n.endswith(word) - or n.startswith(word) - ) - - if any(has_word(w) for w in ["fetch", "http", "web"]): - return "web_fetch" - if any(has_word(w) for w in ["grep", "search", "glob", "find"]): - return "search" - if any(has_word(w) for w in ["bash", "shell", "exec", "run"]): - return "command_run" - if any(has_word(w) for w in ["edit", "update", "patch", "replace"]): - return "file_edit" - if any(has_word(w) for w in ["write", "create"]): - return "file_write" - if any(has_word(w) for w in ["read", "view"]): - return "file_read" - if any(has_word(w) for w in ["task", "agent"]): - return "subagent" - return "other" - - -def extract_files(input_data: Any) -> List[str]: - if not input_data or not isinstance(input_data, dict): - return [] - out = set() - for key in ["file_path", "filepath", "path", "filePath", "file", "pattern"]: - v = input_data.get(key) - if isinstance(v, str) and 0 < len(v) < 512: - out.add(v) - return list(out) - - -def stringify_for_narrative(v: Any) -> str: - if v is None: - return "" - if isinstance(v, str): - return v - try: - return json.dumps(v) - except Exception: - return str(v) - - -def build_synthetic_compression(raw: Dict[str, Any]) -> Dict[str, Any]: - tool_name = raw.get("toolName") or raw.get("hookType") - input_str = stringify_for_narrative(raw.get("toolInput")) - output_str = stringify_for_narrative(raw.get("toolOutput")) - prompt_str = raw.get("userPrompt") or "" - - parts = [s for s in [prompt_str, input_str, output_str] if len(s) > 0] - narrative = " | ".join(parts) - if len(narrative) > 400: - narrative = narrative[:399] + "\u2026" - - title = tool_name or "observation" - if len(title) > 80: - title = title[:79] + "\u2026" - - subtitle = None - if input_str: - subtitle = input_str - if len(subtitle) > 120: - subtitle = subtitle[:119] + "\u2026" - - res = { - "id": raw["id"], - "sessionId": raw["sessionId"], - "timestamp": raw["timestamp"], - "type": infer_type(raw.get("toolName"), raw["hookType"]), - "title": title, - "subtitle": subtitle, - "facts": [], - "narrative": narrative, - "concepts": [], - "files": extract_files(raw.get("toolInput")), - "importance": 5, - "confidence": 0.3, - } - for k in ["modality", "imageData", "agentId"]: - if raw.get(k) is not None: - res[k] = raw[k] - return res - - -def observe(kv: StateKV, payload: Dict[str, Any]) -> Dict[str, Any]: - session_id = payload.get("sessionId") - hook_type = payload.get("hookType") - timestamp = payload.get("timestamp") - - if not session_id or not hook_type or not timestamp: - raise ValueError( - "Invalid payload: sessionId, hookType, and timestamp are required" - ) - - obs_id = generate_id("obs") - sanitized_data = payload.get("data") - try: - json_str = json.dumps(payload.get("data")) - sanitized = strip_private_data(json_str) - sanitized_data = json.loads(sanitized) - except Exception: - sanitized_data = strip_private_data(str(payload.get("data"))) - - raw = { - "id": obs_id, - "sessionId": session_id, - "timestamp": timestamp, - "hookType": hook_type, - "raw": sanitized_data, - } - - extracted_img = extract_image(sanitized_data) - if isinstance(sanitized_data, dict): - if hook_type in ("post_tool_use", "post_tool_failure"): - raw["toolName"] = sanitized_data.get("tool_name") - raw["toolInput"] = sanitized_data.get("tool_input") - raw["toolOutput"] = sanitized_data.get("tool_output") or sanitized_data.get( - "error" - ) - if hook_type == "prompt_submit": - raw["userPrompt"] = sanitized_data.get("prompt") - if extracted_img: - raw["modality"] = ( - "mixed" - if ( - raw.get("toolInput") - or raw.get("toolOutput") - or raw.get("userPrompt") - ) - else "image" - ) - elif isinstance(sanitized_data, str) and extracted_img: - raw["modality"] = "image" - - max_obs = int(os.getenv("MAX_OBS_PER_SESSION", "500")) - if max_obs > 0: - existing = kv.list(KV.observations(session_id)) - actual_obs_count = sum( - 1 for o in existing if not str(o.get("id", "")).endswith(":raw") - ) - if actual_obs_count >= max_obs: - raise ValueError(f"Session observation limit reached ({max_obs})") - - existing_session = kv.get(KV.sessions, session_id) - inherited_agent_id = ( - existing_session.get("agentId") if existing_session else get_agent_id() - ) - if inherited_agent_id: - raw["agentId"] = inherited_agent_id - - if extracted_img and ( - extracted_img.startswith("data:image/") - or extracted_img.startswith("iVBORw0KGgo") - or extracted_img.startswith("/9j/") - ): - try: - file_path, bytes_written = save_image_to_disk(extracted_img) - raw["imageData"] = file_path - - # Increment image ref count - img_refs = kv.get(KV.imageRefs, file_path) or 0 - kv.set(KV.imageRefs, file_path, img_refs + 1) - except Exception as ex: - print(f"[image store] failed: {ex}") - - # Set raw observation - raw["id"] = f"{obs_id}:raw" - kv.set(KV.observations(session_id), raw["id"], raw) - - # Stream raw observation - broadcast_stream( - { - "type": "raw_observation", - "sessionId": session_id, - "data": {"type": "raw", "observation": raw, "sessionId": session_id}, - } - ) - - if existing_session: - updates = [ - { - "type": "set", - "path": "updatedAt", - "value": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - }, - { - "type": "set", - "path": "observationCount", - "value": (existing_session.get("observationCount") or 0) + 1, - }, - ] - if not existing_session.get("firstPrompt") and isinstance( - raw.get("userPrompt"), str - ): - trimmed = " ".join(raw["userPrompt"].split()).strip() - if trimmed: - updates.append( - {"type": "set", "path": "firstPrompt", "value": trimmed[:200]} - ) - kv.update(KV.sessions, session_id, updates) - else: - project = payload.get("project") or "unknown" - auto_complete_old_active_sessions( - kv, session_id, project=project, agent_id=inherited_agent_id - ) - cwd = payload.get("cwd") or os.getcwd() - trimmed_prompt = None - if isinstance(raw.get("userPrompt"), str): - trimmed_prompt = " ".join(raw["userPrompt"].split()).strip()[:200] - ts = ( - datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z") - ) - new_sess = { - "id": session_id, - "project": project, - "cwd": cwd, - "startedAt": payload.get("timestamp") or ts, - "updatedAt": ts, - "status": "active", - "observationCount": 1, - } - if inherited_agent_id: - new_sess["agentId"] = inherited_agent_id - if trimmed_prompt: - new_sess["firstPrompt"] = trimmed_prompt - kv.set(KV.sessions, session_id, new_sess) - - # Perform synthetic compression (we default to synthetic) - raw_for_synthetic = dict(raw) - raw_for_synthetic["id"] = obs_id - synthetic = build_synthetic_compression(raw_for_synthetic) - for k in ["hookType", "raw", "toolName", "toolInput", "toolOutput", "userPrompt"]: - if k in raw_for_synthetic: - synthetic[k] = raw_for_synthetic[k] - kv.set(KV.observations(session_id), obs_id, synthetic) - _bm25_index.add(synthetic) - - comb_text = synthetic["title"] + " " + (synthetic.get("narrative") or "") - vector_index_add_guarded( - synthetic["id"], - synthetic["sessionId"], - comb_text, - {"kind": "synthetic", "logId": synthetic["id"]}, - ) - - if _index_persistence: - _index_persistence.schedule_save() - - # Stream compressed observation - broadcast_stream( - { - "type": "compressed_observation", - "sessionId": session_id, - "data": { - "type": "compressed", - "observation": synthetic, - "sessionId": session_id, - }, - } - ) - - # Commit to Dolt - commit_if_enabled( - kv, - f"Observe: {synthetic.get('title', 'observation')} in session {session_id[:8]}", - synthetic.get("agentId"), - ) - - return {"observationId": obs_id} - - -# ===================================================================== -# Folder-Based Observation Ingestion (folder_observe) -# ===================================================================== - - -def folder_observe(kv: StateKV, payload: Dict[str, Any]) -> Dict[str, Any]: - """Ingest a new observation scoped to a (folder_path, agent_id) pair. - - Required payload fields: - folderPath: str — absolute path of working directory - agentId: str — identity of the agent making the observation - text: str — human-readable observation content - timestamp: str — ISO 8601 UTC - - Optional payload fields: - type: str — observation type (default inferred or "other") - title: str — short title (auto-generated from text[:80] if absent) - concepts: list[str] — concept tags (default []) - files: list[str] — referenced file paths (default [] or extracted) - importance: int — 1-10 (default 5, clamped) - - Returns: {"observationId": str} - Raises: ValueError if required fields are missing or folder cap exceeded. - """ - # 1. Validate required fields (REQ-008) - folder_path_raw = payload.get("folderPath") - agent_id_raw = payload.get("agentId") - text_raw = payload.get("text") - timestamp = payload.get("timestamp") - - if not folder_path_raw: - raise ValueError("Invalid payload: folderPath is required") - if not agent_id_raw: - raise ValueError("Invalid payload: agentId is required") - if not text_raw: - raise ValueError("Invalid payload: text is required") - if not timestamp: - raise ValueError("Invalid payload: timestamp is required") - - # 2. Normalize folder_path and validate agent_id (REQ-002, REQ-063, REQ-064, REQ-066) - folder_path = normalize_folder_path(folder_path_raw) - agent_id = validate_agent_id(agent_id_raw) - - # 3. Strip private data and cap text (REQ-009, REQ-007) - safe_text = strip_private_data(text_raw) - safe_text = safe_text[:4000] - - # 3a. Deduplication check — compute fingerprint over normalized text (REQ-DEDUP) - _dedup_fp = hashlib.sha256( - safe_text[:4000].strip().lower().encode("utf-8") - ).hexdigest() - _dedup_lock = _get_dedup_lock(folder_path, agent_id) - _dedup_lock.acquire() - try: - _existing_dedup = kv.get(KV.obs_dedup(folder_path, agent_id), _dedup_fp) - if ( - _existing_dedup - and isinstance(_existing_dedup, dict) - and _existing_dedup.get("obsId") - ): - return {"observationId": _existing_dedup["obsId"], "deduplicated": True} - - # 10. Enforce MAX_OBS_PER_FOLDER cap before writing (REQ-015) - max_obs = int(os.getenv("MAX_OBS_PER_FOLDER", "2000")) - if max_obs > 0: - existing_obs = kv.list(KV.folder_obs(folder_path, agent_id)) - if len(existing_obs) >= max_obs: - raise ValueError(f"Folder observation limit reached ({max_obs})") - - # 4. Generate obs_id (REQ-010) - obs_id = generate_id("fobs") - - # 5. Determine optional fields - obs_type = payload.get("type") - if not obs_type: - obs_type = infer_type(None, "other") - - title = payload.get("title") - if not title: - title = safe_text[:80] - - concepts = payload.get("concepts") or [] - if not isinstance(concepts, list): - concepts = [] - - files = payload.get("files") - if not isinstance(files, list): - files = extract_files(payload) - - raw_importance = payload.get("importance") - if raw_importance is None: - importance = 5 - else: - try: - importance = max(1, min(10, int(raw_importance))) - except (TypeError, ValueError): - importance = 5 - - # 5. Build FolderObservation dict (REQ-003) - obs: Dict[str, Any] = { - "id": obs_id, - "folderPath": folder_path, - "agentId": agent_id, - "timestamp": timestamp, - "text": safe_text, - "type": obs_type, - "title": title, - "concepts": concepts, - "files": files, - "importance": importance, - } - if "forgetAfter" in payload: - obs["forgetAfter"] = payload["forgetAfter"] - elif ( - payload.get("ttlDays") - and isinstance(payload["ttlDays"], (int, float)) - and payload["ttlDays"] > 0 - ): - try: - import dateutil.parser - - ts_dt = dateutil.parser.parse(timestamp) - forget_time = ts_dt + datetime.timedelta(days=payload["ttlDays"]) - obs["forgetAfter"] = forget_time.isoformat().replace("+00:00", "Z") - except Exception: - pass - - # 6. Write observation to KV (REQ-001) - obs_scope = KV.folder_obs(folder_path, agent_id) - kv.set(obs_scope, obs_id, obs) - - # Write coordinate lookup mapping - kv.set( - KV.obs_lookup, - obs_id, - { - "folderPath": folder_path, - "agentId": agent_id, - }, - ) - - # Write dedup index entry — inside lock so check+write is atomic (REQ-DEDUP) - kv.set( - KV.obs_dedup(folder_path, agent_id), - _dedup_fp, - {"obsId": obs_id, "timestamp": timestamp}, - ) - - finally: - _dedup_lock.release() - - # 7. Upsert folder metadata (REQ-005) - meta_scope = KV.folder_meta(folder_path, agent_id) - meta = kv.get(meta_scope, "meta") or { - "folderPath": folder_path, - "agentId": agent_id, - "obsCount": 0, - "lastUpdated": timestamp, - "summary": None, - } - meta["obsCount"] = meta.get("obsCount", 0) + 1 - meta["lastUpdated"] = timestamp - kv.set(meta_scope, "meta", meta) - - # 8. Upsert global folders index entry (REQ-004, REQ-011) - index_key = f"{folder_path}:{agent_id}" - kv.set( - KV.folders, - index_key, - { - "folderPath": folder_path, - "agentId": agent_id, - "lastUpdated": meta["lastUpdated"], - "obsCount": meta["obsCount"], - }, - ) - - # 9. Add to BM25 index and vector index (REQ-012) - try: - _bm25_index.add(obs) - except Exception as ex: - print(f"[bm25] folder_observe add failed: {ex}") - - comb_text = title + " " + safe_text - vector_index_add_guarded( - obs_id, folder_path, comb_text, {"kind": "folder_obs", "logId": obs_id} - ) - - if _index_persistence: - _index_persistence.schedule_save() - - # 11. Write audit log entry (REQ-014) - kv.commit_version(f"folder_observe: {obs_id}", agent_id) - - # 12. Broadcast via WebSocket stream (REQ-013) - broadcast_stream( - { - "type": "folder_observation", - "folderPath": folder_path, - "agentId": agent_id, - "data": obs, - } - ) - - return {"observationId": obs_id} - - -# ===================================================================== -# Folder Deduplication (dedup_folder_observations) -# ===================================================================== - - -def dedup_folder_observations( - kv: StateKV, - folder_path_raw: Optional[str], - agent_id_raw: Optional[str], -) -> Dict[str, Any]: - """Remove duplicate observations from one or all (folder, agent) pairs. - - For each pair, groups observations by SHA-256 fingerprint of their normalized - text, keeps the earliest observation per group, and deletes the rest. - Also rebuilds the dedup index for each processed pair. - - Args: - folder_path_raw: folder path to deduplicate; None = all pairs. - agent_id_raw: agent ID to deduplicate; None = all pairs. - - Returns: - {"deduplicated": , "pairs_processed": , "kept": } - """ - # Determine which pairs to process - if folder_path_raw and agent_id_raw: - try: - fp = normalize_folder_path(folder_path_raw) - aid = validate_agent_id(agent_id_raw) - except ValueError as exc: - return {"success": False, "error": str(exc)} - pairs = [{"folderPath": fp, "agentId": aid}] - else: - pairs = [ - {"folderPath": e.get("folderPath", ""), "agentId": e.get("agentId", "")} - for e in kv.list(KV.folders) - if e.get("folderPath") and e.get("agentId") - ] - - total_removed = 0 - total_kept = 0 - - for pair in pairs: - fp = pair["folderPath"] - aid = pair["agentId"] - all_obs = kv.list(KV.folder_obs(fp, aid)) - - # Group by fingerprint, keeping earliest by timestamp - fingerprint_map: Dict[str, Dict[str, Any]] = {} - duplicates: List[str] = [] - - for obs in all_obs: - text = obs.get("text") or "" - fp_hash = hashlib.sha256( - text[:4000].strip().lower().encode("utf-8") - ).hexdigest() - if fp_hash not in fingerprint_map: - fingerprint_map[fp_hash] = obs - else: - # Keep the one with the earlier timestamp - existing_ts = fingerprint_map[fp_hash].get("timestamp", "") - this_ts = obs.get("timestamp", "") - if this_ts < existing_ts: - # This one is older — demote the previously-kept one - duplicates.append(fingerprint_map[fp_hash]["id"]) - fingerprint_map[fp_hash] = obs - else: - duplicates.append(obs["id"]) - - if duplicates: - forget(kv, {"folderPath": fp, "agentId": aid, "observationIds": duplicates}) - total_removed += len(duplicates) - - total_kept += len(fingerprint_map) - - # Rebuild the dedup index for this pair from scratch - dedup_scope = KV.obs_dedup(fp, aid) - # Clear existing dedup entries by re-writing from surviving observations - for fp_hash, obs in fingerprint_map.items(): - kv.set( - dedup_scope, - fp_hash, - {"obsId": obs["id"], "timestamp": obs.get("timestamp", "")}, - ) - - print( - f"[dedup] Processed {len(pairs)} pair(s): removed {total_removed}, kept {total_kept}" - ) - return { - "success": True, - "deduplicated": total_removed, - "pairs_processed": len(pairs), - "kept": total_kept, - } - - -# ===================================================================== -# Folder-Based Search (folder_search) -# ===================================================================== - - -def folder_search( - kv: StateKV, - query: str, - limit: int = 20, - folder_path: Optional[str] = None, - agent_id: Optional[str] = None, -) -> List[Dict[str, Any]]: - """Search across all folder observations (and global memories) using BM25 + vector hybrid search. - - Steps: - 1. Run hybrid search to obtain up to ``limit * 2`` candidate obs_ids with scores. - 2. Hydrate each candidate by looking up the observation in the KV store: - - Iterate ``KV.folders`` index to discover all (folder_path, agent_id) pairs. - - For each pair, load observations from ``KV.folder_obs`` and build an obs_id → obs map. - 3. Apply ``folder_path`` and ``agent_id`` post-filters to folder observations. - 4. Also include matching global memories from ``KV.memories``. - 5. Return results sorted by score descending, capped at ``limit``. - - Each result dict contains at minimum: ``folderPath``, ``agentId``, ``score``, - plus all fields from the underlying FolderObservation or Memory object. - - Requirements: REQ-016, REQ-017, REQ-018, REQ-019 - """ - if not query or not query.strip(): - return [] - - candidates = _hybrid_search.search(query, limit * 2) - - # --- Hydrate candidates from the search results (REQ-018, REQ-019) --- - results: List[Dict[str, Any]] = [] - seen_ids: set = set() - - for candidate in candidates: - obs_id = candidate.get("obsId") or candidate.get("id", "") - score = candidate.get("combinedScore") or candidate.get("score", 0.0) - - if not obs_id or obs_id in seen_ids: - continue - - # 1. Try folder observation first via O(1) coordinate lookup index - lookup = kv.get(KV.obs_lookup, obs_id) - if lookup and isinstance(lookup, dict): - fp = lookup.get("folderPath") - aid = lookup.get("agentId") - if fp and aid: - if folder_path is not None and fp != folder_path: - continue - if agent_id is not None and aid != agent_id: - continue - - obs = kv.get(KV.folder_obs(fp, aid), obs_id) - if obs and isinstance(obs, dict): - result = dict(obs) - result["score"] = score - result.setdefault("folderPath", fp) - result.setdefault("agentId", aid) - results.append(result) - seen_ids.add(obs_id) - continue - - # 2. Fallback scan for unindexed folder observations (e.g. from prior versions) - if obs_id.startswith("fobs_"): - found = False - for entry in kv.list(KV.folders): - fp = entry.get("folderPath", "") - aid = entry.get("agentId", "") - if not fp or not aid: - continue - if folder_path is not None and fp != folder_path: - continue - if agent_id is not None and aid != agent_id: - continue - obs = kv.get(KV.folder_obs(fp, aid), obs_id) - if obs and isinstance(obs, dict): - result = dict(obs) - result["score"] = score - result.setdefault("folderPath", fp) - result.setdefault("agentId", aid) - results.append(result) - seen_ids.add(obs_id) - # Lazy backfill the lookup index - kv.set(KV.obs_lookup, obs_id, {"folderPath": fp, "agentId": aid}) - found = True - break - if found: - continue - - # 3. Try global memory (REQ-018) - mem = kv.get(KV.memories, obs_id) - if mem and isinstance(mem, dict): - if mem.get("isLatest") is not False: - result = dict(mem) - result["score"] = score - result.setdefault("folderPath", "") - result.setdefault("agentId", mem.get("agentId") or "") - results.append(result) - seen_ids.add(obs_id) - continue - - # obs_id not found in either map — skip (stale index entry) - - # Sort by score descending and cap at limit (REQ-016) - results.sort(key=lambda r: r.get("score", 0.0), reverse=True) - return results[:limit] - - -def folder_timeline( - kv: StateKV, - limit: int = 100, - folder_path: Optional[str] = None, - agent_id: Optional[str] = None, - before: Optional[str] = None, - after: Optional[str] = None, -) -> List[Dict[str, Any]]: - """Return a folder activity feed — observations sorted by timestamp descending. - - Algorithm: - 1. List all (folder, agent) pairs from ``KV.folders``. - 2. Apply ``folder_path`` exact-match filter if provided. - 3. Apply ``agent_id`` exact-match filter if provided. - 4. For each remaining pair, load all observations from - ``KV.folder_obs(entry["folderPath"], entry["agentId"])``. - 5. Apply ``before`` ISO timestamp upper-bound filter: - exclude obs where ``obs["timestamp"] >= before``. - 6. Apply ``after`` ISO timestamp lower-bound filter: - exclude obs where ``obs["timestamp"] <= after``. - 7. Sort all collected observations by ``timestamp`` descending. - 8. Return the first ``limit`` entries. - - Postconditions (REQ-071): - - ``len(result) <= limit`` - - All results satisfy the provided filter conditions. - - Results are in non-increasing timestamp order. - - Requirements: REQ-020, REQ-021, REQ-022 - """ - # Step 1 — load the global folders index - index_entries = kv.list(KV.folders) - - # Step 2 — filter by folder_path exact match (REQ-021) - if folder_path is not None: - index_entries = [e for e in index_entries if e.get("folderPath") == folder_path] - - # Step 3 — filter by agent_id exact match (REQ-021) - if agent_id is not None: - index_entries = [e for e in index_entries if e.get("agentId") == agent_id] - - all_obs: List[Dict[str, Any]] = [] - - for entry in index_entries: - fp = entry.get("folderPath", "") - aid = entry.get("agentId", "") - if not fp or not aid: - continue - - # Step 4 — load observations for this pair - obs_scope = KV.folder_obs(fp, aid) - obs_list = kv.list(obs_scope) - - # Step 5 — apply before filter: exclude obs where timestamp >= before (REQ-021) - if before is not None: - obs_list = [o for o in obs_list if o.get("timestamp", "") < before] - - # Step 6 — apply after filter: exclude obs where timestamp <= after (REQ-021) - if after is not None: - obs_list = [o for o in obs_list if o.get("timestamp", "") > after] - - all_obs.extend(obs_list) - - # Step 7 — sort by timestamp descending (REQ-071) - all_obs.sort(key=lambda o: o.get("timestamp", ""), reverse=True) - - # Step 8 — return at most limit entries (REQ-022) - return all_obs[:limit] - - -# ===================================================================== -# Memory System (Remember, Forget, Evolve) -# ===================================================================== - - -def memory_to_observation(memory: Dict[str, Any]) -> Dict[str, Any]: - return { - "id": memory["id"], - "sessionId": memory.get("sessionIds", ["memory"])[0] - if memory.get("sessionIds") - else "memory", - "timestamp": memory["createdAt"], - "type": "decision", - "title": memory["title"], - "facts": [memory["content"]], - "narrative": memory["content"], - "concepts": memory.get("concepts", []), - "files": memory.get("files", []), - "importance": memory.get("strength", 7), - } - - -def remember(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - content = data.get("content") - if not content or not content.strip(): - raise ValueError("content is required") - content = strip_private_data(content) - - concepts = data.get("concepts") or [] - files = data.get("files") or [] - source_obs = data.get("sourceObservationIds") or [] - ttl_days = data.get("ttlDays") - mem_type = data.get("type") or "fact" - project = data.get("project") - if project: - project = project.strip() - - now = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - existing_memories = kv.list(KV.memories) - superseded_id = None - superseded_version = 1 - superseded_memory = None - lower_content = content.lower() - - for existing in existing_memories: - if existing.get("isLatest") is False: - continue - if project and existing.get("project") and existing["project"] != project: - continue - similarity = jaccard_similarity( - lower_content, existing.get("content", "").lower() - ) - if similarity > 0.7: - superseded_id = existing["id"] - superseded_version = existing.get("version") or 1 - superseded_memory = existing - break - - call_agent_id = data.get("agentId") or get_agent_id() - new_mem = { - "id": generate_id("mem"), - "createdAt": now, - "updatedAt": now, - "type": mem_type, - "title": content[:80], - "content": content, - "concepts": concepts, - "files": files, - "sessionIds": [], - "strength": 7, - "version": superseded_version + 1 if superseded_id else 1, - "parentId": superseded_id, - "supersedes": [superseded_id] if superseded_id else [], - "sourceObservationIds": [i for i in source_obs if i], - "isLatest": True, - } - if call_agent_id: - new_mem["agentId"] = call_agent_id - if project: - new_mem["project"] = project - - if ttl_days and isinstance(ttl_days, (int, float)) and ttl_days > 0: - forget_time = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta( - days=ttl_days - ) - new_mem["forgetAfter"] = forget_time.isoformat().replace("+00:00", "Z") - elif "forgetAfter" in data: - new_mem["forgetAfter"] = data["forgetAfter"] - - if superseded_memory: - superseded_memory["isLatest"] = False - kv.set(KV.memories, superseded_memory["id"], superseded_memory) - - kv.set(KV.memories, new_mem["id"], new_mem) - - try: - _bm25_index.add(memory_to_observation(new_mem)) - except Exception as ex: - print(f"[bm25] memory add failed: {ex}") - - comb_text = new_mem["title"] + " " + new_mem["content"] - vector_index_add_guarded( - new_mem["id"], "memory", comb_text, {"kind": "memory", "logId": new_mem["id"]} - ) - - if _index_persistence: - _index_persistence.schedule_save() - - # Commit to Dolt - commit_if_enabled( - kv, f"Remember: {new_mem.get('title', '')}", new_mem.get("agentId") - ) - - # Broadcast memory created - broadcast_stream( - { - "type": "memory_created", - "data": new_mem, - } - ) - - return {"success": True, "memory": new_mem} - - -def forget(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - """Delete a global memory, a folder (folder_path+agent_id), or specific observations. - - Dispatch rules (REQ-029, REQ-030, REQ-031, REQ-032, REQ-033): - 1. ``memoryId`` present → delete that global memory from KV.memories - 2. ``folderPath + agentId`` present, - no ``observationIds`` → delete ALL observations for that pair, - remove BM25 entries, delete folder_meta, - remove from KV.folders index - 3. ``folderPath + agentId + - observationIds`` present → delete only the listed observations, - decrement obsCount in metadata - - Legacy session-based paths are preserved for backward compatibility. - """ - memory_id = data.get("memoryId") - session_id = data.get("sessionId") - folder_path_raw = data.get("folderPath") - agent_id_raw = data.get("agentId") - obs_ids = data.get("observationIds") or [] - deleted = 0 - deleted_mem_ids = [] - deleted_obs_ids = [] - deleted_session = False - - # ------------------------------------------------------------------ - # Path 1: delete a global memory (REQ-029) - # ------------------------------------------------------------------ - if memory_id: - mem = kv.get(KV.memories, memory_id) - kv.delete(KV.memories, memory_id) - if mem and mem.get("imageRef"): - ref = mem["imageRef"] - refs = kv.get(KV.imageRefs, ref) or 0 - if refs > 0: - kv.set(KV.imageRefs, ref, refs - 1) - _bm25_index.remove(memory_id) - if _vector_index: - _vector_index.remove(memory_id) - deleted_mem_ids.append(memory_id) - deleted += 1 - # Broadcast memory deleted - broadcast_stream( - { - "type": "memory_deleted", - "memoryId": memory_id, - } - ) - - # ------------------------------------------------------------------ - # Path 2 & 3: folder-based deletion (REQ-030, REQ-031, REQ-032, REQ-033) - # ------------------------------------------------------------------ - if folder_path_raw and agent_id_raw: - try: - fp = normalize_folder_path(folder_path_raw) - aid = validate_agent_id(agent_id_raw) - except ValueError as exc: - return {"success": False, "error": str(exc), "deleted": 0} - - obs_scope = KV.folder_obs(fp, aid) - meta_scope = KV.folder_meta(fp, aid) - index_key = f"{fp}:{aid}" - - if obs_ids: - # ---------------------------------------------------------- - # Path 3: partial deletion — only the listed obs IDs (REQ-031) - # ---------------------------------------------------------- - partial_deleted = 0 - for oid in obs_ids: - obs = kv.get(obs_scope, oid) - existed = kv.delete(obs_scope, oid) - if existed: - kv.delete(KV.obs_lookup, oid) - _bm25_index.remove(oid) - if _vector_index: - _vector_index.remove(oid) - if obs and isinstance(obs, dict) and obs.get("text"): - fp_text = obs["text"][:4000] - dedup_fp = hashlib.sha256( - fp_text.strip().lower().encode("utf-8") - ).hexdigest() - kv.delete(KV.obs_dedup(fp, aid), dedup_fp) - deleted_obs_ids.append(oid) - partial_deleted += 1 - deleted += 1 - - # Decrement obsCount in metadata - if partial_deleted > 0: - meta = kv.get(meta_scope, "meta") - if meta and isinstance(meta, dict): - current_count = meta.get("obsCount", 0) - meta["obsCount"] = max(0, current_count - partial_deleted) - kv.set(meta_scope, "meta", meta) - # Also sync the folders index entry - index_entry = kv.get(KV.folders, index_key) - if index_entry and isinstance(index_entry, dict): - index_entry["obsCount"] = meta["obsCount"] - kv.set(KV.folders, index_key, index_entry) - - # Broadcast observations deleted - if deleted_obs_ids: - broadcast_stream( - { - "type": "observations_deleted", - "folderPath": fp, - "agentId": aid, - "observationIds": deleted_obs_ids, - } - ) - else: - # ---------------------------------------------------------- - # Path 2: full pair deletion (REQ-030, REQ-032) - # ---------------------------------------------------------- - all_obs = kv.list(obs_scope) - for obs in all_obs: - obs_id = obs.get("id") - if obs_id: - kv.delete(obs_scope, obs_id) - kv.delete(KV.obs_lookup, obs_id) - _bm25_index.remove(obs_id) - if _vector_index: - _vector_index.remove(obs_id) - deleted_obs_ids.append(obs_id) - deleted += 1 - - # Delete folder metadata entry - kv.delete(meta_scope, "meta") - - # Remove from global folders index - kv.delete(KV.folders, index_key) - - # Clear dedup entries - dedup_scope = KV.obs_dedup(fp, aid) - for item in kv.list(dedup_scope): - if isinstance(item, dict) and item.get("id"): - kv.delete(dedup_scope, item["id"]) - - # Broadcast folder pair deleted - broadcast_stream( - { - "type": "folder_deleted", - "folderPath": fp, - "agentId": aid, - } - ) - - # ------------------------------------------------------------------ - # Legacy: session-based deletion (unchanged) - # ------------------------------------------------------------------ - if session_id and obs_ids: - for oid in obs_ids: - base_oid = oid.replace(":raw", "") - obs = kv.get(KV.observations(session_id), base_oid) - raw_obs = kv.get(KV.observations(session_id), f"{base_oid}:raw") - - kv.delete(KV.observations(session_id), base_oid) - kv.delete(KV.observations(session_id), f"{base_oid}:raw") - kv.delete(KV.obs_lookup, base_oid) - - for o in (obs, raw_obs): - if o: - img = o.get("imageData") or o.get("imageRef") - if img: - refs = kv.get(KV.imageRefs, img) or 0 - if refs > 0: - kv.set(KV.imageRefs, img, refs - 1) - - _bm25_index.remove(base_oid) - _bm25_index.remove(f"{base_oid}:raw") - if _vector_index: - _vector_index.remove(base_oid) - _vector_index.remove(f"{base_oid}:raw") - deleted_obs_ids.append(oid) - deleted += 1 - - if session_id and not obs_ids and not memory_id and not folder_path_raw: - obs_list = kv.list(KV.observations(session_id)) - for obs in obs_list: - kv.delete(KV.observations(session_id), obs["id"]) - kv.delete(KV.obs_lookup, obs["id"]) - img = obs.get("imageData") or obs.get("imageRef") - if img: - refs = kv.get(KV.imageRefs, img) or 0 - if refs > 0: - kv.set(KV.imageRefs, img, refs - 1) - _bm25_index.remove(obs["id"]) - if _vector_index: - _vector_index.remove(obs["id"]) - deleted_obs_ids.append(obs["id"]) - deleted += 1 - kv.delete(KV.sessions, session_id) - kv.delete(KV.summaries, session_id) - deleted_session = True - deleted += 2 - - if deleted > 0: - if _index_persistence: - _index_persistence.schedule_save() - safe_audit( - kv, - "forget", - "mem::forget", - deleted_mem_ids + deleted_obs_ids, - { - "memoryId": memory_id, - "sessionId": session_id, - "folderPath": folder_path_raw, - "agentId": agent_id_raw, - "deleted": deleted, - "memoriesDeleted": len(deleted_mem_ids), - "observationsDeleted": len(deleted_obs_ids), - "sessionDeleted": deleted_session, - "reason": "user-initiated forget", - }, - ) - - agent_id = data.get("agentId") or get_agent_id() - commit_if_enabled( - kv, f"Forget: memory_id={memory_id} folder_path={folder_path_raw}", agent_id - ) - - return {"success": True, "deleted": deleted} - - -# ===================================================================== -# Prompt Context Compilation System -# ===================================================================== - - -def estimate_tokens(text: str) -> int: - return int(len(text) / 3) - - -def escape_xml_attr(s: str) -> str: - return ( - s.replace("&", "&") - .replace('"', """) - .replace("<", "<") - .replace(">", ">") - ) - - -def context(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - session_id = data.get("sessionId") - project = data.get("project") - budget = data.get("budget") or int(os.getenv("TOKEN_BUDGET", "2000")) - - if not session_id or not project: - raise ValueError("sessionId and project are required") - - blocks = [] - - # 1. Pinned Slots - pinned_slots = list_pinned_slots(kv, project) - slot_content = render_pinned_context(pinned_slots) - if slot_content: - blocks.append( - { - "type": "memory", - "content": slot_content, - "tokens": estimate_tokens(slot_content), - "recency": int(time.time() * 1000), - } - ) - - # 2. Profile - profile = kv.get(KV.profiles, project) - if profile: - profile_parts = [] - if profile.get("topConcepts"): - profile_parts.append( - "Concepts: " - + ", ".join([c["concept"] for c in profile["topConcepts"][:8]]) - ) - if profile.get("topFiles"): - profile_parts.append( - "Key files: " + ", ".join([f["file"] for f in profile["topFiles"][:5]]) - ) - if profile.get("conventions"): - profile_parts.append("Conventions: " + "; ".join(profile["conventions"])) - if profile.get("commonErrors"): - profile_parts.append( - "Common errors: " + "; ".join(profile["commonErrors"][:3]) - ) - - if profile_parts: - profile_content = "## Project Profile\n" + "\n".join(profile_parts) - blocks.append( - { - "type": "memory", - "content": profile_content, - "tokens": estimate_tokens(profile_content), - "recency": int(time.time() * 1000), - } - ) - - # 3. Lessons - lessons = kv.list(KV.lessons) - relevant_lessons = [ - les - for les in lessons - if not les.get("deleted") - and (not les.get("project") or les["project"] == project) - ] - - # Score lessons - def lesson_score(les): - factor = 1.5 if les.get("project") == project else 1.0 - return factor * les.get("confidence", 0.5) - - relevant_lessons.sort(key=lesson_score, reverse=True) - relevant_lessons = relevant_lessons[:10] - - if relevant_lessons: - items = [] - for les in relevant_lessons: - desc = f"- ({les['confidence']:.2f}) {les['content']}" - if les.get("context"): - desc += f" — {les['context']}" - items.append(desc) - lessons_content = "## Lessons Learned\n" + "\n".join(items) - blocks.append( - { - "type": "memory", - "content": lessons_content, - "tokens": estimate_tokens(lessons_content), - "recency": int(time.time() * 1000), - } - ) - - # 4. Sessions & Summaries - all_sessions = kv.list(KV.sessions) - sessions = [ - s for s in all_sessions if s.get("project") == project and s["id"] != session_id - ] - sessions.sort(key=lambda s: s.get("startedAt", ""), reverse=True) - sessions = sessions[:10] - - for s in sessions: - summary = kv.get(KV.summaries, s["id"]) - if summary: - content = ( - f"## {summary.get('title', 'Session summary')}\n{summary.get('narrative', '')}\n" - f"Decisions: {'; '.join(summary.get('keyDecisions', []))}\n" - f"Files: {', '.join(summary.get('filesModified', []))}" - ) - blocks.append( - { - "type": "summary", - "content": content, - "tokens": estimate_tokens(content), - "recency": int(time.time() * 1000), - } - ) - else: - # Fallback to important observations - obs_list = kv.list(KV.observations(s["id"])) - important = [ - o for o in obs_list if o.get("title") and o.get("importance", 0) >= 5 - ] - if important: - important.sort(key=lambda o: o.get("importance", 0), reverse=True) - top = important[:5] - items = [ - f"- [{o.get('type')}] {o.get('title')}: {o.get('narrative')}" - for o in top - ] - content = ( - f"## Session {s['id'][:8]} ({s.get('startedAt')})\n" - + "\n".join(items) - ) - blocks.append( - { - "type": "observation", - "content": content, - "tokens": estimate_tokens(content), - "recency": int(time.time() * 1000), - } - ) - - blocks.sort(key=lambda b: b.get("recency", 0), reverse=True) - - header = f'' - footer = "" - used_tokens = estimate_tokens(header) + estimate_tokens(footer) - - selected = [] - for b in blocks: - if used_tokens + b["tokens"] > budget: - continue - selected.append(b["content"]) - used_tokens += b["tokens"] - - if not selected: - return {"context": "", "blocks": 0, "tokens": 0} - - res_context = f"{header}\n" + "\n\n".join(selected) + f"\n{footer}" - return {"context": res_context, "blocks": len(selected), "tokens": used_tokens} - - -# ===================================================================== -# Memory Slots System -# ===================================================================== - -DEFAULT_SLOTS = [ - { - "label": "persona", - "content": "", - "sizeLimit": 1000, - "description": "How the agent should see itself: role, tone, behavioural guidelines.", - "pinned": True, - "readOnly": False, - "scope": "global", - }, - { - "label": "user_preferences", - "content": "", - "sizeLimit": 2000, - "description": "Coding style, tool preferences, naming conventions, and other habits the user wants preserved across sessions.", - "pinned": True, - "readOnly": False, - "scope": "global", - }, - { - "label": "tool_guidelines", - "content": "", - "sizeLimit": 1500, - "description": "Rules the agent should follow when picking or sequencing tools (e.g. prefer X over Y, never run Z without confirmation).", - "pinned": True, - "readOnly": False, - "scope": "global", - }, - { - "label": "project_context", - "content": "", - "sizeLimit": 3000, - "description": "Architecture decisions, codebase conventions, build/test commands, and cross-cutting constraints for the current project.", - "pinned": True, - "readOnly": False, - "scope": "project", - }, - { - "label": "guidance", - "content": "", - "sizeLimit": 1500, - "description": "Active advice for the next session: what to focus on, what to avoid, open risks.", - "pinned": True, - "readOnly": False, - "scope": "project", - }, - { - "label": "pending_items", - "content": "", - "sizeLimit": 2000, - "description": "Unfinished work, explicit TODOs, and promises made but not yet delivered.", - "pinned": True, - "readOnly": False, - "scope": "project", - }, - { - "label": "session_patterns", - "content": "", - "sizeLimit": 1500, - "description": "Recurring behaviours and common struggles observed across recent sessions.", - "pinned": False, - "readOnly": False, - "scope": "project", - }, - { - "label": "self_notes", - "content": "", - "sizeLimit": 1500, - "description": "Free-form notes the agent keeps for itself: hypotheses, dead ends, things to revisit.", - "pinned": False, - "readOnly": False, - "scope": "project", - }, -] - - -def seed_defaults(kv: StateKV) -> None: - now = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - for tmpl in DEFAULT_SLOTS: - scope = tmpl["scope"] - target = KV.globalSlots if scope == "global" else KV.slots - existing = kv.get(target, tmpl["label"]) - if existing: - continue - slot = dict(tmpl) - slot["createdAt"] = now - slot["updatedAt"] = now - kv.set(target, tmpl["label"], slot) - - -def list_pinned_slots( - kv: StateKV, project: Optional[str] = None -) -> List[Dict[str, Any]]: - p_slots = kv.list(project_slots_scope(kv, project)) - g_slots = kv.list(KV.globalSlots) - merged = {} - for s in g_slots: - merged[s["label"]] = s - for s in p_slots: - merged[s["label"]] = s - pinned = [ - s for s in merged.values() if s.get("pinned") and s.get("content", "").strip() - ] - pinned.sort(key=lambda s: s["label"]) - return pinned - - -def render_pinned_context(slots: List[Dict[str, Any]]) -> str: - if not slots: - return "" - lines = ["# agentcache pinned slots", ""] - for s in slots: - lines.append(f"## {s['label']}") - lines.append(s["content"].strip()) - lines.append("") - return "\n".join(lines) - - -def slot_list(kv: StateKV, project: Optional[str] = None) -> Dict[str, Any]: - p_slots = kv.list(project_slots_scope(kv, project)) - g_slots = kv.list(KV.globalSlots) - merged = {} - for s in g_slots: - merged[s["label"]] = s - for s in p_slots: - merged[s["label"]] = s - slots = sorted(list(merged.values()), key=lambda s: s["label"]) - return {"success": True, "slots": slots} - - -def slot_get(kv: StateKV, label: str, project: Optional[str] = None) -> Dict[str, Any]: - p_scope = project_slots_scope(kv, project) - project_s = kv.get(p_scope, label) - if project_s: - return {"success": True, "slot": project_s, "scope": "project"} - global_s = kv.get(KV.globalSlots, label) - if global_s: - return {"success": True, "slot": global_s, "scope": "global"} - return {"success": False, "error": "slot not found"} - - -def slot_create(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - label = data.get("label") - if not label or not re.match(r"^[a-z][a-z0-9_]*$", label): - return { - "success": False, - "error": "label required (lowercase, starts with letter, [a-z0-9_])", - } - - scope = data.get("scope") or "project" - if scope not in ("project", "global"): - return {"success": False, "error": "scope must be 'project' or 'global'"} - - limit = data.get("sizeLimit") or 2000 - if not isinstance(limit, int) or limit < 1 or limit > 20000: - return { - "success": False, - "error": "sizeLimit must be an integer between 1 and 20000", - } - - content = strip_private_data(data.get("content") or "") - if len(content) > limit: - return { - "success": False, - "error": f"content exceeds sizeLimit ({len(content)} > {limit})", - } - - description = data.get("description") or "" - pinned = data.get("pinned", True) - project = data.get("project") - - target_kv = ( - KV.globalSlots if scope == "global" else project_slots_scope(kv, project) - ) - existing = kv.get(target_kv, label) - if existing: - return {"success": False, "error": f"slot already exists in {scope} scope"} - - now = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - slot = { - "label": label, - "content": content, - "sizeLimit": limit, - "description": description, - "pinned": pinned, - "readOnly": False, - "scope": scope, - "createdAt": now, - "updatedAt": now, - } - kv.set(target_kv, label, slot) - safe_audit( - kv, - "slot_create", - "mem::slot-create", - [label], - {"scope": scope, "sizeLimit": limit, "pinned": pinned}, - ) - - # Commit to Dolt - agent_id = data.get("agentId") or get_agent_id() - commit_if_enabled(kv, f"Create slot: {label}", agent_id) - - return {"success": True, "slot": slot} - - -def slot_append( - kv: StateKV, - label: str, - text: str, - agent_id: Optional[str] = None, - project: Optional[str] = None, -) -> Dict[str, Any]: - res = slot_get(kv, label, project) - if not res.get("success"): - return {"success": False, "error": "slot not found"} - - slot = res["slot"] - scope = res["scope"] - target_kv = ( - KV.globalSlots if scope == "global" else project_slots_scope(kv, project) - ) - - if slot.get("readOnly"): - return {"success": False, "error": "slot is read-only"} - - content = slot.get("content") or "" - sep = "\n" if content and not content.endswith("\n") else "" - next_content = content + sep + strip_private_data(text) - - limit = slot.get("sizeLimit") or 2000 - if len(next_content) > limit: - return { - "success": False, - "error": f"append would exceed sizeLimit ({len(next_content)} > {limit})", - "currentSize": len(content), - "sizeLimit": limit, - } - - slot["content"] = next_content - slot["updatedAt"] = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - kv.set(target_kv, label, slot) - - safe_audit( - kv, - "slot_append", - "mem::slot-append", - [label], - {"scope": scope, "added": len(text), "total": len(next_content)}, - ) - - # Commit to Dolt - commit_if_enabled(kv, f"Append slot: {label}", agent_id or get_agent_id()) - - return {"success": True, "slot": slot, "size": len(next_content)} - - -def slot_replace( - kv: StateKV, - label: str, - content: str, - agent_id: Optional[str] = None, - project: Optional[str] = None, -) -> Dict[str, Any]: - res = slot_get(kv, label, project) - if not res.get("success"): - return {"success": False, "error": "slot not found"} - - slot = res["slot"] - scope = res["scope"] - target_kv = ( - KV.globalSlots if scope == "global" else project_slots_scope(kv, project) - ) - - if slot.get("readOnly"): - return {"success": False, "error": "slot is read-only"} - - content = strip_private_data(content) - limit = slot.get("sizeLimit") or 2000 - if len(content) > limit: - return { - "success": False, - "error": f"content exceeds sizeLimit ({len(content)} > {limit})", - "sizeLimit": limit, - } - - before_len = len(slot.get("content") or "") - slot["content"] = content - slot["updatedAt"] = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - kv.set(target_kv, label, slot) - - safe_audit( - kv, - "slot_replace", - "mem::slot-replace", - [label], - {"scope": scope, "before": before_len, "after": len(content)}, - ) - - # Commit to Dolt - commit_if_enabled(kv, f"Replace slot: {label}", agent_id or get_agent_id()) - - return {"success": True, "slot": slot, "size": len(content)} - - -def slot_delete( - kv: StateKV, - label: str, - agent_id: Optional[str] = None, - project: Optional[str] = None, -) -> Dict[str, Any]: - res = slot_get(kv, label, project) - if not res.get("success"): - return {"success": False, "error": "slot not found"} - - slot = res["slot"] - scope = res["scope"] - target_kv = ( - KV.globalSlots if scope == "global" else project_slots_scope(kv, project) - ) - - if slot.get("readOnly"): - return {"success": False, "error": "slot is read-only"} - - kv.delete(target_kv, label) - safe_audit( - kv, - "slot_delete", - "mem::slot-delete", - [label], - {"scope": scope, "size": len(slot.get("content") or "")}, - ) - - # Commit to Dolt - commit_if_enabled(kv, f"Delete slot: {label}", agent_id or get_agent_id()) - - return {"success": True} - - -def slot_reflect(kv: StateKV, session_id: str, max_obs: int = 50) -> Dict[str, Any]: - session = kv.get(KV.sessions, session_id) - project = session.get("project") if session else None - - observations = kv.list(KV.observations(session_id)) - if not observations: - return {"success": True, "applied": 0, "reason": "no observations for session"} - - recent = sorted(observations, key=lambda x: x.get("timestamp", ""), reverse=True)[ - :max_obs - ] - - pending_lines = [] - pattern_counts = {} - files = set() - - for obs in recent: - title = (obs.get("title") or "").lower() - narrative = (obs.get("narrative") or "").lower() - if "todo" in narrative or "todo" in title: - pending_lines.append(f"- {obs.get('title') or obs['id']}") - if obs.get("type") == "error": - pattern_counts["errors"] = pattern_counts.get("errors", 0) + 1 - if obs.get("type") == "command_run": - pattern_counts["commands"] = pattern_counts.get("commands", 0) + 1 - for f in obs.get("files") or []: - files.add(f) - - applied = 0 - now = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - - if pending_lines: - res = slot_get(kv, "pending_items", project) - if res.get("success"): - slot = res["slot"] - scope = res["scope"] - target_kv = ( - KV.globalSlots - if scope == "global" - else project_slots_scope(kv, project) - ) - already = set((slot.get("content") or "").split("\n")) - fresh = [line for line in pending_lines if line not in already] - if fresh: - sep = ( - "\n" - if slot.get("content") and not slot["content"].endswith("\n") - else "" - ) - next_content = (slot.get("content") or "") + sep + "\n".join(fresh) - limit = slot.get("sizeLimit") or 2000 - if len(next_content) > limit: - next_content = next_content[-limit:] - slot["content"] = next_content - slot["updatedAt"] = now - kv.set(target_kv, "pending_items", slot) - applied += 1 - - if pattern_counts: - res = slot_get(kv, "session_patterns", project) - if res.get("success"): - slot = res["slot"] - scope = res["scope"] - target_kv = ( - KV.globalSlots - if scope == "global" - else project_slots_scope(kv, project) - ) - summary = [f"last reflection: {now}"] - for k, v in pattern_counts.items(): - summary.append(f"- {k}: {v} in last {len(recent)} observations") - next_content = "\n".join(summary) - limit = slot.get("sizeLimit") or 2000 - if len(next_content) > limit: - next_content = next_content[:limit] - slot["content"] = next_content - slot["updatedAt"] = now - kv.set(target_kv, "session_patterns", slot) - applied += 1 - - if files: - res = slot_get(kv, "project_context", project) - if res.get("success"): - slot = res["slot"] - scope = res["scope"] - target_kv = ( - KV.globalSlots - if scope == "global" - else project_slots_scope(kv, project) - ) - already = slot.get("content") or "" - fresh = [f for f in files if f not in already][:20] - if fresh: - header_line = "Files touched in recent sessions:" if not already else "" - sep = "\n" if already and not already.endswith("\n") else "" - lines = [already] - if header_line: - lines.append(header_line) - for f in fresh: - lines.append(f"- {f}") - next_content = sep.join([line for line in lines if line]) - limit = slot.get("sizeLimit") or 2000 - if len(next_content) > limit: - next_content = next_content[-limit:] - slot["content"] = next_content - slot["updatedAt"] = now - kv.set(target_kv, "project_context", slot) - applied += 1 - - if applied > 0: - safe_audit( - kv, - "slot_reflect", - "mem::slot-reflect", - [session_id], - {"observationCount": len(recent), "slotsUpdated": applied}, - ) - commit_if_enabled( - kv, - f"Slot reflect: updated {applied} slots in session {session_id[:8]}", - "system", - ) - - return {"success": True, "applied": applied, "observationsReviewed": len(recent)} - - -# ===================================================================== -# Lessons Learned System -# ===================================================================== - - -def reinforce_lesson(lesson: Dict[str, Any]) -> None: - now = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - lesson["reinforcements"] = lesson.get("reinforcements", 0) + 1 - conf = lesson.get("confidence", 0.5) - lesson["confidence"] = min(1.0, conf + 0.1 * (1 - conf)) - lesson["lastReinforcedAt"] = now - lesson["updatedAt"] = now - - -def lesson_save(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - content = data.get("content") - if not content or not content.strip(): - return {"success": False, "error": "content is required"} - content = strip_private_data(content) - context_str = strip_private_data(data.get("context") or "") - - agent_id = data.get("agentId") or get_agent_id() - fp = fingerprint_id("lsn", content) - existing = kv.get(KV.lessons, fp) - - if existing and not existing.get("deleted"): - reinforce_lesson(existing) - if context_str and not existing.get("context"): - existing["context"] = context_str - kv.set(KV.lessons, existing["id"], existing) - safe_audit(kv, "lesson_strengthen", "mem::lesson-save", [existing["id"]]) - - # Commit to Dolt - commit_if_enabled( - kv, f"Strengthen lesson: {existing.get('content', '')[:60]}", agent_id - ) - - return {"success": True, "action": "strengthened", "lesson": existing} - - confidence = data.get("confidence") - if not isinstance(confidence, (int, float)) or confidence < 0 or confidence > 1: - confidence = 0.5 - - now = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - lesson = { - "id": fp, - "content": content.strip(), - "context": context_str.strip(), - "confidence": confidence, - "reinforcements": 0, - "source": data.get("source") or "manual", - "sourceIds": data.get("sourceIds") or [], - "project": data.get("project"), - "tags": data.get("tags") or [], - "createdAt": now, - "updatedAt": now, - "decayRate": 0.05, - } - kv.set(KV.lessons, lesson["id"], lesson) - safe_audit(kv, "lesson_save", "mem::lesson-save", [lesson["id"]]) - - # Commit to Dolt - commit_if_enabled(kv, f"Create lesson: {lesson['content'][:60]}", agent_id) - - return {"success": True, "action": "created", "lesson": lesson} - - -def lesson_list(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - limit = data.get("limit") or 50 - min_confidence = data.get("minConfidence") or 0.0 - all_lessons = kv.list(KV.lessons) - - lessons = [ - les - for les in all_lessons - if not les.get("deleted") and les.get("confidence", 0.5) >= min_confidence - ] - - project = data.get("project") - if project: - lessons = [les for les in lessons if les.get("project") == project] - source = data.get("source") - if source: - lessons = [les for les in lessons if les.get("source") == source] - - lessons.sort(key=lambda x: x.get("confidence", 0.5), reverse=True) - return {"success": True, "lessons": lessons[:limit]} - - -def lesson_recall(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - query = data.get("query") - if not query or not query.strip(): - return {"success": False, "error": "query is required"} - - query_lower = query.lower() - min_confidence = data.get("minConfidence") or 0.1 - limit = data.get("limit") or 10 - - all_lessons = kv.list(KV.lessons) - lessons = [ - les - for les in all_lessons - if not les.get("deleted") and les.get("confidence", 0.5) >= min_confidence - ] - - project = data.get("project") - if project: - lessons = [les for les in lessons if les.get("project") == project] - - scored = [] - terms = [t for t in query_lower.split() if len(t) > 1] - - for les in lessons: - text = f"{les.get('content', '')} {les.get('context', '')} {' '.join(les.get('tags') or [])}".lower() - match_count = sum(1 for t in terms if t in text) - if match_count == 0: - continue - - relevance = match_count / len(terms) - baseline = les.get("lastReinforcedAt") or les.get("createdAt") - import dateutil.parser - - dt = dateutil.parser.parse(baseline) - days = ( - datetime.datetime.now(datetime.timezone.utc) - - dt.replace(tzinfo=datetime.timezone.utc) - ).total_seconds() / (3600 * 24) - recency_boost = 1 / (1 + days * 0.01) - score = les.get("confidence", 0.5) * relevance * recency_boost - scored.append({"lesson": les, "score": score}) - - scored.sort(key=lambda x: x["score"], reverse=True) - results = [] - for s in scored[:limit]: - item = dict(s["lesson"]) - item["score"] = round(s["score"], 3) - results.append(item) - - safe_audit( - kv, - "lesson_recall", - "mem::lesson-recall", - [], - {"query": query, "resultCount": len(results)}, - ) - return {"success": True, "lessons": results} - - -def lesson_strengthen(kv: StateKV, lesson_id: str) -> Dict[str, Any]: - lesson = kv.get(KV.lessons, lesson_id) - if not lesson or lesson.get("deleted"): - return {"success": False, "error": "lesson not found"} - - reinforce_lesson(lesson) - kv.set(KV.lessons, lesson["id"], lesson) - safe_audit(kv, "lesson_strengthen", "mem::lesson-strengthen", [lesson["id"]]) - - # Commit to Dolt - commit_if_enabled( - kv, f"Strengthen lesson: {lesson.get('content', '')[:60]}", get_agent_id() - ) - - return {"success": True, "lesson": lesson} - - -def lesson_decay_sweep(kv: StateKV) -> Dict[str, Any]: - all_lessons = kv.list(KV.lessons) - decayed = 0 - soft_deleted = 0 - now = datetime.datetime.now(datetime.timezone.utc) - timestamp = now.isoformat().replace("+00:00", "Z") - - for les in all_lessons: - if les.get("deleted"): - continue - baseline_str = ( - les.get("lastDecayedAt") or les.get("lastReinforcedAt") or les["createdAt"] - ) - import dateutil.parser - - dt = dateutil.parser.parse(baseline_str) - weeks = (now - dt.replace(tzinfo=datetime.timezone.utc)).total_seconds() / ( - 3600 * 24 * 7 - ) - if weeks < 1.0: - continue - - decay = les.get("decayRate", 0.05) * weeks - new_conf = max(0.05, les.get("confidence", 0.5) - decay) - - if new_conf != les.get("confidence"): - before = les.get("confidence", 0.5) - les["confidence"] = round(new_conf, 3) - les["lastDecayedAt"] = timestamp - les["updatedAt"] = timestamp - - if les["confidence"] <= 0.1 and les.get("reinforcements", 0) == 0: - les["deleted"] = True - soft_deleted += 1 - else: - decayed += 1 - - kv.set(KV.lessons, les["id"], les) - safe_audit( - kv, - "lesson_strengthen", - "mem::lesson-decay-sweep", - [les["id"]], - { - "action": "soft-delete" if les.get("deleted") else "decay", - "actor": "system", - "reason": "decay-sweep", - "before": {"confidence": before, "deleted": False}, - "after": { - "confidence": les["confidence"], - "deleted": bool(les.get("deleted")), - }, - }, - ) - - if decayed > 0 or soft_deleted > 0: - commit_if_enabled( - kv, - f"Lesson decay sweep: decayed {decayed}, soft-deleted {soft_deleted}", - "system", - ) - - return { - "success": True, - "decayed": decayed, - "softDeleted": soft_deleted, - "total": len(all_lessons), - } - - -# ===================================================================== -# Database Rebuilder (Index Bootstrapper) -# ===================================================================== - - -def rebuild_index(kv: StateKV) -> int: - _bm25_index.clear() - if _vector_index: - _vector_index.clear() - - total_indexed = 0 - - # ---- Path A: folder-based observations (new schema) ---- - folder_pairs = kv.list(KV.folders) - for entry in folder_pairs: - fp = entry.get("folderPath") - aid = entry.get("agentId") - if not fp or not aid: - continue - obs_list = kv.list(KV.folder_obs(fp, aid)) - for obs in obs_list: - if not obs.get("id"): - continue - # Populate coordinate lookup index - kv.set(KV.obs_lookup, obs["id"], {"folderPath": fp, "agentId": aid}) - - _bm25_index.add(obs) - comb_text = (obs.get("title") or "") + " " + (obs.get("text") or "") - vector_index_add_guarded( - obs["id"], - fp, - comb_text.strip(), - {"kind": "folder_observation", "logId": obs["id"]}, - ) - total_indexed += 1 - - # ---- Path B: session-based observations (legacy schema — kept for old data) ---- - try: - sessions = kv.list(KV.sessions) - for sess in sessions: - sid = sess.get("id") - if not sid: - continue - obs_list = kv.list(KV.observations(sid)) - for obs in obs_list: - # Only index compressed (non-raw) observations - if obs.get("title") and obs.get("narrative"): - # Skip if already indexed via folder path (same obs id) - if _bm25_index.has(obs["id"]): - continue - _bm25_index.add(obs) - comb_text = obs["title"] + " " + obs["narrative"] - vector_index_add_guarded( - obs["id"], - sid, - comb_text, - {"kind": "observation", "logId": obs["id"]}, - ) - total_indexed += 1 - except Exception as e: - print(f"[rebuild_index] session-based backfill skipped: {e}") - - # ---- Backfill BM25 with global memories (both schemas) ---- - memories = kv.list(KV.memories) - for mem in memories: - if mem.get("isLatest") is False: - continue - if not mem.get("title") or not mem.get("content"): - continue - converted = memory_to_observation(mem) - _bm25_index.add(converted) - comb_text = mem["title"] + " " + mem["content"] - vector_index_add_guarded( - mem["id"], "memory", comb_text, {"kind": "memory", "logId": mem["id"]} - ) - total_indexed += 1 - - if _index_persistence and total_indexed > 0: - _index_persistence.schedule_save() - - return total_indexed - - -# ===================================================================== -# Advanced Function Stubs / CRUD Operations -# ===================================================================== - - -def list_sessions(kv: StateKV) -> List[Dict[str, Any]]: - sessions = kv.list(KV.sessions) - for s in sessions: - sid = s.get("id") - if sid: - summary = kv.get(KV.summaries, sid) - if summary: - s["title"] = summary.get("title") - s["summary"] = summary.get("narrative") - sessions.sort(key=lambda s: s.get("startedAt", ""), reverse=True) - return sessions - - -def get_session(kv: StateKV, session_id: str) -> Optional[Dict[str, Any]]: - s = kv.get(KV.sessions, session_id) - if s: - summary = kv.get(KV.summaries, session_id) - if summary: - s["title"] = summary.get("title") - s["summary"] = summary.get("narrative") - return s - - -def create_session(kv: StateKV, session: Dict[str, Any]) -> Dict[str, Any]: - auto_complete_old_active_sessions( - kv, - session["id"], - project=session.get("project"), - agent_id=session.get("agentId"), - ) - kv.set(KV.sessions, session["id"], session) - return session - - -def end_session(kv: StateKV, session_id: str) -> bool: - now = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - kv.update( - KV.sessions, - session_id, - [ - {"type": "set", "path": "endedAt", "value": now}, - {"type": "set", "path": "status", "value": "completed"}, - ], - ) - return True - - -def timeline(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - # Simple timeline query returning observations sorted by timestamp - anchor = data.get("anchor") - project = data.get("project") - session_id = data.get("sessionId") - before = data.get("before") or 10 - after = data.get("after") or 10 - - sessions = kv.list(KV.sessions) - if session_id: - sessions = [s for s in sessions if s.get("id") == session_id] - elif project: - sessions = [s for s in sessions if s.get("project") == project] - - all_obs = [] - for s in sessions: - all_obs.extend(kv.list(KV.observations(s["id"]))) - - # sort by timestamp - all_obs.sort(key=lambda x: x.get("timestamp", "")) - - anchor_idx = -1 - for idx, obs in enumerate(all_obs): - if obs["id"] == anchor or obs.get("timestamp", "") >= (anchor or ""): - anchor_idx = idx - break - - if anchor_idx == -1: - anchor_idx = len(all_obs) // 2 - - start = max(0, anchor_idx - before) - end = min(len(all_obs), anchor_idx + after + 1) - - return { - "success": True, - "observations": all_obs[start:end], - "anchorIndex": anchor_idx - start, - } - - -def get_project_profile(kv: StateKV, project: str) -> Dict[str, Any]: - prof = kv.get(KV.profiles, project) - if not prof: - prof = { - "project": project, - "topConcepts": [], - "topFiles": [], - "conventions": [], - "commonErrors": [], - "updatedAt": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - } - if not prof.get("topConcepts") and not prof.get("topFiles"): - prof = build_project_profile(kv, project) - return prof - - -def build_project_profile(kv: StateKV, project: str) -> Dict[str, Any]: - prof = kv.get(KV.profiles, project) - if not prof: - prof = { - "project": project, - "topConcepts": [], - "topFiles": [], - "conventions": [], - "commonErrors": [], - "updatedAt": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - } - - # Stored profile may lack topConcepts/topFiles — compute from observations + memories if empty - if not prof.get("topConcepts") and not prof.get("topFiles"): - import json as _j - import os.path as _osp - import re as _re - from collections import Counter - - sessions = kv.list(KV.sessions) - project_sessions = [s for s in sessions if s.get("project") == project] - concept_counts = Counter() - file_counts = Counter() - - def _harvest_file(path, fc, cc): - if not isinstance(path, str) or not path: - return - fc[path] += 1 - parts = _re.split(r"[\\/]", path) - fname = parts[-1] if parts else "" - skip = {"tmp", "temp", "claude", "appdata", "local", "users", "windows"} - for part in parts[:-1]: - p = part.lower().strip() - if ( - p - and len(p) > 2 - and p not in skip - and not _re.match(r"^[a-z]:|^\.|^--", p) - ): - cc[p] += 1 - stem = _osp.splitext(fname)[0] - if stem and len(stem) > 2: - cc[stem.lower()] += 1 - ext = _osp.splitext(fname)[1].lstrip(".") - if ext in ("py", "ts", "js", "jsx", "tsx", "go", "rs", "java", "cs", "cpp"): - cc[ext] += 1 - - for s in project_sessions: - sid = s.get("id", "") - if not sid: - continue - for o in kv.list(KV.observations(sid)): - for c in o.get("concepts") or []: - if isinstance(c, str) and c: - concept_counts[c] += 1 - for f in o.get("files") or []: - _harvest_file(f, file_counts, concept_counts) - tn = o.get("toolName") - if tn: - concept_counts[tn] += 1 - ti = o.get("toolInput") - if isinstance(ti, str): - try: - ti = _j.loads(ti) - except Exception: - ti = {} - if isinstance(ti, dict): - for fk in ("path", "file_path", "file", "filename"): - _harvest_file(ti.get(fk, ""), file_counts, concept_counts) - narr = o.get("narrative") or o.get("raw") or "" - if isinstance(narr, str) and narr.startswith("{"): - try: - nd = _j.loads(narr) - if isinstance(nd, dict): - tn2 = nd.get("toolName") or nd.get("tool_name") - if tn2: - concept_counts[tn2] += 1 - for fk in ("path", "file_path", "file", "filename"): - _harvest_file( - nd.get(fk, ""), file_counts, concept_counts - ) - except Exception: - pass - - # memories for this project - for m in kv.list(KV.memories): - if m.get("project") == project: - for c in m.get("concepts") or []: - if c: - concept_counts[c] += 1 - for f in m.get("files") or []: - _harvest_file(f, file_counts, concept_counts) - - prof["topConcepts"] = [ - {"concept": c, "frequency": n} for c, n in concept_counts.most_common(20) - ] - prof["topFiles"] = [ - {"file": f, "frequency": n} for f, n in file_counts.most_common(20) - ] - prof["sessionCount"] = len(project_sessions) - - return prof - - -def export_data(kv: StateKV, data: Optional[Dict[str, Any]] = None) -> Dict[str, Any]: - if data is None: - data = {} - - exported_at = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - - # Check isolation - isolated = is_agent_scope_isolated() - isolated_agent_id = get_agent_id() - - # ---- v2 folder-based export (primary path) ---- - folder_pairs = kv.list(KV.folders) - folders_export = [] - for entry in folder_pairs: - fp = entry.get("folderPath") - aid = entry.get("agentId") - if not fp or not aid: - continue - # Apply isolation filter - if isolated and isolated_agent_id and aid != isolated_agent_id: - continue - - meta = kv.get(KV.folder_meta(fp, aid), "meta") or { - "folderPath": fp, - "agentId": aid, - "lastUpdated": entry.get("lastUpdated", ""), - "obsCount": entry.get("obsCount", 0), - } - observations = kv.list(KV.folder_obs(fp, aid)) - folders_export.append( - { - "folderPath": fp, - "agentId": aid, - "meta": meta, - "observations": observations, - } - ) - - memories = kv.list(KV.memories) - if isolated and isolated_agent_id: - memories = [m for m in memories if m.get("agentId") == isolated_agent_id] - - return { - "folders": folders_export, - "memories": memories, - "exportedAt": exported_at, - "version": "2.0", - } - - -def migrate_sessions_to_folders(kv: StateKV, dry_run: bool = False) -> Dict[str, Any]: - """Migrate legacy session-based observations to folder-based storage. - Non-destructive: old mem:sessions / mem:obs:* scopes are never deleted. - """ - sessions = kv.list(KV.sessions) - migrated_sessions = 0 - migrated_observations = 0 - errors = [] - - for session in sessions: - session_id = session.get("id") - if not session_id: - continue - try: - fp_raw = session.get("cwd") or session.get("project") or "unknown" - aid = (session.get("agentId") or "unknown").strip()[:_MAX_PATH_LEN] - try: - fp = normalize_folder_path(fp_raw) - except ValueError: - fp = "unknown" - - obs_list = kv.list(KV.observations(session_id)) - session_obs_count = 0 - for obs in obs_list: - obs_id = obs.get("id", "") - if obs_id.endswith(":raw"): - continue - folder_obs = { - "id": obs_id, - "folderPath": fp, - "agentId": aid, - "timestamp": obs.get("timestamp", ""), - "text": obs.get("narrative") - or obs.get("raw") - or obs.get("title") - or "", - "type": obs.get("type", "other"), - "title": obs.get("title", ""), - "concepts": obs.get("concepts") or [], - "files": obs.get("files") or [], - "importance": obs.get("importance", 5), - } - if isinstance(folder_obs["text"], dict): - import json as _json - - folder_obs["text"] = _json.dumps(folder_obs["text"])[:4000] - folder_obs["text"] = str(folder_obs["text"])[:4000] - - if not dry_run: - kv.set(KV.folder_obs(fp, aid), obs_id, folder_obs) - kv.set( - KV.obs_lookup, - obs_id, - { - "folderPath": fp, - "agentId": aid, - }, - ) - session_obs_count += 1 - migrated_observations += 1 - - if not dry_run and session_obs_count > 0: - meta_scope = KV.folder_meta(fp, aid) - meta = kv.get(meta_scope, "meta") or { - "folderPath": fp, - "agentId": aid, - "obsCount": 0, - "lastUpdated": session.get("updatedAt", ""), - "summary": None, - } - meta["obsCount"] = meta.get("obsCount", 0) + session_obs_count - meta["lastUpdated"] = ( - session.get("updatedAt", "") or meta["lastUpdated"] - ) - kv.set(meta_scope, "meta", meta) - - index_key = f"{fp}:{aid}" - kv.set( - KV.folders, - index_key, - { - "folderPath": fp, - "agentId": aid, - "lastUpdated": meta["lastUpdated"], - "obsCount": meta["obsCount"], - }, - ) - - migrated_sessions += 1 - except Exception as e: - errors.append({"sessionId": session_id, "error": str(e)}) - - return { - "migrated_sessions": migrated_sessions, - "migrated_observations": migrated_observations, - "errors": errors, - "dry_run": dry_run, - } - - -def set_project_profile( - kv: StateKV, project: str, profile: Dict[str, Any] -) -> Dict[str, Any]: - profile["updatedAt"] = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - kv.set(KV.profiles, project, profile) - - # Commit to Dolt - commit_if_enabled(kv, f"Set project profile for {project}", get_agent_id()) - - return profile - - -def get_relations(kv: StateKV) -> List[Dict[str, Any]]: - return kv.list(KV.relations) - - -def add_relation(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - rel = { - "id": generate_id("rel"), - "sourceId": data["sourceId"], - "targetId": data["targetId"], - "type": data["type"], - "createdAt": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - } - kv.set(KV.relations, rel["id"], rel) - - # Commit to Dolt - agent_id = data.get("agentId") or get_agent_id() - commit_if_enabled( - kv, - f"Add relation {rel['type']} between {rel['sourceId']} and {rel['targetId']}", - agent_id, - ) - - return rel - - -def evolve_memory(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - # Update memory content and create a new version - mem_id = data["memoryId"] - new_content = data["newContent"] - new_title = data.get("newTitle") - - existing = kv.get(KV.memories, mem_id) - if not existing: - raise ValueError("Memory not found") - - existing["isLatest"] = False - kv.set(KV.memories, existing["id"], existing) - - now = ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - new_mem = dict(existing) - new_mem["id"] = generate_id("mem") - new_mem["content"] = new_content - if new_title: - new_mem["title"] = new_title - else: - new_mem["title"] = new_content[:80] - new_mem["version"] = existing.get("version", 1) + 1 - new_mem["parentId"] = existing["id"] - new_mem["supersedes"] = [existing["id"]] - new_mem["createdAt"] = now - new_mem["updatedAt"] = now - new_mem["isLatest"] = True - - kv.set(KV.memories, new_mem["id"], new_mem) - - # Re-index - try: - _bm25_index.add(memory_to_observation(new_mem)) - _bm25_index.remove(existing["id"]) - except Exception: - pass - - comb_text = new_mem["title"] + " " + new_mem["content"] - vector_index_add_guarded( - new_mem["id"], "memory", comb_text, {"kind": "memory", "logId": new_mem["id"]} - ) - if _vector_index: - _vector_index.remove(existing["id"]) - - if _index_persistence: - _index_persistence.schedule_save() - - # Commit to Dolt - agent_id = data.get("agentId") or get_agent_id() or new_mem.get("agentId") - commit_if_enabled( - kv, - f"Evolve memory {new_mem['id']} (v{new_mem['version']}): {new_mem['title']}", - agent_id, - ) - - return {"success": True, "memory": new_mem} - - -def auto_forget(kv: StateKV, dry_run: bool = False) -> Dict[str, Any]: - now_dt = datetime.datetime.now(datetime.timezone.utc).replace(tzinfo=None) - evicted_memories = [] - evicted_observations = [] - evicted_folder_observations = [] - - # 1. Evict expired memories - memories = kv.list(KV.memories) - for mem in memories: - forget_after = mem.get("forgetAfter") - if forget_after: - try: - import dateutil.parser - - fa_dt = dateutil.parser.parse(forget_after) - if fa_dt.tzinfo: - fa_dt = fa_dt.replace(tzinfo=None) - if fa_dt < now_dt: - evicted_memories.append(mem["id"]) - except Exception as e: - print( - f"[auto_forget] Failed to parse forgetAfter '{forget_after}': {e}" - ) - - # 2. Evict low-value old session-based observations (importance <= 2, age > 180 days) - sessions = kv.list(KV.sessions) - for sess in sessions: - sid = sess.get("id") - if not sid: - continue - obs_list = kv.list(KV.observations(sid)) - for obs in obs_list: - importance = obs.get("importance") - ts = obs.get("timestamp") - if importance is not None and ts: - try: - import dateutil.parser - - ts_dt = dateutil.parser.parse(ts) - if ts_dt.tzinfo: - ts_dt = ts_dt.replace(tzinfo=None) - age_days = (now_dt - ts_dt).days - if importance <= 2 and age_days > 180: - evicted_observations.append((sid, obs["id"])) - except Exception as e: - print(f"[auto_forget] Failed to parse timestamp '{ts}': {e}") - - # 3. Evict expired or low-value old folder-based observations - folder_pairs = kv.list(KV.folders) - for entry in folder_pairs: - fp = entry.get("folderPath") - aid = entry.get("agentId") - if not fp or not aid: - continue - obs_list = kv.list(KV.folder_obs(fp, aid)) - for obs in obs_list: - obs_id = obs.get("id") - if not obs_id: - continue - - # Case A: Explicit forgetAfter - forget_after = obs.get("forgetAfter") - is_expired = False - if forget_after: - try: - import dateutil.parser - - fa_dt = dateutil.parser.parse(forget_after) - if fa_dt.tzinfo: - fa_dt = fa_dt.replace(tzinfo=None) - if fa_dt < now_dt: - is_expired = True - except Exception as e: - print( - f"[auto_forget] Failed to parse folder obs forgetAfter '{forget_after}': {e}" - ) - - # Case B: Low-value old observations (importance <= 2, age > 180 days) - is_stale_low_value = False - importance = obs.get("importance") - ts = obs.get("timestamp") - if importance is not None and ts: - try: - import dateutil.parser - - ts_dt = dateutil.parser.parse(ts) - if ts_dt.tzinfo: - ts_dt = ts_dt.replace(tzinfo=None) - age_days = (now_dt - ts_dt).days - if importance <= 2 and age_days > 180: - is_stale_low_value = True - except Exception as e: - print( - f"[auto_forget] Failed to parse folder obs timestamp '{ts}': {e}" - ) - - if is_expired or is_stale_low_value: - evicted_folder_observations.append((fp, aid, obs_id, obs)) - - if not dry_run: - # Commit evictions for memories - for mem_id in evicted_memories: - mem = kv.get(KV.memories, mem_id) - kv.delete(KV.memories, mem_id) - if mem and mem.get("imageRef"): - ref = mem["imageRef"] - refs = kv.get(KV.imageRefs, ref) or 0 - if refs > 0: - kv.set(KV.imageRefs, ref, refs - 1) - _bm25_index.remove(mem_id) - if _vector_index: - _vector_index.remove(mem_id) - - # Commit evictions for session-based observations - for sid, obs_id in evicted_observations: - base_oid = obs_id.replace(":raw", "") - obs = kv.get(KV.observations(sid), base_oid) - raw_obs = kv.get(KV.observations(sid), f"{base_oid}:raw") - - kv.delete(KV.observations(sid), base_oid) - kv.delete(KV.observations(sid), f"{base_oid}:raw") - - for o in (obs, raw_obs): - if o: - img = o.get("imageData") or o.get("imageRef") - if img: - refs = kv.get(KV.imageRefs, img) or 0 - if refs > 0: - kv.set(KV.imageRefs, img, refs - 1) - - _bm25_index.remove(base_oid) - _bm25_index.remove(f"{base_oid}:raw") - if _vector_index: - _vector_index.remove(base_oid) - _vector_index.remove(f"{base_oid}:raw") - - # Commit evictions for folder-based observations - folder_deletes = {} - for fp, aid, obs_id, obs in evicted_folder_observations: - kv.delete(KV.folder_obs(fp, aid), obs_id) - kv.delete(KV.obs_lookup, obs_id) - - if obs and isinstance(obs, dict) and obs.get("text"): - import hashlib - - fp_text = obs["text"][:4000] - dedup_fp = hashlib.sha256( - fp_text.strip().lower().encode("utf-8") - ).hexdigest() - kv.delete(KV.obs_dedup(fp, aid), dedup_fp) - - _bm25_index.remove(obs_id) - if _vector_index: - _vector_index.remove(obs_id) - - pair_key = (fp, aid) - folder_deletes[pair_key] = folder_deletes.get(pair_key, 0) + 1 - - for (fp, aid), count in folder_deletes.items(): - meta_scope = KV.folder_meta(fp, aid) - meta = kv.get(meta_scope, "meta") - if meta and isinstance(meta, dict): - current_count = meta.get("obsCount", 0) - meta["obsCount"] = max(0, current_count - count) - kv.set(meta_scope, "meta", meta) - - index_key = f"{fp}:{aid}" - index_entry = kv.get(KV.folders, index_key) - if index_entry and isinstance(index_entry, dict): - index_entry["obsCount"] = meta["obsCount"] - kv.set(KV.folders, index_key, index_entry) - - if evicted_memories or evicted_observations or evicted_folder_observations: - if _index_persistence: - _index_persistence.schedule_save() - safe_audit( - kv, - "auto_forget", - "mem::auto_forget", - evicted_memories - + [oid for _, oid in evicted_observations] - + [oid for _, _, oid, _ in evicted_folder_observations], - { - "evictedMemoriesCount": len(evicted_memories), - "evictedObservationsCount": len(evicted_observations) - + len(evicted_folder_observations), - "dryRun": False, - }, - ) - commit_if_enabled( - kv, - f"Auto forget: evicted {len(evicted_memories)} memories, {len(evicted_observations) + len(evicted_folder_observations)} observations", - "system", - ) - - return { - "success": True, - "evictedMemories": evicted_memories, - "evictedObservations": [oid for _, oid in evicted_observations] - + [oid for _, _, oid, _ in evicted_folder_observations], - "evicted": len(evicted_memories) - + len(evicted_observations) - + len(evicted_folder_observations), - "dryRun": dry_run, - } - - -def health_check(kv: StateKV) -> Dict[str, Any]: - db_status = "connected" - try: - kv._get_conn() # connection stays open per-thread (A3.1) - except Exception: - db_status = "disconnected" - - # ---- Folder-based counts ---- - folder_count = 0 - agent_count = 0 - pair_count = 0 - observation_count = 0 - try: - folder_pairs = kv.list(KV.folders) - pair_count = len(folder_pairs) - unique_folders: Set[str] = set() - unique_agents: Set[str] = set() - for entry in folder_pairs: - fp = entry.get("folderPath") - aid = entry.get("agentId") - if fp: - unique_folders.add(fp) - if aid: - unique_agents.add(aid) - observation_count += int(entry.get("obsCount") or 0) - folder_count = len(unique_folders) - agent_count = len(unique_agents) - except Exception as e: - print(f"[health_check] folder count failed: {e}") - - memory_count = 0 - try: - memory_count = len(kv.list(KV.memories)) - except Exception: - pass - - bm25_index_size = 0 - try: - bm25_index_size = _bm25_index.size - except Exception: - pass - - vector_index_size = 0 - try: - if _vector_index: - vector_index_size = _vector_index.size - except Exception: - pass - - # C4.2: Read sync state written by sync.py - sync_status = "never" - last_sync_at = None - db_size_bytes = 0 - wal_size_bytes = 0 - try: - sync_state_path = os.path.join( - os.path.expanduser("~"), ".agentcache", ".sync_state" - ) - if os.path.exists(sync_state_path): - with open(sync_state_path, "r", encoding="utf-8") as _sf: - _sync = json.loads(_sf.read()) - sync_status = _sync.get("sync_status", "never") - last_sync_at = _sync.get("last_sync_at") - except Exception: - pass - - # A3.3: DB file sizes - try: - db_stats = kv.stats() - db_size_bytes = db_stats.get("db_size_bytes", 0) - wal_size_bytes = db_stats.get("wal_size_bytes", 0) - except Exception: - pass - - return { - "status": "ok" if db_status == "connected" else "degraded", - "folderCount": folder_count, - "agentCount": agent_count, - "pairCount": pair_count, - "observationCount": observation_count, - "memoryCount": memory_count, - "bm25IndexSize": bm25_index_size, - "vectorIndexSize": vector_index_size, - "dbPath": kv.db_path, - "dbSizeBytes": db_size_bytes, - "walSizeBytes": wal_size_bytes, - "syncStatus": sync_status, - "lastSyncAt": last_sync_at, - } - - -def strip_xml_wrappers(raw: str) -> str: - if not raw: - return "" - cleaned = raw.strip() - cleaned = re.sub(r"```xml\s*\n?", "", cleaned, flags=re.IGNORECASE) - cleaned = re.sub(r"```", "", cleaned) - cleaned = cleaned.strip() - root_match = re.search( - r"(<[a-zA-Z_][a-zA-Z0-9_-]*>[\s\S]*<\/[a-zA-Z_][a-zA-Z0-9_-]*>)", cleaned - ) - if root_match: - return root_match.group(1).strip() - return cleaned - - -def get_xml_tag(text: str, tag: str) -> Optional[str]: - cleaned = strip_xml_wrappers(text) - pattern = rf"<{tag}>(.*?)" - match = re.search(pattern, cleaned, re.DOTALL) - return match.group(1).strip() if match else None - - -def get_xml_children(text: str, parent_tag: str, child_tag: str) -> List[str]: - parent_content = get_xml_tag(text, parent_tag) - if not parent_content: - return [] - pattern = rf"<{child_tag}>(.*?)" - return [m.strip() for m in re.findall(pattern, parent_content, re.DOTALL)] - - -def generate_content(system_instruction: str, prompt: str) -> str: - api_key = os.getenv("GEMINI_API_KEY") or os.getenv("GOOGLE_API_KEY") - if not api_key: - raise ValueError("No Gemini/Google API key found") - model = os.getenv("GEMINI_MODEL", "gemini-2.5-flash") - url = f"https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent?key={api_key}" - payload = { - "contents": [{"role": "user", "parts": [{"text": prompt}]}], - "systemInstruction": {"parts": [{"text": system_instruction}]}, - "generationConfig": {"temperature": 0.2}, - } - - req_data = json.dumps(payload).encode("utf-8") - import urllib.request - - req = urllib.request.Request( - url, data=req_data, headers={"Content-Type": "application/json"}, method="POST" - ) - - try: - with urllib.request.urlopen(req, timeout=60.0) as response: # nosec B310 - resp_data = json.loads(response.read().decode("utf-8")) - - candidates = resp_data.get("candidates", []) - if not candidates: - raise RuntimeError("Gemini generateContent returned no candidates") - - parts = candidates[0].get("content", {}).get("parts", []) - if not parts: - raise RuntimeError("Gemini generateContent candidate content had no parts") - - return parts[0].get("text", "") - except Exception as e: - raise RuntimeError(f"Gemini generateContent call failed: {e}") - - -def summarize(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: - session_id = data.get("sessionId") - if not session_id: - return {"success": False, "error": "sessionId is required"} - - session = kv.get(KV.sessions, session_id) - if not session: - return {"success": False, "error": "session_not_found"} - - observations = kv.list(KV.observations(session_id)) - compressed = [o for o in observations if o.get("title")] - if not compressed: - return {"success": False, "error": "no_observations"} - - SUMMARY_SYSTEM = """You are a session summarization assistant. Your job is to read all raw tool executions and outcomes from a coding session and produce a high-fidelity summary. - - Output XML: - - Concise title summarizing the session - 1-2 paragraphs of narrative describing what was done, what succeeded, and what failed - - Architectural decision, key insight, or choice made - - - path/to/modified/file - - - important concept, library, tool, or command used - - """ - - chunk_size = 400 - chunks = [ - compressed[i : i + chunk_size] for i in range(0, len(compressed), chunk_size) - ] - - partial_summaries = [] - for idx, chunk in enumerate(chunks): - obs_text = "" - for o in chunk: - obs_text += f"[{o.get('type')}] {o.get('title')}\n{o.get('narrative') or ''}\nFiles: {', '.join(o.get('files') or [])}\n\n" - - prompt = f"Summarize this chunk {idx + 1}/{len(chunks)} of observations:\n\n{obs_text}" - try: - response = generate_content(SUMMARY_SYSTEM, prompt) - cleaned = strip_xml_wrappers(response) - title = get_xml_tag(cleaned, "title") - if not title: - continue - partial_summaries.append( - { - "title": title, - "narrative": get_xml_tag(cleaned, "narrative") or "", - "keyDecisions": get_xml_children(cleaned, "decisions", "decision"), - "filesModified": get_xml_children(cleaned, "files", "file"), - "concepts": get_xml_children(cleaned, "concepts", "concept"), - } - ) - except Exception as e: - last_error = str(e) - print(f"[summarize] Chunk {idx + 1} failed: {e}") - - if not partial_summaries: - return { - "success": False, - "error": f"No chunks summarized successfully. Last error: {last_error}", - } - - if len(partial_summaries) == 1: - final_summary = { - "sessionId": session_id, - "project": session.get("project"), - "createdAt": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - "title": partial_summaries[0]["title"], - "narrative": partial_summaries[0]["narrative"], - "keyDecisions": partial_summaries[0]["keyDecisions"], - "filesModified": partial_summaries[0]["filesModified"], - "concepts": partial_summaries[0]["concepts"], - "observationCount": len(compressed), - } - else: - REDUCE_SYSTEM = """You are a session summarization reducer. Reduce multiple partial chunk summaries into a single final summary. - - Output XML: - - Concise final title summarizing the entire session - Comprehensive narrative describing what was done, what succeeded, and what failed - - Architectural decision, key insight, or choice made - - - path/to/modified/file - - - important concept, library, tool, or command used - - """ - - reduce_prompt = "Reduce these partial summaries:\n\n" - for idx, ps in enumerate(partial_summaries): - reduce_prompt += f"[Chunk {idx + 1}]\nTitle: {ps['title']}\nNarrative: {ps['narrative']}\nDecisions: {', '.join(ps['keyDecisions'])}\nFiles: {', '.join(ps['filesModified'])}\nConcepts: {', '.join(ps['concepts'])}\n\n" - - try: - response = generate_content(REDUCE_SYSTEM, reduce_prompt) - cleaned = strip_xml_wrappers(response) - final_summary = { - "sessionId": session_id, - "project": session.get("project"), - "createdAt": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - "title": get_xml_tag(cleaned, "title") or partial_summaries[0]["title"], - "narrative": get_xml_tag(cleaned, "narrative") or "", - "keyDecisions": get_xml_children(cleaned, "decisions", "decision"), - "filesModified": get_xml_children(cleaned, "files", "file"), - "concepts": get_xml_children(cleaned, "concepts", "concept"), - "observationCount": len(compressed), - } - except Exception as e: - return {"success": False, "error": f"Reduction failed: {e}"} - - kv.set(KV.summaries, session_id, final_summary) - - session = kv.get(KV.sessions, session_id) - if session: - session["title"] = final_summary["title"] - session["summary"] = final_summary["narrative"] - kv.set(KV.sessions, session_id, session) - - safe_audit( - kv, - "compress", - "mem::summarize", - [session_id], - {"title": final_summary["title"], "observationCount": len(compressed)}, - ) - - return {"success": True, "summary": final_summary} - - -def consolidate( - kv: StateKV, project: Optional[str] = None, min_observations: int = 10 -) -> Dict[str, Any]: - sessions = list_sessions(kv) - if project: - sessions = [s for s in sessions if s.get("project") == project] - - all_obs = [] - for s in sessions: - obs_list = kv.list(KV.observations(s["id"])) - for o in obs_list: - if o.get("title") and o.get("importance", 5) >= 5: - all_obs.append((o, s["id"])) - - if len(all_obs) < min_observations: - return { - "consolidated": 0, - "reason": "insufficient_observations", - "success": True, - } - - # Group observations by concepts - concept_groups = {} - for obs, sid in all_obs: - concepts = obs.get("concepts") or [] - for c in concepts: - key = c.lower().strip() - if not key: - continue - if key not in concept_groups: - concept_groups[key] = [] - concept_groups[key].append((obs, sid)) - - # Sort groups that have >= 3 observations by size descending - sorted_groups = sorted( - [(k, g) for k, g in concept_groups.items() if len(g) >= 3], - key=lambda x: len(x[1]), - reverse=True, - ) - - consolidated_count = 0 - existing_memories = kv.list(KV.memories) - - MAX_LLM_CALLS = 10 - llm_calls = 0 - - # Prompt templates - CONSOLIDATION_SYSTEM = """You are a memory consolidation engine. Given a set of related observations from coding sessions, synthesize them into a single long-term memory. - - Output XML: - - pattern|preference|architecture|bug|workflow|fact - Concise memory title (max 80 chars) - 2-4 sentence description of the learned insight - - key term - - - relevant/file/path - - 1-10 how confident/important this memory is - """ - - for concept, obs_group in sorted_groups: - if llm_calls >= MAX_LLM_CALLS: - break - - # Get top 8 by importance - top = sorted(obs_group, key=lambda x: x[0].get("importance", 5), reverse=True)[ - :8 - ] - session_ids = list(set([x[1] for x in top])) - obs_ids = list(set([x[0]["id"] for x in top])) - - prompt_parts = [] - for obs, sid in top: - prompt_parts.append( - f"[{obs.get('type')}] {obs.get('title')}\n{obs.get('narrative') or ''}\nFiles: {', '.join(obs.get('files') or [])}\nImportance: {obs.get('importance', 5)}" - ) - obs_prompt = "\n\n".join(prompt_parts) - - try: - response = generate_content( - CONSOLIDATION_SYSTEM, - f'Concept: "{concept}"\n\nObservations:\n{obs_prompt}', - ) - llm_calls += 1 - - cleaned = strip_xml_wrappers(response) - m_type = get_xml_tag(cleaned, "type") or "fact" - m_title = get_xml_tag(cleaned, "title") - m_content = get_xml_tag(cleaned, "content") - - if not m_title or not m_content: - continue - - m_strength_str = get_xml_tag(cleaned, "strength") or "5" - try: - m_strength = max(1, min(10, int(m_strength_str))) - except Exception: - m_strength = 5 - - concepts_list = get_xml_children(cleaned, "concepts", "concept") - files_list = get_xml_children(cleaned, "files", "file") - - now = ( - datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z") - ) - - # Find existing memory with same title - existing_match = None - for mem in existing_memories: - if ( - mem.get("title", "").lower() == m_title.lower() - and mem.get("isLatest") is not False - ): - if ( - not project - or not mem.get("project") - or mem.get("project") == project - ): - existing_match = mem - break - - if existing_match: - existing_match["isLatest"] = False - kv.set(KV.memories, existing_match["id"], existing_match) - - evolved = { - "id": generate_id("mem"), - "createdAt": now, - "updatedAt": now, - "type": m_type, - "title": m_title, - "content": m_content, - "concepts": concepts_list, - "files": files_list, - "sessionIds": session_ids, - "strength": m_strength, - "version": (existing_match.get("version") or 1) + 1, - "parentId": existing_match["id"], - "supersedes": [existing_match["id"]] - + (existing_match.get("supersedes") or []), - "sourceObservationIds": obs_ids, - "isLatest": True, - } - if project: - evolved["project"] = project - kv.set(KV.memories, evolved["id"], evolved) - try: - _bm25_index.add(memory_to_observation(evolved)) - if existing_match: - _bm25_index.remove(existing_match["id"]) - except Exception: - pass - comb_text = evolved["title"] + " " + evolved["content"] - vector_index_add_guarded( - evolved["id"], - "memory", - comb_text, - {"kind": "memory", "logId": evolved["id"]}, - ) - if _vector_index and existing_match: - try: - _vector_index.remove(existing_match["id"]) - except Exception: - pass - consolidated_count += 1 - else: - memory = { - "id": generate_id("mem"), - "createdAt": now, - "updatedAt": now, - "type": m_type, - "title": m_title, - "content": m_content, - "concepts": concepts_list, - "files": files_list, - "sessionIds": session_ids, - "strength": m_strength, - "version": 1, - "sourceObservationIds": obs_ids, - "isLatest": True, - } - if project: - memory["project"] = project - kv.set(KV.memories, memory["id"], memory) - try: - _bm25_index.add(memory_to_observation(memory)) - except Exception: - pass - comb_text = memory["title"] + " " + memory["content"] - vector_index_add_guarded( - memory["id"], - "memory", - comb_text, - {"kind": "memory", "logId": memory["id"]}, - ) - consolidated_count += 1 - - except Exception as e: - print(f"[consolidate] Concept '{concept}' failed: {e}") - - # === Semantic Memory Fact Merger === - summaries = kv.list(KV.summaries) - new_facts_count = 0 - if len(summaries) >= 5: - recent_summaries = sorted( - summaries, key=lambda s: s.get("createdAt", ""), reverse=True - )[:20] - - SEMANTIC_MERGE_SYSTEM = """You are a memory consolidation engine. Given overlapping episodic memories (session summaries), extract stable factual knowledge. - - Output format (XML): - - Concise factual statement - - - Rules: - - Extract only facts that appear in 2+ episodes or are highly confident - - Confidence reflects how well-supported the fact is across episodes - - Combine overlapping information into single concise facts - - Skip ephemeral details (specific error messages, temporary states)""" - - prompt_parts = [] - for i, s in enumerate(recent_summaries): - prompt_parts.append( - f"[Episode {i + 1}]\nTitle: {s.get('title')}\nNarrative: {s.get('narrative') or ''}\nConcepts: {', '.join(s.get('concepts') or [])}" - ) - merge_prompt = ( - "Consolidate these episodic memories into stable facts:\n\n" - + "\n\n".join(prompt_parts) - ) - - try: - response = generate_content(SEMANTIC_MERGE_SYSTEM, merge_prompt) - fact_matches = re.findall( - r'([^<]+)', response, re.DOTALL - ) - - existing_semantic = kv.list(KV.semantic) - now = ( - datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z") - ) - - for conf_str, fact_text in fact_matches: - fact_text = fact_text.strip() - try: - confidence = float(conf_str) - except Exception: - confidence = 0.5 - - existing = None - for es in existing_semantic: - if es.get("fact", "").lower() == fact_text.lower(): - existing = es - break - - if existing: - existing["accessCount"] = (existing.get("accessCount") or 0) + 1 - existing["lastAccessedAt"] = now - existing["updatedAt"] = now - existing["confidence"] = max( - existing.get("confidence", 0.5), confidence - ) - kv.set(KV.semantic, existing["id"], existing) - else: - sem = { - "id": generate_id("sem"), - "fact": fact_text, - "confidence": confidence, - "sourceSessionIds": [ - s["sessionId"] for s in recent_summaries if "sessionId" in s - ], - "sourceMemoryIds": [], - "accessCount": 1, - "lastAccessedAt": now, - "strength": confidence, - "createdAt": now, - "updatedAt": now, - } - kv.set(KV.semantic, sem["id"], sem) - new_facts_count += 1 - except Exception as e: - print(f"[consolidate] Semantic merge failed: {e}") - - # === Procedural Memory Extraction === - memories = kv.list(KV.memories) - new_procs_count = 0 - patterns = [] - for m in memories: - if m.get("isLatest") is not False and m.get("type") == "pattern": - freq = len(m.get("sessionIds") or []) - if freq >= 2: - patterns.append({"content": m.get("content", ""), "frequency": freq}) - - if len(patterns) >= 2: - PROCEDURAL_EXTRACTION_SYSTEM = """You are a procedural memory extractor. Given repeated patterns and workflows observed across sessions, extract reusable procedures. - - Output format (XML): - - - Step 1 description - Step 2 description - - - - Rules: - - Only extract procedures observed 2+ times - - Steps should be concrete and actionable - - Trigger condition should be specific enough to match automatically""" - - prompt_parts = [] - for i, p in enumerate(patterns): - prompt_parts.append( - f"[Pattern {i + 1}] (seen {p['frequency']}x)\n{p['content']}" - ) - proc_prompt = ( - "Extract reusable procedures from these recurring patterns:\n\n" - + "\n\n".join(prompt_parts) - ) - - try: - response = generate_content(PROCEDURAL_EXTRACTION_SYSTEM, proc_prompt) - proc_matches = re.findall( - r'([\s\S]*?)', - response, - re.DOTALL, - ) - - existing_procs = kv.list(KV.procedural) - now = ( - datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z") - ) - - for name, trigger, steps_block in proc_matches: - steps = [ - s.strip() - for s in re.findall(r"([^<]+)", steps_block, re.DOTALL) - ] - - existing = None - for ep in existing_procs: - if ep.get("name", "").lower() == name.lower(): - existing = ep - break - - if existing: - existing["frequency"] = (existing.get("frequency") or 1) + 1 - existing["updatedAt"] = now - existing["strength"] = min( - 1.0, (existing.get("strength") or 0.5) + 0.1 - ) - kv.set(KV.procedural, existing["id"], existing) - else: - proc = { - "id": generate_id("proc"), - "name": name, - "steps": steps, - "triggerCondition": trigger, - "frequency": 1, - "sourceSessionIds": [], - "strength": 0.5, - "createdAt": now, - "updatedAt": now, - } - kv.set(KV.procedural, proc["id"], proc) - new_procs_count += 1 - except Exception as e: - print(f"[consolidate] Procedural extraction failed: {e}") - - res_summary = { - "success": True, - "consolidated": consolidated_count, - "totalObservations": len(all_obs), - "semantic": {"newFacts": new_facts_count, "totalSummaries": len(summaries)}, - "procedural": { - "newProcedures": new_procs_count, - "patternsAnalyzed": len(patterns), - }, - } - if _index_persistence and consolidated_count > 0: - _index_persistence.schedule_save() - safe_audit(kv, "consolidate", "mem::consolidate-pipeline", [], res_summary) - commit_if_enabled( - kv, - f"Consolidation complete: consolidated={consolidated_count}, facts={new_facts_count}, procs={new_procs_count}", - "system", - ) - return res_summary - - -# ===================================================================== -# Folder Graph -# ===================================================================== - - -def folder_color(path: str) -> str: - """Hash a folder path string to an HSL color string. - - Replicates the JS ``folderColor(id)`` function in src/viewer/index.html - exactly, using the light-mode lightness range (38 + h%14). - - Algorithm (matches JS): - h = 0 - for each char: h = (h * 31 + ord(char)) & 0xfffffff - hue = (h % 360 + 360) % 360 - sat = 55 + (h % 25) # percent, 55-79 - lig = 38 + (h % 14) # percent, 38-51 (light mode) - - Returns: - A CSS ``hsl(hue, sat%, lig%)`` string. - """ - h = 0 - for ch in path: - h = (h * 31 + ord(ch)) & 0xFFFFFFF - - hue = (h % 360 + 360) % 360 - sat_pct = 55 + (h % 25) - lig_pct = 38 + (h % 14) - - return f"hsl({hue}, {sat_pct}%, {lig_pct}%)" - - -def folder_graph_build(kv: StateKV) -> Dict[str, Any]: - """Build graph data for the viewer's Graph tab. - - Reads all (folder_path, agent_id) pairs from ``KV.folders``, - aggregates per-folder node metadata, loads observations to collect - text for cross-reference edge detection, then emits three edge types: - - - ``same-parent``: two folders share the same ``os.path.dirname`` - - ``cross-ref``: folder A's combined obs text contains folder B's path - - ``agent-shared``: two folders share a common agentId - - Returns: - {"nodes": [...], "edges": [...]} - - Each node:: - - { - "id": folderPath, - "label": basename(folderPath), - "folderPath": folderPath, - "agentIds": [...], - "obsCount": int, - "color": "#rrggbb", - } - - Each edge:: - - { - "source": folderPath, - "target": folderPath, - "type": "same-parent" | "cross-ref" | "agent-shared", - # agent-shared only: - "agentId": str, - } - - Edges are deduplicated on (source, target, type). - """ - index_entries = kv.list(KV.folders) - if is_agent_scope_isolated(): - aid = get_agent_id() - if aid: - index_entries = [e for e in index_entries if e.get("agentId") == aid] - - # --- Build folder_map and collect obs text per (folder, agent) pair --- - # folder_map: folderPath -> {"folderPath", "agentIds": set, "obsCount", "color"} - folder_map: Dict[str, Dict[str, Any]] = {} - # pair_obs_texts: (folder_path, agent_id) -> combined text string - pair_obs_texts: Dict[Tuple[str, str], str] = {} - - for entry in index_entries: - fp = entry.get("folderPath", "") - aid = entry.get("agentId", "") - if not fp: - continue - - if fp not in folder_map: - folder_map[fp] = { - "folderPath": fp, - "agentIds": set(), - "obsCount": 0, - "color": folder_color(fp), - } - - folder_map[fp]["agentIds"].add(aid) - folder_map[fp]["obsCount"] += entry.get("obsCount", 0) - - # Load observations to build combined text for cross-ref detection - obs_scope = KV.folder_obs(fp, aid) - obs_list = kv.list(obs_scope) - combined_parts = [] - for obs in obs_list: - text = obs.get("text") or "" - title = obs.get("title") or "" - combined_parts.append(f"{text} {title}") - pair_obs_texts[(fp, aid)] = " ".join(combined_parts) - - # --- Build nodes --- - nodes = [] - for fp, info in folder_map.items(): - nodes.append( - { - "id": fp, - "label": os.path.basename(fp) or fp, - "folderPath": fp, - "agentIds": sorted(info["agentIds"]), - "obsCount": info["obsCount"], - "color": info["color"], - } - ) - - # --- Build edges --- - edges: List[Dict[str, Any]] = [] - # Deduplicate on (frozenset(source, target), type) so that (A,B) and (B,A) - # are treated as the same edge (REQ-028). - seen_edges: Set[Tuple[Any, str]] = set() - - def add_edge(edge: Dict[str, Any]) -> None: - key = (frozenset([edge["source"], edge["target"]]), edge["type"]) - if key not in seen_edges: - seen_edges.add(key) - edges.append(edge) - - folder_paths = list(folder_map.keys()) - - # Edge type 1 — same-parent - for i in range(len(folder_paths)): - for j in range(i + 1, len(folder_paths)): - a = folder_paths[i] - b = folder_paths[j] - # Use posixpath-style dirname on forward-slash paths - if a.rsplit("/", 1)[0] == b.rsplit("/", 1)[0] and "/" in a and "/" in b: - add_edge({"source": a, "target": b, "type": "same-parent"}) - elif os.path.dirname(a) == os.path.dirname(b) and os.path.dirname(a) != "": - add_edge({"source": a, "target": b, "type": "same-parent"}) - - # Edge type 2 — cross-reference (folder A's obs text mentions folder B's path) - for (fp_a, _agent_a), text_a in pair_obs_texts.items(): - for fp_b in folder_paths: - if fp_b != fp_a and fp_b in text_a: - add_edge({"source": fp_a, "target": fp_b, "type": "cross-ref"}) - - # Edge type 3 — agent-shared (two folders share the same agentId) - # Build: agentId -> [folder_paths that have this agent] - agent_to_folders: Dict[str, List[str]] = {} - for fp, info in folder_map.items(): - for aid in info["agentIds"]: - agent_to_folders.setdefault(aid, []).append(fp) - - for aid, fps in agent_to_folders.items(): - for i in range(len(fps)): - for j in range(i + 1, len(fps)): - add_edge( - { - "source": fps[i], - "target": fps[j], - "type": "agent-shared", - "agentId": aid, - } - ) - - return {"nodes": nodes, "edges": edges} - - -# Setup persistence helper wire-ups -def set_index_persistence(persistence: IndexPersistence) -> None: - global _index_persistence - _index_persistence = persistence - - -def set_embedding_provider(provider) -> None: - global _embedding_provider, _hybrid_search - _embedding_provider = provider - _hybrid_search = HybridSearch(_bm25_index, _vector_index, _embedding_provider, None) - - -def set_stream_broadcaster(broadcaster) -> None: - global _stream_broadcaster - _stream_broadcaster = broadcaster - - -def broadcast_stream(payload: Dict[str, Any]) -> None: - if _stream_broadcaster: - try: - _stream_broadcaster(payload) - except Exception as e: - print(f"[broadcaster] Failed: {e}") - - -def backfill_obs_lookup_if_needed(kv: StateKV) -> None: - """Ensure every folder observation has an entry in KV.obs_lookup.""" - folders = kv.list(KV.folders) - if not folders: - return - - # Check if lookup index needs populating - lookups = kv.list(KV.obs_lookup) - if len(lookups) >= sum(int(f.get("obsCount", 0)) for f in folders): - return # already populated - - print("[db] Backfilling obs_lookup index...") - for entry in folders: - fp = entry.get("folderPath") - aid = entry.get("agentId") - if not fp or not aid: - continue - obs_list = kv.list(KV.folder_obs(fp, aid)) - for obs in obs_list: - oid = obs.get("id") - if oid: - kv.set(KV.obs_lookup, oid, {"folderPath": fp, "agentId": aid}) - print("[db] obs_lookup backfill complete.") - - -def verify_index_sync_on_boot(kv: StateKV) -> bool: - """Check if the search index size matches the database counts. - Returns True if in sync, False if a rebuild is needed. - """ - try: - # 1. Total folder obs count - folders = kv.list(KV.folders) - folder_obs_count = sum(int(f.get("obsCount", 0)) for f in folders) - - # 2. Total memories count - memories = kv.list(KV.memories) - latest_memories_count = len( - [m for m in memories if m.get("isLatest") is not False] - ) - - # 3. Total legacy observations count - legacy_obs_count = 0 - try: - sessions = kv.list(KV.sessions) - for s in sessions: - sid = s.get("id") - if sid: - obs_list = kv.list(KV.observations(sid)) - # Only legacy observations that were indexed (having title and narrative) - legacy_obs_count += len( - [o for o in obs_list if o.get("title") and o.get("narrative")] - ) - except Exception: - pass - - total_db_count = folder_obs_count + latest_memories_count + legacy_obs_count - index_size = _bm25_index.size - - if total_db_count != index_size: - print( - f"[persistence] Index out of sync with DB (DB={total_db_count}, Index={index_size}). Rebuild required." - ) - return False - - print(f"[persistence] Index is in sync with DB (size={index_size}).") - return True - except Exception as e: - print(f"[persistence] verify_index_sync_on_boot failed: {e}") - return False diff --git a/src/agentcache/legacy.py b/src/agentcache/legacy.py index c180c8f108e5cf001737e669ab669dd229c814c7..e3266def1a15119c6d26e9283f8a31118fc6b964 100644 --- a/src/agentcache/legacy.py +++ b/src/agentcache/legacy.py @@ -2,11 +2,9 @@ import datetime import hashlib - import json import os import re -import sqlite3 import threading import time import uuid @@ -15,7 +13,6 @@ from typing import Any, Dict, List, Optional, Set, Tuple from .core.kv_scopes import KV # noqa: F401 re-exported for backward compat from .core.search_service import IndexPersistence # noqa: F401 from .db import StateKV -from .search import HybridSearch, SearchIndex, VectorIndex # ===================================================================== # Global Variables / Module State @@ -811,7 +808,8 @@ def observe(kv: StateKV, payload: Dict[str, Any]) -> Dict[str, Any]: if k in raw_for_synthetic: synthetic[k] = raw_for_synthetic[k] kv.set(KV.observations(session_id), obs_id, synthetic) - if _search_service: _search_service.bm25.add(synthetic) + if _search_service: + _search_service.bm25.add(synthetic) comb_text = synthetic["title"] + " " + (synthetic.get("narrative") or "") vector_index_add_guarded( @@ -854,9 +852,11 @@ def observe(kv: StateKV, payload: Dict[str, Any]) -> Dict[str, Any]: def _get_observation_store(kv: StateKV): from . import app as app_module + if getattr(app_module, "observation_store", None) is not None: return app_module.observation_store from .core.observation_store import ObservationStore + return ObservationStore(kv, search_service=_search_service) @@ -876,7 +876,6 @@ def dedup_folder_observations( return store.dedup(folder_path_raw, agent_id_raw) - # ===================================================================== # Folder-Based Search (folder_search) # ===================================================================== @@ -918,7 +917,6 @@ def folder_timeline( ) - # ===================================================================== # Memory System (Remember, Forget, Evolve) # ===================================================================== @@ -1052,7 +1050,6 @@ def forget(kv: StateKV, data: Dict[str, Any]) -> Dict[str, Any]: return store.forget(data) - # ===================================================================== # Prompt Context Compilation System # ===================================================================== @@ -1958,7 +1955,6 @@ def rebuild_index(kv: StateKV) -> int: return store.rebuild_index() - # ===================================================================== # Advanced Function Stubs / CRUD Operations # ===================================================================== @@ -2540,7 +2536,8 @@ def auto_forget(kv: StateKV, dry_run: bool = False) -> Dict[str, Any]: refs = kv.get(KV.imageRefs, ref) or 0 if refs > 0: kv.set(KV.imageRefs, ref, refs - 1) - if _search_service: _search_service.remove(mem_id) + if _search_service: + _search_service.remove(mem_id) # Commit evictions for session-based observations for sid, obs_id in evicted_observations: @@ -2578,7 +2575,8 @@ def auto_forget(kv: StateKV, dry_run: bool = False) -> Dict[str, Any]: ).hexdigest() kv.delete(KV.obs_dedup(fp, aid), dedup_fp) - if _search_service: _search_service.remove(obs_id) + if _search_service: + _search_service.remove(obs_id) pair_key = (fp, aid) folder_deletes[pair_key] = folder_deletes.get(pair_key, 0) + 1 @@ -2729,7 +2727,6 @@ def health_check(kv: StateKV) -> Dict[str, Any]: } - def strip_xml_wrappers(raw: str) -> str: if not raw: return "" @@ -3543,7 +3540,9 @@ def backfill_obs_lookup_if_needed(kv: StateKV) -> None: store.backfill_lookup() -def verify_index_sync_on_boot(kv: StateKV, search_service: Optional[Any] = None) -> bool: +def verify_index_sync_on_boot( + kv: StateKV, search_service: Optional[Any] = None +) -> bool: """Check if the search index size matches the database counts. Returns True if in sync, False if a rebuild is needed. """ @@ -3551,6 +3550,7 @@ def verify_index_sync_on_boot(kv: StateKV, search_service: Optional[Any] = None) svc = search_service or _search_service if svc is None: from . import app as app_module + svc = getattr(app_module, "search_service", None) # 1. Total folder obs count @@ -3577,4 +3577,3 @@ def verify_index_sync_on_boot(kv: StateKV, search_service: Optional[Any] = None) except Exception as e: print(f"[persistence] verify_index_sync_on_boot failed: {e}") return False - diff --git a/src/agentcache/replay_import.py b/src/agentcache/replay_import.py index cc02a77e8f9de03fa3e104409df7c07c785aeba0..484fc56dbbc4bbf8477ef92b401280bdd6b32a7c 100644 --- a/src/agentcache/replay_import.py +++ b/src/agentcache/replay_import.py @@ -219,7 +219,7 @@ def derive_crystal_and_lessons( compressed: List[Dict[str, Any]], first_prompt: str = None, ) -> None: - from .functions import KV + from .core import KV if not raw_obs: return @@ -368,7 +368,8 @@ def find_jsonl_files(root: str, limit=200) -> Tuple[List[str], bool, int, bool]: def import_jsonl_data(kv, path: str = None, max_files: int = None) -> Dict[str, Any]: - from .functions import KV, _bm25_index, build_synthetic_compression + from .core import KV + from .legacy import build_synthetic_compression default_root = os.path.expanduser(os.path.join("~", ".claude", "projects")) raw_path = path or default_root @@ -481,24 +482,12 @@ def import_jsonl_data(kv, path: str = None, max_files: int = None) -> Dict[str, } kv.set(KV.sessions, session["id"], session) - from .functions import vector_index_add_guarded - compressed = [] for obs in parsed["observations"]: synthetic = build_synthetic_compression(obs) compressed.append(synthetic) kv.set(KV.observations(parsed["sessionId"]), obs["id"], synthetic) - # Index - _bm25_index.add(synthetic) - comb_text = synthetic["title"] + " " + (synthetic.get("narrative") or "") - vector_index_add_guarded( - synthetic["id"], - synthetic["sessionId"], - comb_text, - {"kind": "synthetic", "logId": synthetic["id"]}, - ) - observation_count += len(parsed["observations"]) session_ids.append(parsed["sessionId"]) @@ -511,18 +500,9 @@ def import_jsonl_data(kv, path: str = None, max_files: int = None) -> Dict[str, first_prompt, ) - # Save the updated persistence state - from . import functions - - if functions._index_persistence: - try: - functions._index_persistence.save() - except Exception as e: - print(f"[import-jsonl] Warning saving index persistence: {e}") - # Audit trail try: - from .functions import log_audit + from .legacy import log_audit log_audit( kv, diff --git a/src/agentcache/routes/__init__.py b/src/agentcache/routes/__init__.py index d271f5fcc8927f3549633a0df668d9c094cf5ee4..9af8d021085361959525b37a784351d496f6b6eb 100644 --- a/src/agentcache/routes/__init__.py +++ b/src/agentcache/routes/__init__.py @@ -4,31 +4,26 @@ Flask blueprints for agentmemory-python. Import and register all blueprints via register_blueprints(app). """ -from .graph import create_graph_bp -from .health import create_health_bp +from .graph import graph_bp +from .health import health_bp from .mcp import mcp_bp -from .memories import create_memories_bp +from .memories import memories_bp from .migration import migration_bp from .observations import create_observations_bp, observations_bp from .search import search_bp -def register_blueprints(app, observation_store=None, search_service=None, kv=None): +def register_blueprints(app, observation_store=None, search_service=None): """Register all route blueprints on a Flask application instance.""" obs_bp = ( create_observations_bp(observation_store) if observation_store else observations_bp ) - if kv is None and observation_store is not None: - kv = observation_store.kv app.register_blueprint(obs_bp) - app.register_blueprint(create_memories_bp(kv)) + app.register_blueprint(memories_bp) app.register_blueprint(search_bp) - app.register_blueprint(create_graph_bp(kv)) - app.register_blueprint(create_health_bp(kv)) + app.register_blueprint(graph_bp) + app.register_blueprint(health_bp) app.register_blueprint(mcp_bp) app.register_blueprint(migration_bp) - - - diff --git a/src/agentcache/routes/graph.py b/src/agentcache/routes/graph.py index 47f4f3a7cdd091dbaadca376c375ca354a327e87..88448f13dfcff8d51718c468818890d8833c8d09 100644 --- a/src/agentcache/routes/graph.py +++ b/src/agentcache/routes/graph.py @@ -22,6 +22,7 @@ def create_graph_bp(kv=None): if kv is not None: return kv from .. import app as app_module + return app_module.kv # ------------------------------------------------------------------ @@ -81,5 +82,4 @@ def create_graph_bp(kv=None): return bp - graph_bp = create_graph_bp(None) diff --git a/src/agentcache/routes/health.py b/src/agentcache/routes/health.py index 7c59272b4044ddb0ac7702d7c305f066e9d21695..31c24b4497f2dff675d35e3e87540ffc0a49d66c 100644 --- a/src/agentcache/routes/health.py +++ b/src/agentcache/routes/health.py @@ -25,6 +25,7 @@ def create_health_bp(kv=None, embedding_provider=None): if kv is not None: return kv from .. import app as app_module + return app_module.kv # ------------------------------------------------------------------ @@ -90,7 +91,6 @@ def create_health_bp(kv=None, embedding_provider=None): res = query_audit(_get_kv(), {"operation": op, "limit": limit}) return jsonify({"entries": res, "success": True}), 200 - # ------------------------------------------------------------------ # GET /agentcache/config/flags # ------------------------------------------------------------------ diff --git a/src/agentcache/routes/mcp.py b/src/agentcache/routes/mcp.py index 082eb4af42dc2469c266658a7661737654726d7d..12082a3cdce162e66fb532d059f28c53e14f8e0f 100644 --- a/src/agentcache/routes/mcp.py +++ b/src/agentcache/routes/mcp.py @@ -15,7 +15,6 @@ from flask import Blueprint, jsonify, request from .. import legacy from ..core import KV - mcp_bp = Blueprint("mcp", __name__) @@ -52,7 +51,6 @@ def _get_observation_store(): return app_module.observation_store - def _datetime_now_iso() -> str: return ( datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") @@ -401,7 +399,11 @@ def mcp_tools_call(): search_svc = _get_search_service() if search_svc is not None: res = search_svc.search( - query=q, limit=limit, folder_path=folder_path, agent_id=agent_id, kv=kv + query=q, + limit=limit, + folder_path=folder_path, + agent_id=agent_id, + kv=kv, ) else: res = [] @@ -440,7 +442,11 @@ def mcp_tools_call(): search_svc = _get_search_service() if search_svc is not None: res = search_svc.search( - query=q, limit=limit, folder_path=folder_path, agent_id=agent_id, kv=kv + query=q, + limit=limit, + folder_path=folder_path, + agent_id=agent_id, + kv=kv, ) else: res = [] @@ -475,7 +481,6 @@ def mcp_tools_call(): res = {"success": False, "deleted": 0} text_out = json.dumps(res, indent=2) - elif name in ("cache_export", "memory_export"): res = legacy.export_data(kv, {}) text_out = json.dumps(res, indent=2) diff --git a/src/agentcache/routes/memories.py b/src/agentcache/routes/memories.py index 4331129030e96f7a422f3aaa073c2eded0922a4f..775e5933f1971397aec1028de3b426450e1181ea 100644 --- a/src/agentcache/routes/memories.py +++ b/src/agentcache/routes/memories.py @@ -23,6 +23,7 @@ def create_memories_bp(kv=None): if kv is not None: return kv from .. import app as app_module + return app_module.kv # ------------------------------------------------------------------ @@ -40,9 +41,6 @@ def create_memories_bp(kv=None): except Exception as e: return jsonify({"error": str(e)}), 400 - - - # ------------------------------------------------------------------ # POST /agentmemory/agent/remember # ------------------------------------------------------------------ @@ -113,7 +111,6 @@ def create_memories_bp(kv=None): res = functions.forget(_get_kv(), body) return jsonify(res), 200 except Exception as e: - return jsonify({"error": str(e)}), 400 return bp diff --git a/src/agentcache/routes/migration.py b/src/agentcache/routes/migration.py index 04d021fd4eb65e1cbb06f82da7410831b9d70aa0..d5d696f7607bffcb3aa3c13f01a44a04f2d8dabb 100644 --- a/src/agentcache/routes/migration.py +++ b/src/agentcache/routes/migration.py @@ -9,7 +9,7 @@ import os from flask import Blueprint, jsonify, request -from .. import functions +from .. import legacy as functions migration_bp = Blueprint("migration", __name__) diff --git a/src/agentcache/routes/observations.py b/src/agentcache/routes/observations.py index 36eda72d841346bc8961700b16b070faaccb185a..fdeb27df5c981d446a24e916a3e8970453562800 100644 --- a/src/agentcache/routes/observations.py +++ b/src/agentcache/routes/observations.py @@ -2,21 +2,21 @@ Observation routes blueprint. Handles: - POST /agentmemory/observe - POST /agentmemory/agent/observe - GET /agentmemory/folder/observations - GET /agentmemory/folders + POST /agentcache/observe + POST /agentcache/agent/observe + GET /agentcache/folder/observations + GET /agentcache/folders + POST /agentcache/folder/dedup """ import datetime import os +from typing import Optional from flask import Blueprint, jsonify, request -from .. import functions -from ..functions import KV - -observations_bp = Blueprint("observations", __name__) +from ..core.kv_scopes import KV +from ..core.observation_store import ObservationStore def _datetime_now_iso() -> str: @@ -41,271 +41,255 @@ def _check_auth(): return None -def _get_kv(): - """Retrieve the shared kv instance from the app module.""" - from .. import app as app_module - - return app_module.kv - - -# --------------------------------------------------------------------------- -# POST /agentmemory/observe (legacy raw hook endpoint + auto-compat shim) -# --------------------------------------------------------------------------- - - -@observations_bp.route("/agentcache/observe", methods=["POST"]) -@observations_bp.route("/agentmemory/observe", methods=["POST"]) -def api_observe(): - auth_err = _check_auth() - if auth_err: - return auth_err - - body = {} - try: - body = request.get_json(force=True) or {} - folder_path = body.get("folderPath") - agent_id = body.get("agentId") - text = body.get("text") or body.get("content") or "" - - if not folder_path or not agent_id or not text: - return jsonify({"error": "folderPath, agentId, and text are required"}), 400 - - payload = { - "folderPath": folder_path, - "agentId": agent_id, - "text": text, - "timestamp": body.get("timestamp") or _datetime_now_iso(), - "type": body.get("type"), - "title": body.get("title"), - "concepts": body.get("concepts"), - "files": body.get("files"), - "importance": body.get("importance"), - } - res = functions.folder_observe(_get_kv(), payload) - return jsonify(res), 201 - except Exception as e: - import traceback - - tb = traceback.format_exc() - print(f"[observe] 400 — keys={list(body.keys())} {type(e).__name__}: {e}\n{tb}") - return jsonify( - { - "error": str(e), - "detail": type(e).__name__, - "keys": list(body.keys()), - "tb": tb, +def create_observations_bp( + observation_store: Optional[ObservationStore] = None, +) -> Blueprint: + bp = Blueprint("observations", __name__) + + def get_store() -> ObservationStore: + if observation_store is not None: + return observation_store + from flask import current_app + + store = current_app.extensions.get("observation_store") + if store is None: + from .. import app as app_module + + store = getattr(app_module, "observation_store", None) + if store is None: + raise RuntimeError("ObservationStore is not initialized") + return store + + def get_kv(): + return get_store().kv + + @bp.route("/agentcache/observe", methods=["POST"]) + @bp.route("/agentmemory/observe", methods=["POST"]) + def api_observe(): + auth_err = _check_auth() + if auth_err: + return auth_err + + body = {} + try: + body = request.get_json(force=True) or {} + folder_path = body.get("folderPath") + agent_id = body.get("agentId") + text = body.get("text") or body.get("content") or "" + + if not folder_path or not agent_id or not text: + return ( + jsonify({"error": "folderPath, agentId, and text are required"}), + 400, + ) + + payload = { + "folderPath": folder_path, + "agentId": agent_id, + "text": text, + "timestamp": body.get("timestamp") or _datetime_now_iso(), + "type": body.get("type"), + "title": body.get("title"), + "concepts": body.get("concepts"), + "files": body.get("files"), + "importance": body.get("importance"), + } + res = get_store().ingest(payload) + return jsonify(res), 201 + except Exception as e: + import traceback + + tb = traceback.format_exc() + print( + f"[observe] 400 — keys={list(body.keys())} {type(e).__name__}: {e}\n{tb}" + ) + return ( + jsonify( + { + "error": str(e), + "detail": type(e).__name__, + "keys": list(body.keys()), + "tb": tb, + } + ), + 400, + ) + + @bp.route("/agentcache/agent/observe", methods=["POST"]) + @bp.route("/agentmemory/agent/observe", methods=["POST"]) + def api_agent_observe(): + auth_err = _check_auth() + if auth_err: + return auth_err + + try: + body = request.get_json(force=True) or {} + folder_path = body.get("folderPath") + agent_id = body.get("agentId") + text = body.get("text") or body.get("content") or "" + + if not folder_path or not agent_id or not text: + return ( + jsonify({"error": "folderPath, agentId, and text are required"}), + 400, + ) + + timestamp = body.get("timestamp") or _datetime_now_iso() + + payload = { + "folderPath": folder_path, + "agentId": agent_id, + "text": text, + "timestamp": timestamp, + "type": body.get("type"), + "title": body.get("title"), + "concepts": body.get("concepts"), + "files": body.get("files"), + "importance": body.get("importance"), } - ), 400 - -# --------------------------------------------------------------------------- -# POST /agentmemory/agent/observe -# --------------------------------------------------------------------------- + res = get_store().ingest(payload) + return jsonify(res), 201 + except ValueError as e: + print( + f"[agent_observe] 400 ValueError — body keys: {list(body.keys())} — {e}" + ) + return jsonify({"error": str(e)}), 400 + except Exception as e: + import traceback + + print( + f"[agent_observe] 400 error — body keys: {list(body.keys())} — {type(e).__name__}: {e}" + ) + print(traceback.format_exc()) + return jsonify({"error": str(e), "detail": type(e).__name__}), 400 + + @bp.route("/agentcache/folders", methods=["GET"]) + @bp.route("/agentmemory/folders", methods=["GET"]) + def api_folders(): + auth_err = _check_auth() + if auth_err: + return auth_err + from .. import legacy + + folders = sorted( + get_kv().list(KV.folders), + key=lambda x: x.get("lastUpdated", ""), + reverse=True, + ) + if legacy.is_agent_scope_isolated(): + aid = legacy.get_agent_id() + if aid: + folders = [f for f in folders if f.get("agentId") == aid] + return jsonify({"folders": folders}), 200 + + @bp.route("/agentcache/folder/observations", methods=["GET"]) + @bp.route("/agentmemory/folder/observations", methods=["GET"]) + def api_folder_observations(): + auth_err = _check_auth() + if auth_err: + return auth_err + fp = request.args.get("folderPath") + aid = request.args.get("agentId") + if not fp or not aid: + return jsonify({"error": "folderPath and agentId are required"}), 400 + from .. import legacy + + if legacy.is_agent_scope_isolated(): + current_aid = legacy.get_agent_id() + + if current_aid and aid != current_aid: + return ( + jsonify( + { + "error": "Unauthorized: Agent scope is isolated to another agent" + } + ), + 403, + ) + observations = sorted( + get_kv().list(KV.folder_obs(fp, aid)), + key=lambda x: x.get("timestamp", ""), + reverse=True, + ) + return ( + jsonify({"observations": observations, "folderPath": fp, "agentId": aid}), + 200, + ) + @bp.route("/agentcache/session/start", methods=["POST"]) + @bp.route("/agentmemory/session/start", methods=["POST"]) + def api_session_start(): + auth_err = _check_auth() + if auth_err: + return auth_err -@observations_bp.route("/agentcache/agent/observe", methods=["POST"]) -@observations_bp.route("/agentmemory/agent/observe", methods=["POST"]) -def api_agent_observe(): - auth_err = _check_auth() - if auth_err: - return auth_err + import uuid - try: body = request.get_json(force=True) or {} - folder_path = body.get("folderPath") - agent_id = body.get("agentId") - text = body.get("text") or body.get("content") or "" - - if not folder_path or not agent_id or not text: - return jsonify({"error": "folderPath, agentId, and text are required"}), 400 - - # sessionId accepted but ignored (folder-based model) - timestamp = body.get("timestamp") or _datetime_now_iso() - - payload = { - "folderPath": folder_path, - "agentId": agent_id, - "text": text, - "timestamp": timestamp, - "type": body.get("type"), - "title": body.get("title"), - "concepts": body.get("concepts"), - "files": body.get("files"), - "importance": body.get("importance"), - } - - res = functions.folder_observe(_get_kv(), payload) - return jsonify(res), 201 - except ValueError as e: - import traceback - - print(f"[agent_observe] 400 ValueError — body keys: {list(body.keys())} — {e}") - return jsonify({"error": str(e)}), 400 - except Exception as e: - import traceback - - print( - f"[agent_observe] 400 error — body keys: {list(body.keys())} — {type(e).__name__}: {e}" + session_id = body.get("sessionId") or f"compat_{uuid.uuid4().hex[:16]}" + return ( + jsonify( + { + "sessionId": session_id, + "status": "active", + "message": "Session model migrated to folder-based. Use /agentmemory/agent/observe.", + } + ), + 200, ) - print(traceback.format_exc()) - return jsonify({"error": str(e), "detail": type(e).__name__}), 400 - - -# --------------------------------------------------------------------------- -# GET /agentmemory/folders -# --------------------------------------------------------------------------- - -@observations_bp.route("/agentcache/folders", methods=["GET"]) -@observations_bp.route("/agentmemory/folders", methods=["GET"]) -def api_folders(): - auth_err = _check_auth() - if auth_err: - return auth_err - folders = sorted( - _get_kv().list(KV.folders), - key=lambda x: x.get("lastUpdated", ""), - reverse=True, - ) - if functions.is_agent_scope_isolated(): - aid = functions.get_agent_id() - if aid: - folders = [f for f in folders if f.get("agentId") == aid] - return jsonify({"folders": folders}), 200 - - -# --------------------------------------------------------------------------- -# GET /agentmemory/folder/observations -# --------------------------------------------------------------------------- - - -@observations_bp.route("/agentcache/folder/observations", methods=["GET"]) -@observations_bp.route("/agentmemory/folder/observations", methods=["GET"]) -def api_folder_observations(): - auth_err = _check_auth() - if auth_err: - return auth_err - fp = request.args.get("folderPath") - aid = request.args.get("agentId") - if not fp or not aid: - return jsonify({"error": "folderPath and agentId are required"}), 400 - if functions.is_agent_scope_isolated(): - current_aid = functions.get_agent_id() - if current_aid and aid != current_aid: - return jsonify( - {"error": "Unauthorized: Agent scope is isolated to another agent"} - ), 403 - observations = sorted( - _get_kv().list(KV.folder_obs(fp, aid)), - key=lambda x: x.get("timestamp", ""), - reverse=True, - ) - return jsonify( - {"observations": observations, "folderPath": fp, "agentId": aid} - ), 200 - - -# --------------------------------------------------------------------------- -# POST /agentmemory/session/start (legacy compat shim → 200 no-op) -# --------------------------------------------------------------------------- - - -@observations_bp.route("/agentcache/session/start", methods=["POST"]) -@observations_bp.route("/agentmemory/session/start", methods=["POST"]) -def api_session_start(): - """Legacy session/start — clients in the wild still call this. - Return a synthetic session ID so callers don't error out. - """ - auth_err = _check_auth() - if auth_err: - return auth_err - - import uuid - - body = request.get_json(force=True) or {} - session_id = body.get("sessionId") or f"compat_{uuid.uuid4().hex[:16]}" - return jsonify( - { - "sessionId": session_id, - "status": "active", - "message": "Session model migrated to folder-based. Use /agentmemory/agent/observe.", - } - ), 200 - - -# --------------------------------------------------------------------------- -# POST /agentmemory/session/end (legacy compat shim → 200 no-op) -# --------------------------------------------------------------------------- - - -@observations_bp.route("/agentcache/session/end", methods=["POST"]) -@observations_bp.route("/agentmemory/session/end", methods=["POST"]) -def api_session_end(): - auth_err = _check_auth() - if auth_err: - return auth_err - return jsonify( - {"success": True, "message": "Session model is now folder-based."} - ), 200 - - -# --------------------------------------------------------------------------- -# GET /agentmemory/observations (legacy compat shim) -# --------------------------------------------------------------------------- - - -@observations_bp.route("/agentcache/folder/dedup", methods=["POST"]) -@observations_bp.route("/agentmemory/folder/dedup", methods=["POST"]) -def api_folder_dedup(): - """POST /agentmemory/folder/dedup — remove duplicate observations. - - Body (both optional): - folderPath: str — deduplicate only this folder pair - agentId: str — deduplicate only this agent - - If both are omitted all folder pairs are processed. - Returns: {"success": bool, "deduplicated": int, "pairs_processed": int, "kept": int} - """ - auth_err = _check_auth() - if auth_err: - return auth_err - try: - body = request.get_json(force=True) or {} - folder_path = body.get("folderPath") or None - agent_id = body.get("agentId") or None - res = functions.dedup_folder_observations(_get_kv(), folder_path, agent_id) - return jsonify(res), 200 - except Exception as e: - return jsonify({"error": str(e)}), 400 - - -# --------------------------------------------------------------------------- -# GET /agentmemory/observations (legacy compat shim) -# --------------------------------------------------------------------------- - - -@observations_bp.route("/agentcache/observations", methods=["GET"]) -@observations_bp.route("/agentmemory/observations", methods=["GET"]) -def api_observations_legacy(): - """Legacy /observations?sessionId=... shim. - Reads from legacy KV scope if data exists, otherwise returns empty list. - """ - auth_err = _check_auth() - if auth_err: - return auth_err - - session_id = request.args.get("sessionId", "") - if not session_id: - return jsonify({"observations": [], "sessionId": ""}), 200 - - try: - obs = sorted( - _get_kv().list(functions.KV.observations(session_id)), - key=lambda x: x.get("timestamp", ""), - reverse=True, + @bp.route("/agentcache/session/end", methods=["POST"]) + @bp.route("/agentmemory/session/end", methods=["POST"]) + def api_session_end(): + auth_err = _check_auth() + if auth_err: + return auth_err + return ( + jsonify({"success": True, "message": "Session model is now folder-based."}), + 200, ) - return jsonify({"observations": obs, "sessionId": session_id}), 200 - except Exception as e: - return jsonify( - {"observations": [], "sessionId": session_id, "error": str(e)} - ), 200 + + @bp.route("/agentcache/folder/dedup", methods=["POST"]) + @bp.route("/agentmemory/folder/dedup", methods=["POST"]) + def api_folder_dedup(): + auth_err = _check_auth() + if auth_err: + return auth_err + try: + body = request.get_json(force=True) or {} + folder_path = body.get("folderPath") or None + agent_id = body.get("agentId") or None + res = get_store().dedup(folder_path, agent_id) + return jsonify(res), 200 + except Exception as e: + return jsonify({"error": str(e)}), 400 + + @bp.route("/agentcache/observations", methods=["GET"]) + @bp.route("/agentmemory/observations", methods=["GET"]) + def api_observations_legacy(): + auth_err = _check_auth() + if auth_err: + return auth_err + + session_id = request.args.get("sessionId", "") + if not session_id: + return jsonify({"observations": [], "sessionId": ""}), 200 + + try: + obs = sorted( + get_kv().list(KV.observations(session_id)), + key=lambda x: x.get("timestamp", ""), + reverse=True, + ) + return jsonify({"observations": obs, "sessionId": session_id}), 200 + except Exception as e: + return ( + jsonify({"observations": [], "sessionId": session_id, "error": str(e)}), + 200, + ) + + return bp + + +observations_bp = create_observations_bp() diff --git a/src/agentcache/routes/search.py b/src/agentcache/routes/search.py index d0d5ddbd58864f32da86f8bd870be53cca0af967..89fd1624e9512e5dcc64020822840058abee9368 100644 --- a/src/agentcache/routes/search.py +++ b/src/agentcache/routes/search.py @@ -10,8 +10,6 @@ import os from flask import Blueprint, jsonify, request -from .. import functions - search_bp = Blueprint("search", __name__) @@ -36,6 +34,18 @@ def _get_kv(): return app_module.kv +def _get_search_service(): + from .. import app as app_module + + return app_module.search_service + + +def _get_observation_store(): + from .. import app as app_module + + return app_module.observation_store + + # --------------------------------------------------------------------------- # POST /agentcache/search # --------------------------------------------------------------------------- @@ -57,9 +67,17 @@ def api_search(): folder_path = body.get("folderPath") agent_id = body.get("agentId") - res = functions.folder_search( - _get_kv(), query, limit, folder_path=folder_path, agent_id=agent_id - ) + search_svc = _get_search_service() + if search_svc is not None: + res = search_svc.search( + query=query, + limit=limit, + folder_path=folder_path, + agent_id=agent_id, + kv=_get_kv(), + ) + else: + res = [] return jsonify(res), 200 except Exception as e: return jsonify({"error": str(e)}), 400 @@ -84,9 +102,17 @@ def api_timeline(): limit = body.get("limit") or 100 before = body.get("before") after = body.get("after") - result = functions.folder_timeline( - _get_kv(), limit, folder_path, agent_id, before, after - ) + obs_store = _get_observation_store() + if obs_store is not None: + result = obs_store.timeline( + limit=limit, + folder_path=folder_path, + agent_id=agent_id, + before=before, + after=after, + ) + else: + result = [] return jsonify({"observations": result}), 200 except Exception as e: return jsonify({"error": str(e)}), 400 diff --git a/src/agentcache/search.py b/src/agentcache/search.py index fa3ec24c23f22d00c6e027388e361f05862c90bf..aa6638380da46a7a7f6d152ab03e4406321b871c 100644 --- a/src/agentcache/search.py +++ b/src/agentcache/search.py @@ -522,6 +522,7 @@ class SearchIndex: parts = [ obs.get("title", ""), obs.get("subtitle", "") or "", + obs.get("text", "") or "", obs.get("narrative", "") or "", " ".join(obs.get("facts", []) or []), " ".join(obs.get("concepts", []) or []), diff --git a/src/agentcache/storage/scopes.py b/src/agentcache/storage/scopes.py index 6c9ae0bd779ef2ba77b64e34cafc07cd74090a76..aa111e11ea5eab217df2ed259030fbc61dc42de8 100644 --- a/src/agentcache/storage/scopes.py +++ b/src/agentcache/storage/scopes.py @@ -1,78 +1,9 @@ """ src/storage/scopes.py — KV scope registry (A2.3). -Copied from src/functions.py — do NOT delete the original (backward compat). -The KV class defines all storage scope keys used across agentcache-python. +Re-exports KV from core.kv_scopes for backward compatibility. """ +from ..core.kv_scopes import KV -class KV: - # ---- Folder memory scopes (new) ---- - - # Global index of all (folder_path, agent_id) pairs known to the system. - # Key = "{safe_folder_path}:{agent_id}", value = FolderIndexEntry dict. - folders = "mem:folders" - - @staticmethod - def folder_obs(folder_path: str, agent_id: str) -> str: - """Per-(folder, agent) observations scope. - Key = obs_id, value = FolderObservation dict. - """ - safe_path = folder_path.replace("\\", "/").strip("/") - safe_agent = agent_id.strip() - return f"mem:folder:{safe_path}:{safe_agent}" - - @staticmethod - def folder_meta(folder_path: str, agent_id: str) -> str: - """Per-(folder, agent) metadata scope. - Key = "meta", value = FolderMeta dict (obsCount, lastUpdated, summary). - """ - safe_path = folder_path.replace("\\", "/").strip("/") - safe_agent = agent_id.strip() - return f"mem:foldermeta:{safe_path}:{safe_agent}" - - @staticmethod - def obs_dedup(folder_path: str, agent_id: str) -> str: - """Deduplication index scope for (folder, agent) pairs. - Key = SHA-256 fingerprint hex of normalized text. - Value = {"obsId": str, "timestamp": str} - """ - safe_path = folder_path.replace("\\", "/").strip("/") - safe_agent = agent_id.strip() - return f"mem:obs_dedup:{safe_path}:{safe_agent}" - - # ---- Global / shared scopes (kept) ---- - - # Long-term memories — unchanged from previous implementation. - memories = "mem:memories" - - # BM25 index shards — unchanged. - bm25Index = "mem:index:bm25" - - # Audit log — unchanged. - audit = "mem:audit" - - # Graph edges — repurposed for folder graph edges. - relations = "mem:relations" - - # ---- Legacy scopes (read-only; kept for migration and backward compat) ---- - - # Legacy session store — read by migrate_sessions_to_folders() and legacy observe(). - sessions = "mem:sessions" - - @staticmethod - def observations(session_id: str) -> str: - """Legacy per-session observations scope. - Key = obs_id, value = raw/synthetic observation dict. - Read by migrate_sessions_to_folders() and legacy observe(). - """ - return f"mem:obs:{session_id}" - - # Legacy summary / profile / slot / image-ref scopes retained for legacy code paths. - summaries = "mem:summaries" - profiles = "mem:profiles" - slots = "mem:slots" - imageRefs = "mem:image-refs" - - # Global (cross-project) pinned slots. - globalSlots = "mem:global-slots" +__all__ = ["KV"] diff --git a/src/agentcache/workers.py b/src/agentcache/workers.py index 2bbb39494d484440a205e044a9bf426b95a04403..64e0c24922f346dcedb0033bcba5a218599ffb34 100644 --- a/src/agentcache/workers.py +++ b/src/agentcache/workers.py @@ -14,14 +14,14 @@ import sys import threading import time -from . import functions +from . import legacy # Module-level shutdown flag — set by signal handlers _shutting_down = threading.Event() -# Reference to the persistence object (set by start_background_workers) _persistence_ref = None -# Reference to the kv object (set by start_background_workers) +_search_svc_ref = None +_obs_store_ref = None _kv_ref = None @@ -30,7 +30,7 @@ def _shutdown_handler(signum, frame) -> None: # noqa: ARG001 Steps: 1. Set the global _shutting_down flag to stop background loops. - 2. Flush the debounce timer and save the index synchronously. + 2. Flush the debounce timer and save the index synchronously via SearchService.flush_persist(). 3. Run a WAL checkpoint via StateKV.teardown(). 4. Exit cleanly with code 0. """ @@ -39,9 +39,15 @@ def _shutdown_handler(signum, frame) -> None: # noqa: ARG001 _shutting_down.set() - # Flush in-flight persistence debounce timer and save immediately - global _persistence_ref - if _persistence_ref is not None: + global _search_svc_ref, _persistence_ref + if _search_svc_ref is not None: + try: + print("[workers] Flushing SearchService persistence...") + _search_svc_ref.flush_persist() + print("[workers] SearchService persistence flushed.") + except Exception as e: + print(f"[workers] Error flushing SearchService: {e}") + elif _persistence_ref is not None: try: print("[workers] Flushing index persistence...") _persistence_ref.flush() @@ -89,7 +95,7 @@ def _auto_forget_loop(kv) -> None: if kv.acquire_lock("auto_forget", lease_seconds=300): try: print("[scheduler] Running auto_forget sweep...") - res = functions.auto_forget(kv, dry_run=False) + res = legacy.auto_forget(kv, dry_run=False) print(f"[scheduler] auto_forget sweep completed: {res}") finally: kv.release_lock("auto_forget") @@ -107,7 +113,13 @@ def _rebuild_index(kv) -> None: try: if kv.acquire_lock("index_rebuild", lease_seconds=600): try: - count = functions.rebuild_index(kv) + from . import app as app_module + + obs_store = getattr(app_module, "observation_store", None) + if obs_store is not None: + count = obs_store.rebuild_index() + else: + count = 0 print(f"[persistence] Rebuild completed: indexed {count} items.") finally: kv.release_lock("index_rebuild") @@ -127,21 +139,34 @@ def start_background_workers(kv, tasks=None) -> None: kv: Initialised StateKV instance. tasks: Optional list of tasks to run ("index", "forget"). Defaults to running both. """ - global _kv_ref, _persistence_ref + global _kv_ref, _persistence_ref, _search_svc_ref, _obs_store_ref _kv_ref = kv - # Capture the persistence reference for the shutdown handler - _persistence_ref = functions._index_persistence + from . import app as app_module + + search_svc = getattr(app_module, "search_service", None) + obs_store = getattr(app_module, "observation_store", None) + + _search_svc_ref = search_svc + _obs_store_ref = obs_store + + if search_svc is not None: + _persistence_ref = search_svc._persistence + else: + _persistence_ref = None # Register graceful shutdown signal handlers (C5.1) _register_signal_handlers() if tasks is None or "index" in tasks: # Rebuild search index if empty or out of sync (Step 5) - index_empty = functions._bm25_index.size == 0 + bm25_size = search_svc.bm25_size if search_svc is not None else 0 + index_empty = bm25_size == 0 index_in_sync = True if not index_empty: - index_in_sync = functions.verify_index_sync_on_boot(kv) + index_in_sync = legacy.verify_index_sync_on_boot( + kv, search_service=search_svc + ) if index_empty or not index_in_sync: reason = "empty" if index_empty else "out of sync" diff --git a/tests/__init__.py b/tests/__init__.py index 33a488174919151b1b4c43d5cc962c2d58f2d3ec..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 100644 --- a/tests/__init__.py +++ b/tests/__init__.py @@ -1,2 +0,0 @@ -# tests package - diff --git a/tests/conftest.py b/tests/conftest.py index 8c5331804e818870363350f07b13d18a0ea8095b..7f0ca03a1256006b4fe623fdc8cfbacf0b56e315 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -3,6 +3,7 @@ Shared pytest fixtures for agentcache test suite. """ import pytest + import agentcache.app as app_mod from agentcache.app import create_app from agentcache.db import StateKV diff --git a/tests/test_api.py b/tests/test_api.py deleted file mode 100644 index 93ef05365a46296e3ef6e3ddd821855dabaa01cd..0000000000000000000000000000000000000000 --- a/tests/test_api.py +++ /dev/null @@ -1,294 +0,0 @@ -""" -tests/test_api.py — C3.1 - -Integration tests for REST endpoints using the Flask test client. -""" - -import datetime -import json -import os - -import pytest - -# --------------------------------------------------------------------------- -# Fixtures -# --------------------------------------------------------------------------- - - -@pytest.fixture(scope="module") -def flask_app(tmp_path_factory): - tmp_dir = tmp_path_factory.mktemp("api_test_db") - db_path = str(tmp_dir / "test.db") - os.environ.pop("AGENTCACHE_SECRET", None) - os.environ.pop("AGENTMEMORY_SECRET", None) - - from agentcache.db import StateKV - - original_init = StateKV.__init__ - - def patched_init(self, db_path_arg=None, **kwargs): - original_init(self, db_path=db_path, **kwargs) - - StateKV.__init__ = patched_init - import agentcache.app as app_module - - os.environ.pop("AGENTCACHE_SECRET", None) - os.environ.pop("AGENTMEMORY_SECRET", None) - flask_application = app_module.create_app() - StateKV.__init__ = original_init - flask_application.config["TESTING"] = True - return flask_application - - -@pytest.fixture(scope="module") -def client(flask_app): - return flask_app.test_client() - - -def _now(): - return ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - - -def _post(client, url, payload): - return client.post(url, data=json.dumps(payload), content_type="application/json") - - -# --------------------------------------------------------------------------- -# POST /agentcache/agent/observe -# --------------------------------------------------------------------------- - - -class TestAgentObserve: - def test_valid_payload_returns_201(self, client): - resp = _post( - client, - "/agentcache/agent/observe", - { - "folderPath": "/home/user/test-project", - "agentId": "kiro", - "text": "Implemented new authentication middleware", - "timestamp": _now(), - }, - ) - assert resp.status_code == 201 - data = resp.get_json() - assert "observationId" in data - assert data["observationId"].startswith("fobs_") - - def test_missing_folder_path_returns_400(self, client): - resp = _post( - client, - "/agentcache/agent/observe", - { - "agentId": "kiro", - "text": "Some work", - "timestamp": _now(), - }, - ) - assert resp.status_code == 400 - - def test_missing_agent_id_returns_400(self, client): - resp = _post( - client, - "/agentcache/agent/observe", - { - "folderPath": "/home/user/proj", - "text": "Some work", - "timestamp": _now(), - }, - ) - assert resp.status_code == 400 - - def test_missing_text_returns_400(self, client): - resp = _post( - client, - "/agentcache/agent/observe", - { - "folderPath": "/home/user/proj", - "agentId": "kiro", - "timestamp": _now(), - }, - ) - assert resp.status_code == 400 - - -# --------------------------------------------------------------------------- -# POST /agentcache/search -# --------------------------------------------------------------------------- - - -class TestSearch: - def test_search_with_query_returns_200(self, client): - # Seed data first - _post( - client, - "/agentcache/agent/observe", - { - "folderPath": "/home/user/search-proj", - "agentId": "kiro", - "text": "Refactored the authentication system", - "timestamp": _now(), - }, - ) - resp = _post(client, "/agentcache/search", {"query": "authentication"}) - assert resp.status_code == 200 - data = resp.get_json() - assert isinstance(data, list) or isinstance(data, dict) - - def test_search_missing_query_returns_400(self, client): - resp = _post(client, "/agentcache/search", {}) - assert resp.status_code == 400 - - def test_search_empty_query_returns_400(self, client): - resp = _post(client, "/agentcache/search", {"query": " "}) - assert resp.status_code == 400 - - -# --------------------------------------------------------------------------- -# GET /agentcache/folders -# --------------------------------------------------------------------------- - - -class TestFolders: - def test_get_folders_returns_200(self, client): - # Ensure at least one folder exists from earlier tests - _post( - client, - "/agentcache/agent/observe", - { - "folderPath": "/home/user/folders-check", - "agentId": "kiro", - "text": "Check folders endpoint", - "timestamp": _now(), - }, - ) - resp = client.get("/agentcache/folders") - assert resp.status_code == 200 - data = resp.get_json() - assert "folders" in data - assert isinstance(data["folders"], list) - - -# --------------------------------------------------------------------------- -# GET /agentcache/health -# --------------------------------------------------------------------------- - - -class TestHealth: - def test_health_returns_200(self, client): - resp = client.get("/agentcache/health") - assert resp.status_code == 200 - data = resp.get_json() - assert "folderCount" in data - assert "observationCount" in data - assert "memoryCount" in data - - def test_health_status_ok(self, client): - resp = client.get("/agentcache/health") - data = resp.get_json() - assert data.get("status") in ("ok", "degraded") - - -# --------------------------------------------------------------------------- -# GET /agentcache/livez -# --------------------------------------------------------------------------- - - -class TestLivez: - def test_livez_returns_200_no_auth(self, client): - resp = client.get("/agentcache/livez") - assert resp.status_code == 200 - data = resp.get_json() - assert data["status"] == "ok" - - def test_livez_open_with_secret_set(self, client): - os.environ["AGENTCACHE_SECRET"] = "test-secret-123" - try: - resp = client.get("/agentcache/livez") - assert resp.status_code == 200 - finally: - del os.environ["AGENTCACHE_SECRET"] - - -# --------------------------------------------------------------------------- -# Authentication -# --------------------------------------------------------------------------- - - -class TestAuthentication: - def test_protected_endpoint_returns_401_with_wrong_token(self, client): - os.environ["AGENTCACHE_SECRET"] = "correct-secret" - try: - resp = client.get( - "/agentcache/audit", - headers={"Authorization": "Bearer wrong-token"}, - ) - assert resp.status_code == 401 - finally: - del os.environ["AGENTCACHE_SECRET"] - - def test_protected_endpoint_passes_with_correct_token(self, client): - secret = "my-test-secret-xyz" - os.environ["AGENTCACHE_SECRET"] = secret - try: - resp = client.get( - "/agentcache/audit", - headers={"Authorization": f"Bearer {secret}"}, - ) - assert resp.status_code == 200 - finally: - del os.environ["AGENTCACHE_SECRET"] - - def test_livez_always_open_regardless_of_secret(self, client): - os.environ["AGENTCACHE_SECRET"] = "any-secret" - try: - resp = client.get("/agentcache/livez") - assert resp.status_code == 200 - finally: - del os.environ["AGENTCACHE_SECRET"] - - -# --------------------------------------------------------------------------- -# Additional endpoint smoke tests -# --------------------------------------------------------------------------- - - -class TestMemoriesEndpoint: - def test_memories_list_returns_200(self, client): - resp = client.get("/agentcache/memories") - assert resp.status_code == 200 - data = resp.get_json() - assert "memories" in data - - def test_remember_valid_payload_returns_201(self, client): - resp = _post( - client, - "/agentcache/remember", - { - "content": "API test memory content", - "type": "fact", - }, - ) - assert resp.status_code == 201 - - def test_forget_nonexistent_id(self, client): - resp = _post(client, "/agentcache/forget", {"memoryId": "mem_nonexistent"}) - assert resp.status_code == 200 - - def test_graph_endpoint_returns_200(self, client): - resp = client.get("/agentcache/graph") - assert resp.status_code == 200 - data = resp.get_json() - assert "nodes" in data - assert "edges" in data - - def test_mcp_tools_list_returns_200(self, client): - resp = client.get("/agentcache/mcp/tools") - assert resp.status_code == 200 - data = resp.get_json() - assert "tools" in data - tool_names = {t["name"] for t in data["tools"]} - assert "agent_observe" in tool_names - assert "cache_recall" in tool_names diff --git a/tests/test_auth.py b/tests/test_auth.py index 712c3c19ca79efee2dd3a9158997472b486a72d4..b46b63dfd0ef520206ecbeb0b452dc68a707a98c 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -3,8 +3,8 @@ Unit and integration tests for authentication and authorization. """ from flask import Flask, jsonify -from agentcache.routes.auth import require_auth, verify_token +from agentcache.routes.auth import require_auth, verify_token # ------------------------------------------------------------------------------ # Unit Tests — verify_token & require_auth @@ -85,7 +85,11 @@ def test_protected_routes_require_auth(authed_client): headers = {"Authorization": f"Bearer {secret}"} protected_endpoints = [ - ("POST", "/agentcache/observe", {"folderPath": "src/test", "agentId": "a1", "text": "test"}), + ( + "POST", + "/agentcache/observe", + {"folderPath": "src/test", "agentId": "a1", "text": "test"}, + ), ("POST", "/agentcache/remember", {"content": "test memory"}), ("POST", "/agentcache/search", {"query": "test"}), ("POST", "/agentcache/timeline", {}), @@ -104,8 +108,12 @@ def test_protected_routes_require_auth(authed_client): res_unauth = client.get(path) res_auth = client.get(path, headers=headers) - assert res_unauth.status_code == 401, f"{method} {path} should require auth (got {res_unauth.status_code})" - assert res_auth.status_code in (200, 201), f"{method} {path} failed with valid auth (got {res_auth.status_code})" + assert res_unauth.status_code == 401, ( + f"{method} {path} should require auth (got {res_unauth.status_code})" + ) + assert res_auth.status_code in (200, 201), ( + f"{method} {path} failed with valid auth (got {res_auth.status_code})" + ) def test_unprotected_routes_accessible_without_auth(authed_client): @@ -120,7 +128,9 @@ def test_unprotected_routes_accessible_without_auth(authed_client): for path in unprotected_paths: res = client.get(path) - assert res.status_code == 200, f"Unprotected route {path} failed (got {res.status_code})" + assert res.status_code == 200, ( + f"Unprotected route {path} failed (got {res.status_code})" + ) def test_wrong_token_on_any_blueprint_returns_401(authed_client): @@ -129,7 +139,11 @@ def test_wrong_token_on_any_blueprint_returns_401(authed_client): bad_headers = {"Authorization": "Bearer wrong-token-value"} protected_endpoints = [ - ("POST", "/agentcache/observe", {"folderPath": "src/test", "agentId": "a1", "text": "test"}), + ( + "POST", + "/agentcache/observe", + {"folderPath": "src/test", "agentId": "a1", "text": "test"}, + ), ("POST", "/agentcache/remember", {"content": "test memory"}), ("POST", "/agentcache/search", {"query": "test"}), ("POST", "/agentcache/timeline", {}), @@ -139,7 +153,6 @@ def test_wrong_token_on_any_blueprint_returns_401(authed_client): ("POST", "/agentcache/migrate", {}), ] - for method, path, payload in protected_endpoints: if method == "POST": res = client.post(path, json=payload or {}, headers=bad_headers) diff --git a/tests/test_auto_forget.py b/tests/test_auto_forget.py deleted file mode 100644 index 4096e8105d6cf2d6aaed0b0d71d62264c112568e..0000000000000000000000000000000000000000 --- a/tests/test_auto_forget.py +++ /dev/null @@ -1,139 +0,0 @@ -"""Unit tests for auto_forget() folder-based and memory-based eviction.""" - -import datetime -import os - -from agentcache.db import StateKV -from agentcache.functions import KV, auto_forget, folder_observe, remember - - -def make_kv(tmp_path): - db_path = os.path.join(str(tmp_path), "test.db") - return StateKV(db_path=db_path) - - -def test_auto_forget_memories(tmp_path): - kv = make_kv(tmp_path) - - # 1. Create a memory that expires in the past - past_time = ( - (datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=1)) - .isoformat() - .replace("+00:00", "Z") - ) - res1 = remember(kv, {"content": "Stale memory", "forgetAfter": past_time}) - mem1_id = res1["memory"]["id"] - - # 2. Create a memory that expires in the future - future_time = ( - (datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=5)) - .isoformat() - .replace("+00:00", "Z") - ) - res2 = remember(kv, {"content": "Fresh memory", "forgetAfter": future_time}) - mem2_id = res2["memory"]["id"] - - # Run auto_forget - results = auto_forget(kv, dry_run=False) - assert len(results["evictedMemories"]) == 1 - - # Verify mem1 is deleted, mem2 exists - assert kv.get(KV.memories, mem1_id) is None - assert kv.get(KV.memories, mem2_id) is not None - - -def test_auto_forget_expired_folder_observations(tmp_path): - kv = make_kv(tmp_path) - folder = "/home/user/myproject" - agent = "kiro" - - # 1. Create expired folder observation - past_time = ( - (datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=1)) - .isoformat() - .replace("+00:00", "Z") - ) - res1 = folder_observe( - kv, - { - "folderPath": folder, - "agentId": agent, - "text": "Stale observation", - "timestamp": past_time, - "forgetAfter": past_time, - }, - ) - obs1_id = res1["observationId"] - - # 2. Create fresh folder observation - future_time = ( - (datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=1)) - .isoformat() - .replace("+00:00", "Z") - ) - res2 = folder_observe( - kv, - { - "folderPath": folder, - "agentId": agent, - "text": "Fresh observation", - "timestamp": past_time, - "forgetAfter": future_time, - }, - ) - obs2_id = res2["observationId"] - - # Run auto_forget - results = auto_forget(kv, dry_run=False) - assert len(results["evictedObservations"]) == 1 - - # Verify eviction - fp = "home/user/myproject" - assert kv.get(KV.folder_obs(fp, agent), obs1_id) is None - assert kv.get(KV.folder_obs(fp, agent), obs2_id) is not None - - -def test_auto_forget_low_importance_stale_observations(tmp_path): - kv = make_kv(tmp_path) - folder = "/home/user/myproject" - agent = "kiro" - - # 1. Create old low-importance folder observation (importance = 1, 200 days old) - old_time = ( - (datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=200)) - .isoformat() - .replace("+00:00", "Z") - ) - res1 = folder_observe( - kv, - { - "folderPath": folder, - "agentId": agent, - "text": "Stale low value observation", - "timestamp": old_time, - "importance": 1, - }, - ) - obs1_id = res1["observationId"] - - # 2. Create old high-importance folder observation (importance = 8, 200 days old) - res2 = folder_observe( - kv, - { - "folderPath": folder, - "agentId": agent, - "text": "Stale high value observation", - "timestamp": old_time, - "importance": 8, - }, - ) - obs2_id = res2["observationId"] - - # Run auto_forget - results = auto_forget(kv, dry_run=False) - assert len(results["evictedObservations"]) == 1 - - # Verify eviction - fp = "home/user/myproject" - assert kv.get(KV.folder_obs(fp, agent), obs1_id) is None - assert kv.get(KV.folder_obs(fp, agent), obs2_id) is not None diff --git a/tests/test_cli_context.py b/tests/test_cli_context.py deleted file mode 100644 index a5beeab65a0a2a7c05f8a32c5f60c4be5100ceee..0000000000000000000000000000000000000000 --- a/tests/test_cli_context.py +++ /dev/null @@ -1,70 +0,0 @@ -"""Unit tests for the agentcache context CLI command.""" - -import argparse -import os -from unittest.mock import patch - -from agentcache.cli import cmd_context -from agentcache.db import StateKV -from agentcache.functions import folder_observe, remember - - -def make_kv(tmp_path): - db_path = os.path.join(str(tmp_path), "test.db") - return StateKV(db_path=db_path) - - -def test_cli_context_generation(tmp_path): - kv = make_kv(tmp_path) - - # 1. Add observations and memories - folder = "/home/user/myproject" - agent = "test-agent" - - folder_observe( - kv, - { - "folderPath": folder, - "agentId": agent, - "text": "First observation", - "timestamp": "2026-07-15T10:00:00Z", - }, - ) - - remember( - kv, {"content": "A crucial project rule", "type": "fact", "agentId": agent} - ) - - remember( - kv, - { - "content": "A project-wide memory", - "type": "architecture", - "project": "myproject", - "agentId": "some-other-agent", - }, - ) - - output_file = os.path.join(str(tmp_path), "context.md") - args = argparse.Namespace(agent=agent, output=output_file, watch=False) - - # Mock os.getcwd to match the project path and init_services to return our test db - with ( - patch("os.getcwd", return_value=folder), - patch("agentcache.app.init_services", return_value=(kv, None, None)), - ): - cmd_context(args) - - # 2. Verify file output - assert os.path.exists(output_file) - with open(output_file, "r", encoding="utf-8") as f: - content = f.read() - - # Check that metadata and values are written - assert "Agent Cache Context" in content - assert "Project Metadata" in content - assert "myproject" in content - assert "test-agent" in content - assert "First observation" in content - assert "A crucial project rule" in content - assert "A project-wide memory" in content diff --git a/tests/test_context.py b/tests/test_context.py deleted file mode 100644 index dc879813cea97fd4f6c3aae3b7fbedb6d7f2ea48..0000000000000000000000000000000000000000 --- a/tests/test_context.py +++ /dev/null @@ -1,204 +0,0 @@ -""" -tests/test_context.py — C1.4 - -Tests for context(), export_data(), and token budget enforcement. -""" - -import datetime -import os - -import pytest - -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - - -def _make_kv(tmp_path): - from agentcache.db import StateKV - - os.environ.pop("AGENTCACHE_SECRET", None) - return StateKV(db_path=str(tmp_path / "test.db")) - - -def _now(): - return ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - - -# --------------------------------------------------------------------------- -# context() tests -# --------------------------------------------------------------------------- - - -class TestContext: - def test_empty_db_returns_minimal_context(self, tmp_path): - """Empty DB should return a well-formed but empty context.""" - from agentcache.functions import context - - kv = _make_kv(tmp_path) - result = context( - kv, - { - "sessionId": "sess_test_empty", - "project": "/home/user/my-project", - "budget": 2000, - }, - ) - assert isinstance(result, dict) - assert "context" in result - assert "blocks" in result - assert "tokens" in result - assert result["blocks"] == 0 - assert result["tokens"] == 0 - - def test_raises_on_missing_session_id(self, tmp_path): - from agentcache.functions import context - - kv = _make_kv(tmp_path) - with pytest.raises(ValueError): - context(kv, {"project": "/home/user/proj"}) - - def test_raises_on_missing_project(self, tmp_path): - from agentcache.functions import context - - kv = _make_kv(tmp_path) - with pytest.raises(ValueError): - context(kv, {"sessionId": "sess_x"}) - - def test_respects_token_budget(self, tmp_path): - """Context output tokens should not exceed the requested budget.""" - from agentcache.functions import context, lesson_save - - kv = _make_kv(tmp_path) - project = "/home/user/budget-test" - - # Add many lessons to push towards the budget - for i in range(20): - lesson_save( - kv, - { - "content": f"Lesson {i}: " + ("x " * 100), - "project": project, - "confidence": 0.9, - }, - ) - - budget = 500 - result = context( - kv, - { - "sessionId": "sess_budget", - "project": project, - "budget": budget, - }, - ) - # Token estimate is len/3 — check that total tokens respects budget - assert result["tokens"] <= budget + 50 # small headroom for header/footer - - def test_context_includes_xml_wrapper(self, tmp_path): - """Non-empty context should be wrapped in .""" - from agentcache.functions import context, lesson_save - - kv = _make_kv(tmp_path) - project = "/home/user/xml-test" - - lesson_save( - kv, - { - "content": "Always validate user input before processing", - "project": project, - "confidence": 0.8, - }, - ) - - result = context( - kv, - { - "sessionId": "sess_xml", - "project": project, - "budget": 2000, - }, - ) - - if result["blocks"] > 0: - assert "" in result["context"] - - def test_token_budget_env_var_respected(self, tmp_path, monkeypatch): - """TOKEN_BUDGET env var should be used when no budget param given.""" - from agentcache.functions import context, lesson_save - - kv = _make_kv(tmp_path) - project = "/home/user/env-budget" - monkeypatch.setenv("TOKEN_BUDGET", "100") - - for i in range(10): - lesson_save( - kv, - { - "content": f"Important lesson {i}: " + ("word " * 50), - "project": project, - "confidence": 0.9, - }, - ) - - result = context(kv, {"sessionId": "sess_env_budget", "project": project}) - # Should use TOKEN_BUDGET=100 from env - assert result["tokens"] <= 150 # with some headroom for XML wrapper - - -# --------------------------------------------------------------------------- -# export_data() tests -# --------------------------------------------------------------------------- - - -class TestExportData: - def test_export_returns_folders_and_memories(self, tmp_path): - from agentcache.functions import export_data, folder_observe, remember - - kv = _make_kv(tmp_path) - - folder_observe( - kv, - { - "folderPath": "/home/user/export-test", - "agentId": "kiro", - "text": "Working on export feature", - "timestamp": _now(), - }, - ) - remember(kv, {"content": "Export data uses v2 format"}) - - result = export_data(kv, {}) - assert isinstance(result, dict) - assert "folders" in result or "observations" in result or "memories" in result - - def test_export_empty_db(self, tmp_path): - from agentcache.functions import export_data - - kv = _make_kv(tmp_path) - result = export_data(kv, {}) - assert isinstance(result, dict) - # Should not crash on empty DB - - -# --------------------------------------------------------------------------- -# estimate_tokens() -# --------------------------------------------------------------------------- - - -class TestEstimateTokens: - def test_empty_string(self): - from agentcache.functions import estimate_tokens - - assert estimate_tokens("") == 0 - - def test_typical_text(self): - from agentcache.functions import estimate_tokens - - text = "hello world this is a test" * 10 - tokens = estimate_tokens(text) - # Should be approximately len/3 - assert tokens == len(text) // 3 diff --git a/tests/test_debounce.py b/tests/test_debounce.py deleted file mode 100644 index c9b3fddc2fd97955290e7bf4716eea37b77dd864..0000000000000000000000000000000000000000 --- a/tests/test_debounce.py +++ /dev/null @@ -1,166 +0,0 @@ -"""A4.3 — Unit tests for IndexPersistence debounce behavior. - -Tests: -- 100 rapid schedule_save() calls result in exactly 1 save() call. -- flush() triggers immediate save without waiting for debounce timer. -""" - -import time -import unittest.mock as mock - -from agentcache.db import StateKV -from agentcache.functions import IndexPersistence -from agentcache.search import SearchIndex, VectorIndex - -# Speed up debounce for tests -FAST_DEBOUNCE = 0.05 - - -def make_kv(tmp_path): - return StateKV(db_path=str(tmp_path / "test_debounce.db")) - - -class TestDebounce: - def test_100_rapid_calls_result_in_1_save(self, tmp_path): - """100 rapid schedule_save() calls must fire exactly 1 save().""" - kv = make_kv(tmp_path) - bm25 = SearchIndex() - vector = VectorIndex() - - persistence = IndexPersistence(kv, bm25, vector) - persistence.DEBOUNCE_SECONDS = FAST_DEBOUNCE - - save_call_count = [0] - - original_save = persistence.save - - def counting_save(): - save_call_count[0] += 1 - original_save() - - with mock.patch.object(persistence, "save", side_effect=counting_save): - for _ in range(100): - persistence.schedule_save() - # Wait for the debounce timer to fire (2× debounce window is plenty) - time.sleep(FAST_DEBOUNCE * 4) - - assert save_call_count[0] == 1, ( - f"Expected exactly 1 save() call; got {save_call_count[0]}" - ) - - def test_rapid_calls_with_dirty_bm25(self, tmp_path): - """schedule_save() fires exactly once even when BM25 is dirty.""" - kv = make_kv(tmp_path) - bm25 = SearchIndex() - vector = VectorIndex() - - # Add a doc so the index is dirty - bm25.add( - { - "id": "obs_test1", - "sessionId": "sess1", - "title": "hello world", - "facts": [], - "concepts": [], - "files": [], - "type": "other", - } - ) - - persistence = IndexPersistence(kv, bm25, vector) - persistence.DEBOUNCE_SECONDS = FAST_DEBOUNCE - - save_call_count = [0] - original_save = persistence.save - - def counting_save(): - save_call_count[0] += 1 - original_save() - - with mock.patch.object(persistence, "save", side_effect=counting_save): - for _ in range(100): - persistence.schedule_save() - time.sleep(FAST_DEBOUNCE * 4) - - assert save_call_count[0] == 1 - - def test_flush_triggers_immediate_save(self, tmp_path): - """flush() must call save() immediately without waiting for the debounce timer.""" - kv = make_kv(tmp_path) - bm25 = SearchIndex() - vector = VectorIndex() - - persistence = IndexPersistence(kv, bm25, vector) - persistence.DEBOUNCE_SECONDS = 60.0 # very long timer — flush must bypass it - - save_call_count = [0] - original_save = persistence.save - - def counting_save(): - save_call_count[0] += 1 - original_save() - - with mock.patch.object(persistence, "save", side_effect=counting_save): - persistence.schedule_save() - # Timer is set but hasn't fired yet (60s window) - assert save_call_count[0] == 0, "save() should not have been called yet" - - # flush() must cancel the timer and call save() synchronously - persistence.flush() - - assert save_call_count[0] == 1, ( - f"flush() should trigger exactly 1 save(); got {save_call_count[0]}" - ) - - def test_flush_after_no_pending_save_is_safe(self, tmp_path): - """flush() with no pending timer should still call save() once.""" - kv = make_kv(tmp_path) - bm25 = SearchIndex() - vector = VectorIndex() - - persistence = IndexPersistence(kv, bm25, vector) - - save_call_count = [0] - original_save = persistence.save - - def counting_save(): - save_call_count[0] += 1 - original_save() - - with mock.patch.object(persistence, "save", side_effect=counting_save): - persistence.flush() - - assert save_call_count[0] == 1 - - def test_subsequent_schedule_after_fire_starts_new_timer(self, tmp_path): - """Two bursts of saves separated by more than DEBOUNCE_SECONDS should fire 2 saves.""" - kv = make_kv(tmp_path) - bm25 = SearchIndex() - vector = VectorIndex() - - persistence = IndexPersistence(kv, bm25, vector) - persistence.DEBOUNCE_SECONDS = FAST_DEBOUNCE - - save_call_count = [0] - original_save = persistence.save - - def counting_save(): - save_call_count[0] += 1 - original_save() - - with mock.patch.object(persistence, "save", side_effect=counting_save): - # First burst - for _ in range(10): - persistence.schedule_save() - # Wait for first timer to fire - time.sleep(FAST_DEBOUNCE * 4) - - # Second burst - for _ in range(10): - persistence.schedule_save() - # Wait for second timer to fire - time.sleep(FAST_DEBOUNCE * 4) - - assert save_call_count[0] == 2, ( - f"Expected 2 save() calls for two separate bursts; got {save_call_count[0]}" - ) diff --git a/tests/test_folder_graph_build.py b/tests/test_folder_graph_build.py deleted file mode 100644 index 5422ce5753f68c89cfc558ea2cf2dd832a401c71..0000000000000000000000000000000000000000 --- a/tests/test_folder_graph_build.py +++ /dev/null @@ -1,432 +0,0 @@ -"""Unit tests for folderColor() and folder_graph_build() — REQ-023–REQ-028.""" - -import pytest - -from agentcache.db import StateKV -from agentcache.functions import KV, folder_graph_build -from agentcache.functions import folder_color as folderColor - -# --------------------------------------------------------------------------- -# Fixtures -# --------------------------------------------------------------------------- - - -@pytest.fixture() -def kv(tmp_path): - """Return a fresh in-file StateKV backed by a temp SQLite database.""" - db_file = str(tmp_path / "test.db") - return StateKV(db_path=db_file) - - -def _write_pair( - kv: StateKV, - folder_path: str, - agent_id: str, - obs_texts: list = None, - obs_count: int = None, -) -> None: - """Insert a (folder_path, agent_id) entry into KV.folders and optionally write observations.""" - obs_texts = obs_texts or [] - count = obs_count if obs_count is not None else len(obs_texts) - - # Write folders index entry - index_key = f"{folder_path}:{agent_id}" - kv.set( - KV.folders, - index_key, - { - "folderPath": folder_path, - "agentId": agent_id, - "obsCount": count, - "lastUpdated": "2025-01-15T12:00:00.000Z", - }, - ) - - # Write observation objects if text supplied - for i, text in enumerate(obs_texts): - obs_id = f"obs_{folder_path.replace('/', '_')}_{agent_id}_{i}" - obs = { - "id": obs_id, - "folderPath": folder_path, - "agentId": agent_id, - "timestamp": "2025-01-15T12:00:00.000Z", - "text": text, - "type": "other", - "title": f"title {i}", - "concepts": [], - "files": [], - "importance": 5, - } - kv.set(KV.folder_obs(folder_path, agent_id), obs_id, obs) - - -# --------------------------------------------------------------------------- -# Tests — folderColor helper -# --------------------------------------------------------------------------- - - -def test_folder_color_returns_hsl_string(): - """folderColor should return a string matching hsl(...) format.""" - color = folderColor("projects/alpha") - assert color.startswith("hsl(") - assert color.endswith(")") - - -def test_folder_color_deterministic(): - """Same path always returns the same color.""" - assert folderColor("projects/alpha") == folderColor("projects/alpha") - - -def test_folder_color_different_paths_produce_different_colors(): - """Different paths should (almost always) produce different colors.""" - # Use very distinct paths to ensure hash difference - assert folderColor("projects/alpha") != folderColor( - "projects/omega-completely-different" - ) - - -def test_folder_color_hsl_values_in_range(): - """HSL values should be within expected ranges.""" - color = folderColor("some/path") - # Strip "hsl(" and ")" then parse - inner = color[4:-1] # e.g. "200, 70%, 55%" - parts = [p.strip().rstrip("%") for p in inner.split(",")] - hue, sat, lig = int(parts[0]), int(parts[1]), int(parts[2]) - assert 0 <= hue < 360 - assert 55 <= sat <= 79 # 55 + (h % 25) - assert 38 <= lig <= 51 # 38 + (h % 14) - - -def test_folder_color_empty_string(): - """folderColor on empty string should not raise.""" - color = folderColor("") - assert color.startswith("hsl(") - - -# --------------------------------------------------------------------------- -# Tests — empty KV returns empty graph (REQ-023) -# --------------------------------------------------------------------------- - - -def test_empty_kv_returns_empty_graph(kv): - """Empty KV returns {nodes: [], edges: []}.""" - result = folder_graph_build(kv) - assert result == {"nodes": [], "edges": []} - - -# --------------------------------------------------------------------------- -# Tests — node construction (REQ-023, REQ-024) -# --------------------------------------------------------------------------- - - -def test_one_node_per_unique_folder_path(kv): - """Two agents in the same folder produce a single node (REQ-023).""" - _write_pair(kv, "projects/alpha", "kiro", obs_count=3) - _write_pair(kv, "projects/alpha", "claude", obs_count=2) - - result = folder_graph_build(kv) - assert len(result["nodes"]) == 1 - node = result["nodes"][0] - assert node["folderPath"] == "projects/alpha" - - -def test_multiple_folders_produce_multiple_nodes(kv): - """Each distinct folder_path produces exactly one node.""" - _write_pair(kv, "projects/alpha", "kiro") - _write_pair(kv, "projects/beta", "kiro") - _write_pair(kv, "projects/gamma", "claude") - - result = folder_graph_build(kv) - folder_paths = {n["folderPath"] for n in result["nodes"]} - assert folder_paths == {"projects/alpha", "projects/beta", "projects/gamma"} - - -def test_node_fields_present(kv): - """Each node contains all required fields (REQ-024).""" - _write_pair(kv, "projects/alpha", "kiro", obs_count=5) - - result = folder_graph_build(kv) - node = result["nodes"][0] - assert "id" in node - assert "label" in node - assert "folderPath" in node - assert "agentIds" in node - assert "obsCount" in node - assert "color" in node - - -def test_node_id_equals_folder_path(kv): - """Node id is the folderPath string.""" - _write_pair(kv, "projects/alpha", "kiro") - - result = folder_graph_build(kv) - node = result["nodes"][0] - assert node["id"] == "projects/alpha" - assert node["folderPath"] == "projects/alpha" - - -def test_node_label_is_basename(kv): - """Node label is the last path component.""" - _write_pair(kv, "home/user/projects/myapp", "kiro") - - result = folder_graph_build(kv) - node = result["nodes"][0] - assert node["label"] == "myapp" - - -def test_node_agent_ids_aggregated_and_sorted(kv): - """agentIds is the sorted union of all agents for that folder.""" - _write_pair(kv, "projects/alpha", "zorro", obs_count=1) - _write_pair(kv, "projects/alpha", "alice", obs_count=1) - _write_pair(kv, "projects/alpha", "bob", obs_count=1) - - result = folder_graph_build(kv) - node = result["nodes"][0] - assert node["agentIds"] == ["alice", "bob", "zorro"] - - -def test_node_obs_count_summed_across_agents(kv): - """obsCount is the sum across all agents for that folder.""" - _write_pair(kv, "projects/alpha", "kiro", obs_count=4) - _write_pair(kv, "projects/alpha", "claude", obs_count=6) - - result = folder_graph_build(kv) - node = result["nodes"][0] - assert node["obsCount"] == 10 - - -def test_node_color_is_hsl(kv): - """Node color comes from folderColor and is an HSL string.""" - _write_pair(kv, "projects/alpha", "kiro") - - result = folder_graph_build(kv) - node = result["nodes"][0] - assert node["color"].startswith("hsl(") - # Must match folderColor directly - assert node["color"] == folderColor("projects/alpha") - - -# --------------------------------------------------------------------------- -# Tests — same-parent edges (REQ-025) -# --------------------------------------------------------------------------- - - -def test_same_parent_edge_created(kv): - """Two folders with the same parent get a same-parent edge.""" - _write_pair(kv, "projects/alpha", "kiro") - _write_pair(kv, "projects/beta", "kiro") # both under "projects" - - result = folder_graph_build(kv) - same_parent_edges = [e for e in result["edges"] if e["type"] == "same-parent"] - assert len(same_parent_edges) == 1 - edge = same_parent_edges[0] - assert set([edge["source"], edge["target"]]) == {"projects/alpha", "projects/beta"} - - -def test_no_same_parent_edge_for_different_parents(kv): - """Folders with different parents do not get a same-parent edge.""" - _write_pair(kv, "projects/alpha", "kiro") - _write_pair(kv, "work/beta", "kiro") - - result = folder_graph_build(kv) - same_parent_edges = [e for e in result["edges"] if e["type"] == "same-parent"] - assert same_parent_edges == [] - - -def test_same_parent_edge_only_for_sharing_pairs(kv): - """Only pairs sharing a parent get same-parent edges; non-sharing pairs do not.""" - _write_pair(kv, "a/x", "kiro") - _write_pair(kv, "a/y", "kiro") # shares parent "a" with a/x - _write_pair(kv, "b/z", "kiro") # different parent "b" - - result = folder_graph_build(kv) - same_parent_edges = [e for e in result["edges"] if e["type"] == "same-parent"] - assert len(same_parent_edges) == 1 - edge = same_parent_edges[0] - assert set([edge["source"], edge["target"]]) == {"a/x", "a/y"} - - -# --------------------------------------------------------------------------- -# Tests — cross-reference edges (REQ-026) -# --------------------------------------------------------------------------- - - -def test_cross_ref_edge_when_obs_mentions_other_folder(kv): - """A cross-ref edge is created when folder A's obs text mentions folder B's path.""" - _write_pair( - kv, "projects/alpha", "kiro", obs_texts=["I worked on projects/beta today"] - ) - _write_pair(kv, "projects/beta", "kiro", obs_texts=["nothing"]) - - result = folder_graph_build(kv) - cross_edges = [e for e in result["edges"] if e["type"] == "cross-ref"] - assert len(cross_edges) >= 1 - sources_targets = {(e["source"], e["target"]) for e in cross_edges} - assert ("projects/alpha", "projects/beta") in sources_targets - - -def test_no_cross_ref_edge_when_no_mention(kv): - """No cross-ref edge when obs texts don't mention another folder path.""" - _write_pair(kv, "projects/alpha", "kiro", obs_texts=["Just some work here"]) - _write_pair(kv, "projects/beta", "kiro", obs_texts=["Unrelated content"]) - - result = folder_graph_build(kv) - cross_edges = [e for e in result["edges"] if e["type"] == "cross-ref"] - assert cross_edges == [] - - -def test_cross_ref_edge_from_title_mention(kv): - """Cross-ref edges are also detected via obs titles.""" - _write_pair(kv, "projects/alpha", "kiro", obs_texts=["some text"]) - # Manually insert obs with a title that mentions the other folder - obs = { - "id": "obs_special", - "folderPath": "projects/alpha", - "agentId": "kiro", - "timestamp": "2025-01-15T12:00:00.000Z", - "text": "normal text", - "type": "other", - "title": "work on projects/beta", - "concepts": [], - "files": [], - "importance": 5, - } - kv.set(KV.folder_obs("projects/alpha", "kiro"), "obs_special", obs) - _write_pair(kv, "projects/beta", "kiro", obs_texts=["nothing"]) - - result = folder_graph_build(kv) - cross_edges = [e for e in result["edges"] if e["type"] == "cross-ref"] - sources = {e["source"] for e in cross_edges} - assert "projects/alpha" in sources - - -# --------------------------------------------------------------------------- -# Tests — agent-shared edges (REQ-027) -# --------------------------------------------------------------------------- - - -def test_agent_shared_edge_created(kv): - """Two folders with a common agent get an agent-shared edge.""" - _write_pair(kv, "projects/alpha", "kiro") - _write_pair(kv, "projects/beta", "kiro") # same agent "kiro" - - result = folder_graph_build(kv) - agent_edges = [e for e in result["edges"] if e["type"] == "agent-shared"] - assert len(agent_edges) >= 1 - edge = agent_edges[0] - assert set([edge["source"], edge["target"]]) == {"projects/alpha", "projects/beta"} - - -def test_no_agent_shared_edge_when_no_common_agent(kv): - """Folders with no common agents do not get an agent-shared edge.""" - _write_pair(kv, "projects/alpha", "kiro") - _write_pair(kv, "projects/beta", "claude") # different agents - - result = folder_graph_build(kv) - agent_edges = [e for e in result["edges"] if e["type"] == "agent-shared"] - assert agent_edges == [] - - -def test_agent_shared_edge_with_partial_overlap(kv): - """Two folders with one common agent among several agents still get an edge.""" - _write_pair(kv, "projects/alpha", "kiro") - _write_pair(kv, "projects/alpha", "claude") - _write_pair(kv, "projects/beta", "claude") - _write_pair(kv, "projects/beta", "cursor") - - result = folder_graph_build(kv) - agent_edges = [e for e in result["edges"] if e["type"] == "agent-shared"] - endpoints = {frozenset([e["source"], e["target"]]) for e in agent_edges} - assert frozenset({"projects/alpha", "projects/beta"}) in endpoints - - -# --------------------------------------------------------------------------- -# Tests — edge deduplication (REQ-028) -# --------------------------------------------------------------------------- - - -def test_no_duplicate_edges(kv): - """No two edges share the same (source, target, type) pair.""" - _write_pair(kv, "projects/alpha", "kiro", obs_texts=["mentions projects/beta"]) - _write_pair(kv, "projects/beta", "kiro", obs_texts=["mentions projects/alpha"]) - - result = folder_graph_build(kv) - seen = set() - for edge in result["edges"]: - key = (frozenset([edge["source"], edge["target"]]), edge["type"]) - assert key not in seen, f"Duplicate edge: {edge}" - seen.add(key) - - -def test_ab_and_ba_treated_as_same_edge(kv): - """(a, b, type) and (b, a, type) are considered the same edge.""" - # Both folders reference each other — should produce only one cross-ref edge - _write_pair( - kv, "projects/alpha", "kiro", obs_texts=["See also projects/beta for details"] - ) - _write_pair( - kv, "projects/beta", "kiro", obs_texts=["Related to projects/alpha work"] - ) - - result = folder_graph_build(kv) - cross_edges = [e for e in result["edges"] if e["type"] == "cross-ref"] - # Should be exactly 1 cross-ref edge (not 2) - assert len(cross_edges) == 1 - - -def test_same_parent_and_agent_shared_are_separate_edge_types(kv): - """same-parent and agent-shared edges between the same pair are both kept.""" - # Both folders share parent "projects" AND share agent "kiro" - _write_pair(kv, "projects/alpha", "kiro") - _write_pair(kv, "projects/beta", "kiro") - - result = folder_graph_build(kv) - edge_types = {e["type"] for e in result["edges"]} - # We expect both types to appear - assert "same-parent" in edge_types - assert "agent-shared" in edge_types - - -# --------------------------------------------------------------------------- -# Tests — return structure -# --------------------------------------------------------------------------- - - -def test_return_has_nodes_and_edges_keys(kv): - """Result always has 'nodes' and 'edges' keys.""" - _write_pair(kv, "projects/alpha", "kiro") - result = folder_graph_build(kv) - assert "nodes" in result - assert "edges" in result - - -def test_edge_has_required_fields(kv): - """Each edge has source, target, and type fields.""" - _write_pair(kv, "projects/alpha", "kiro") - _write_pair(kv, "projects/beta", "kiro") - - result = folder_graph_build(kv) - for edge in result["edges"]: - assert "source" in edge - assert "target" in edge - assert "type" in edge - - -def test_single_folder_produces_no_edges(kv): - """A graph with only one folder produces no edges.""" - _write_pair(kv, "projects/alpha", "kiro") - - result = folder_graph_build(kv) - assert len(result["nodes"]) == 1 - assert result["edges"] == [] - - -def test_edge_types_are_valid(kv): - """All edge types are one of the three valid values.""" - _write_pair(kv, "projects/alpha", "kiro", obs_texts=["mentions projects/beta"]) - _write_pair(kv, "projects/beta", "kiro") - - result = folder_graph_build(kv) - valid_types = {"same-parent", "cross-ref", "agent-shared"} - for edge in result["edges"]: - assert edge["type"] in valid_types diff --git a/tests/test_folder_observe.py b/tests/test_folder_observe.py deleted file mode 100644 index 8a87690811dfeeeea2c77d43287bc23b1f313137..0000000000000000000000000000000000000000 --- a/tests/test_folder_observe.py +++ /dev/null @@ -1,121 +0,0 @@ -"""Unit tests for folder_observe (REQ-008, REQ-010, REQ-011, REQ-015).""" - -import datetime -import os - -import pytest - -from agentcache.db import StateKV -from agentcache.functions import KV, folder_observe - - -def make_kv(tmp_path): - db_path = os.path.join(str(tmp_path), "test.db") - return StateKV(db_path=db_path) - - -def base_payload(**overrides): - payload = { - "folderPath": "/home/user/projects/myapp", - "agentId": "kiro", - "text": "Edited src/app.py to add a new route", - "timestamp": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - } - payload.update(overrides) - return payload - - -class TestFolderObserveMissingFields: - def test_missing_folder_path(self, tmp_path): - kv = make_kv(tmp_path) - with pytest.raises(ValueError, match="folderPath"): - folder_observe(kv, base_payload(folderPath="")) - - def test_missing_agent_id(self, tmp_path): - kv = make_kv(tmp_path) - with pytest.raises(ValueError, match="agentId"): - folder_observe(kv, base_payload(agentId="")) - - def test_missing_text(self, tmp_path): - kv = make_kv(tmp_path) - with pytest.raises(ValueError, match="text"): - folder_observe(kv, base_payload(text="")) - - def test_missing_timestamp_defaults(self, tmp_path): - kv = make_kv(tmp_path) - payload = base_payload() - del payload["timestamp"] - # timestamp is required — should raise - with pytest.raises(ValueError, match="timestamp"): - folder_observe(kv, payload) - - -class TestFolderObserveSuccess: - def test_returns_observation_id(self, tmp_path): - kv = make_kv(tmp_path) - result = folder_observe(kv, base_payload()) - assert "observationId" in result - assert result["observationId"].startswith("fobs_") - - def test_obs_stored_in_kv(self, tmp_path): - kv = make_kv(tmp_path) - result = folder_observe(kv, base_payload()) - obs_id = result["observationId"] - fp = "home/user/projects/myapp" # normalized - stored = kv.get(KV.folder_obs(fp, "kiro"), obs_id) - assert stored is not None - assert stored["id"] == obs_id - - def test_obs_count_incremented(self, tmp_path): - kv = make_kv(tmp_path) - folder_observe(kv, base_payload(text="First observation")) - folder_observe(kv, base_payload(text="Second observation")) - fp = "home/user/projects/myapp" - meta = kv.get(KV.folder_meta(fp, "kiro"), "meta") - assert meta is not None - assert meta["obsCount"] == 2 - - def test_folders_index_upserted(self, tmp_path): - kv = make_kv(tmp_path) - folder_observe(kv, base_payload()) - fp = "home/user/projects/myapp" - entry = kv.get(KV.folders, f"{fp}:kiro") - assert entry is not None - assert entry["folderPath"] == fp - assert entry["agentId"] == "kiro" - - def test_text_capped_at_4000(self, tmp_path): - kv = make_kv(tmp_path) - long_text = "x" * 5000 - result = folder_observe(kv, base_payload(text=long_text)) - fp = "home/user/projects/myapp" - stored = kv.get(KV.folder_obs(fp, "kiro"), result["observationId"]) - assert len(stored["text"]) <= 4000 - - -class TestFolderObserveCap: - def test_cap_enforced(self, tmp_path, monkeypatch): - monkeypatch.setenv("MAX_OBS_PER_FOLDER", "3") - kv = make_kv(tmp_path) - for i in range(3): - folder_observe(kv, base_payload(text=f"observation {i}")) - with pytest.raises(ValueError, match="limit"): - folder_observe(kv, base_payload(text="observation 4")) - - -class TestFolderObservePairIsolation: - def test_different_pairs_isolated(self, tmp_path): - kv = make_kv(tmp_path) - folder_observe(kv, base_payload(folderPath="/home/user/proj-a", agentId="kiro")) - folder_observe( - kv, base_payload(folderPath="/home/user/proj-b", agentId="claude") - ) - fp_a = "home/user/proj-a" - fp_b = "home/user/proj-b" - obs_a = kv.list(KV.folder_obs(fp_a, "kiro")) - obs_b = kv.list(KV.folder_obs(fp_b, "claude")) - ids_a = {o["id"] for o in obs_a} - ids_b = {o["id"] for o in obs_b} - assert ids_a.isdisjoint(ids_b) diff --git a/tests/test_forget.py b/tests/test_forget.py deleted file mode 100644 index 11e190a344f29fca58e00f1734b5451ce8740467..0000000000000000000000000000000000000000 --- a/tests/test_forget.py +++ /dev/null @@ -1,112 +0,0 @@ -"""Unit tests for forget() folder-based deletion (REQ-029–REQ-033).""" - -import datetime -import os - -from agentcache.db import StateKV -from agentcache.functions import KV, folder_observe, forget - - -def make_kv(tmp_path): - db_path = os.path.join(str(tmp_path), "test.db") - return StateKV(db_path=db_path) - - -def add_obs(kv, folder="/home/user/proj", agent="kiro", n=1): - ids = [] - for i in range(n): - result = folder_observe( - kv, - { - "folderPath": folder, - "agentId": agent, - "text": f"observation {i}", - "timestamp": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - }, - ) - ids.append(result["observationId"]) - return ids - - -class TestForgetFullPair: - def test_full_deletion_clears_obs(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, n=3) - fp = "home/user/proj" - result = forget(kv, {"folderPath": "/home/user/proj", "agentId": "kiro"}) - assert result["deleted"] >= 3 - obs = kv.list(KV.folder_obs(fp, "kiro")) - assert len(obs) == 0 - - def test_full_deletion_removes_index_entry(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv) - fp = "home/user/proj" - forget(kv, {"folderPath": "/home/user/proj", "agentId": "kiro"}) - entry = kv.get(KV.folders, f"{fp}:kiro") - assert entry is None - - def test_full_deletion_removes_meta(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv) - fp = "home/user/proj" - forget(kv, {"folderPath": "/home/user/proj", "agentId": "kiro"}) - meta = kv.get(KV.folder_meta(fp, "kiro"), "meta") - assert meta is None - - def test_deleted_count_matches(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, n=5) - result = forget(kv, {"folderPath": "/home/user/proj", "agentId": "kiro"}) - assert result["deleted"] == 5 - - -class TestForgetPartial: - def test_partial_deletion(self, tmp_path): - kv = make_kv(tmp_path) - ids = add_obs(kv, n=4) - fp = "home/user/proj" - to_delete = ids[:2] - result = forget( - kv, - { - "folderPath": "/home/user/proj", - "agentId": "kiro", - "observationIds": to_delete, - }, - ) - assert result["deleted"] == 2 - remaining = kv.list(KV.folder_obs(fp, "kiro")) - remaining_ids = {o["id"] for o in remaining} - for oid in to_delete: - assert oid not in remaining_ids - - def test_partial_decrements_obs_count(self, tmp_path): - kv = make_kv(tmp_path) - ids = add_obs(kv, n=4) - fp = "home/user/proj" - forget( - kv, - { - "folderPath": "/home/user/proj", - "agentId": "kiro", - "observationIds": ids[:2], - }, - ) - meta = kv.get(KV.folder_meta(fp, "kiro"), "meta") - assert meta["obsCount"] == 2 - - -class TestForgetMemory: - def test_delete_global_memory(self, tmp_path): - kv = make_kv(tmp_path) - from agentcache.functions import remember - - result = remember(kv, {"content": "Important insight", "type": "fact"}) - mem_id = result["memory"]["id"] - forget_result = forget(kv, {"memoryId": mem_id}) - assert forget_result["deleted"] >= 1 - stored = kv.get(KV.memories, mem_id) - assert stored is None diff --git a/tests/test_graph.py b/tests/test_graph.py deleted file mode 100644 index 386df372936c06a432463da909ca7120e9694f69..0000000000000000000000000000000000000000 --- a/tests/test_graph.py +++ /dev/null @@ -1,102 +0,0 @@ -"""Unit tests for folder_graph_build (REQ-023–REQ-028).""" - -import datetime -import os - -from agentcache.db import StateKV -from agentcache.functions import folder_graph_build, folder_observe - - -def make_kv(tmp_path): - db_path = os.path.join(str(tmp_path), "test.db") - return StateKV(db_path=db_path) - - -def add_obs(kv, folder, agent="kiro", text="obs"): - folder_observe( - kv, - { - "folderPath": folder, - "agentId": agent, - "text": text, - "timestamp": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - }, - ) - - -class TestGraphEmpty: - def test_empty_kv_returns_empty(self, tmp_path): - kv = make_kv(tmp_path) - result = folder_graph_build(kv) - assert result == {"nodes": [], "edges": []} - - -class TestGraphNodes: - def test_one_node_per_folder(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, "/home/user/proj-a", agent="kiro") - add_obs(kv, "/home/user/proj-a", agent="claude") # same folder, different agent - add_obs(kv, "/home/user/proj-b", agent="kiro") - result = folder_graph_build(kv) - node_ids = [n["id"] for n in result["nodes"]] - assert len(node_ids) == len(set(node_ids)) # no duplicates - # 2 unique folders - assert len(result["nodes"]) == 2 - - def test_node_has_required_fields(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, "/home/user/proj") - result = folder_graph_build(kv) - node = result["nodes"][0] - assert "id" in node - assert "label" in node - assert "folderPath" in node - assert "agentIds" in node - assert "obsCount" in node - assert "color" in node - - def test_agent_ids_aggregated(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, "/home/user/proj", agent="kiro") - add_obs(kv, "/home/user/proj", agent="claude") - result = folder_graph_build(kv) - node = result["nodes"][0] - assert "kiro" in node["agentIds"] - assert "claude" in node["agentIds"] - - -class TestGraphEdges: - def test_same_parent_edge(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, "/home/user/proj/src") - add_obs(kv, "/home/user/proj/tests") - result = folder_graph_build(kv) - same_parent = [e for e in result["edges"] if e["type"] == "same-parent"] - assert len(same_parent) >= 1 - - def test_no_duplicate_edges(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, "/home/user/proj/src") - add_obs(kv, "/home/user/proj/tests") - result = folder_graph_build(kv) - edge_keys = [(e["source"], e["target"], e["type"]) for e in result["edges"]] - assert len(edge_keys) == len(set(edge_keys)) - - def test_agent_shared_edge(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, "/home/user/proj-a", agent="kiro") - add_obs(kv, "/home/user/proj-b", agent="kiro") - result = folder_graph_build(kv) - agent_edges = [e for e in result["edges"] if e["type"] == "agent-shared"] - assert len(agent_edges) >= 1 - - def test_cross_ref_edge(self, tmp_path): - kv = make_kv(tmp_path) - fp_b = "home/user/proj-b" - add_obs(kv, "/home/user/proj-a", text=f"Modified files in {fp_b}") - add_obs(kv, "/home/user/proj-b", text="Normal work") - result = folder_graph_build(kv) - cross_ref = [e for e in result["edges"] if e["type"] == "cross-ref"] - assert len(cross_ref) >= 1 diff --git a/tests/test_migration.py b/tests/test_migration.py deleted file mode 100644 index 31d84043e70cf9196fe9be33e382b7fe889046f9..0000000000000000000000000000000000000000 --- a/tests/test_migration.py +++ /dev/null @@ -1,144 +0,0 @@ -"""Unit tests for migrate_sessions_to_folders (REQ-058–REQ-062).""" - -import datetime -import os - -from agentcache.db import StateKV -from agentcache.functions import KV, migrate_sessions_to_folders - - -def make_kv(tmp_path): - db_path = os.path.join(str(tmp_path), "test.db") - return StateKV(db_path=db_path) - - -def seed_session( - kv, session_id="sess_001", cwd="/home/user/proj", agent="kiro", obs_count=3 -): - """Seed a legacy session with observations into the old schema.""" - ts = datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - session = { - "id": session_id, - "project": cwd, - "cwd": cwd, - "agentId": agent, - "startedAt": ts, - "updatedAt": ts, - "status": "completed", - } - kv.set(KV.sessions, session_id, session) - obs_ids = [] - for i in range(obs_count): - obs_id = f"obs_{i}" - obs = { - "id": obs_id, - "sessionId": session_id, - "timestamp": ts, - "type": "file_edit", - "title": f"Edit {i}", - "narrative": f"Edited file {i}", - "concepts": ["python"], - "files": [f"src/file_{i}.py"], - "importance": 5, - } - kv.set(KV.observations(session_id), obs_id, obs) - obs_ids.append(obs_id) - return obs_ids - - -class TestMigrateDryRun: - def test_dry_run_writes_nothing(self, tmp_path): - kv = make_kv(tmp_path) - seed_session(kv) - result = migrate_sessions_to_folders(kv, dry_run=True) - assert result["migrated_sessions"] > 0 - assert result["migrated_observations"] > 0 - # No folder obs should have been written - fp = "home/user/proj" - obs = kv.list(KV.folder_obs(fp, "kiro")) - assert len(obs) == 0 - - def test_dry_run_returns_counts(self, tmp_path): - kv = make_kv(tmp_path) - seed_session(kv, obs_count=5) - result = migrate_sessions_to_folders(kv, dry_run=True) - assert result["migrated_observations"] == 5 - assert result["dry_run"] is True - - -class TestMigrateActual: - def test_migrates_observations(self, tmp_path): - kv = make_kv(tmp_path) - seed_session(kv, obs_count=3) - migrate_sessions_to_folders(kv, dry_run=False) - fp = "home/user/proj" - obs = kv.list(KV.folder_obs(fp, "kiro")) - assert len(obs) == 3 - - def test_skips_raw_observations(self, tmp_path): - kv = make_kv(tmp_path) - seed_session(kv, obs_count=2) - # Add a raw obs - kv.set( - KV.observations("sess_001"), - "obs_0:raw", - { - "id": "obs_0:raw", - "sessionId": "sess_001", - "timestamp": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - }, - ) - result = migrate_sessions_to_folders(kv, dry_run=False) - # Raw obs should not be counted - assert result["migrated_observations"] == 2 - - def test_nondestructive(self, tmp_path): - kv = make_kv(tmp_path) - seed_session(kv) - migrate_sessions_to_folders(kv, dry_run=False) - # Old session data still there - session = kv.get(KV.sessions, "sess_001") - assert session is not None - - def test_unknown_fallback(self, tmp_path): - kv = make_kv(tmp_path) - # Session with no cwd or project - ts = ( - datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z") - ) - kv.set( - KV.sessions, - "sess_no_path", - { - "id": "sess_no_path", - "startedAt": ts, - "updatedAt": ts, - "status": "completed", - }, - ) - kv.set( - KV.observations("sess_no_path"), - "obs_x", - { - "id": "obs_x", - "sessionId": "sess_no_path", - "timestamp": ts, - "type": "other", - "title": "x", - "narrative": "x", - }, - ) - result = migrate_sessions_to_folders(kv, dry_run=False) - # Should succeed with 'unknown' fallback - assert result["migrated_sessions"] >= 1 - - def test_returns_error_list(self, tmp_path): - kv = make_kv(tmp_path) - seed_session(kv) - result = migrate_sessions_to_folders(kv, dry_run=False) - assert "errors" in result - assert isinstance(result["errors"], list) diff --git a/tests/test_normalize.py b/tests/test_normalize.py deleted file mode 100644 index 0927cbe8be49374012f6bd7e36c23cd2adbcce68..0000000000000000000000000000000000000000 --- a/tests/test_normalize.py +++ /dev/null @@ -1,61 +0,0 @@ -"""Unit tests for normalize_folder_path (REQ-002, REQ-063, REQ-064, REQ-066).""" - -import pytest - -from agentcache.functions import normalize_folder_path - - -class TestNormalizeFolderPath: - def test_unix_path(self): - assert ( - normalize_folder_path("/home/user/projects/myapp") - == "home/user/projects/myapp" - ) - - def test_windows_path(self): - result = normalize_folder_path("C:\\Users\\foo\\projects\\myapp") - assert "\\" not in result - assert "Users" in result or "users" in result.lower() - - def test_trailing_slash_stripped(self): - result = normalize_folder_path("/home/user/projects/") - assert not result.endswith("/") - - def test_leading_slash_stripped(self): - result = normalize_folder_path("/home/user/projects") - assert not result.startswith("/") - - def test_double_slashes_collapsed(self): - result = normalize_folder_path("/home//user///projects") - assert "//" not in result - - def test_empty_string_raises(self): - with pytest.raises(ValueError): - normalize_folder_path("") - - def test_path_traversal_raises(self): - with pytest.raises(ValueError): - normalize_folder_path("/home/user/../../etc/passwd") - - def test_length_cap(self): - long_path = "a/" * 300 - result = normalize_folder_path(long_path) - assert len(result) <= 512 - - def test_idempotent(self): - path = "/home/user/projects/myapp" - once = normalize_folder_path(path) - twice = normalize_folder_path(once) - assert once == twice - - def test_relative_path(self): - result = normalize_folder_path("projects/myapp/src") - assert result == "projects/myapp/src" - - def test_windows_forward_slashes(self): - result = normalize_folder_path("C:/Users/foo/projects") - assert "\\" not in result - - def test_single_segment(self): - result = normalize_folder_path("/workspace") - assert result == "workspace" diff --git a/tests/test_obs_lookup.py b/tests/test_obs_lookup.py deleted file mode 100644 index 5288255bd18ff4e9a40a5506abaa7209b97313d3..0000000000000000000000000000000000000000 --- a/tests/test_obs_lookup.py +++ /dev/null @@ -1,116 +0,0 @@ -"""Unit tests for observation lookup index, backfill, and index sync validation.""" - -import datetime -import os - -from agentcache import functions -from agentcache.db import StateKV -from agentcache.functions import ( - KV, - backfill_obs_lookup_if_needed, - folder_observe, - forget, - verify_index_sync_on_boot, -) - - -def make_kv(tmp_path): - db_path = os.path.join(str(tmp_path), "test_obs_lookup.db") - return StateKV(db_path=db_path) - - -def base_payload(folder="/home/user/proj", agent="kiro", text="Test observation"): - return { - "folderPath": folder, - "agentId": agent, - "text": text, - "timestamp": datetime.datetime.now(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z"), - } - - -class TestObsLookupFlows: - def test_lookup_added_on_observe(self, tmp_path): - kv = make_kv(tmp_path) - res = folder_observe(kv, base_payload()) - obs_id = res["observationId"] - - # Verify entry in KV.obs_lookup - lookup = kv.get(KV.obs_lookup, obs_id) - assert lookup is not None - assert lookup["folderPath"] == "home/user/proj" - assert lookup["agentId"] == "kiro" - - def test_lookup_deleted_on_forget(self, tmp_path): - kv = make_kv(tmp_path) - res = folder_observe(kv, base_payload()) - obs_id = res["observationId"] - - # Confirm it exists - assert kv.get(KV.obs_lookup, obs_id) is not None - - # Delete it using forget - forget( - kv, - { - "folderPath": "/home/user/proj", - "agentId": "kiro", - "observationIds": [obs_id], - }, - ) - - # Confirm it is gone from both stores - assert kv.get(KV.obs_lookup, obs_id) is None - assert kv.get(KV.folder_obs("home/user/proj", "kiro"), obs_id) is None - - def test_backfill_populates_missing_lookups(self, tmp_path): - kv = make_kv(tmp_path) - - # Ingest observations - res1 = folder_observe(kv, base_payload(text="First")) - res2 = folder_observe(kv, base_payload(text="Second")) - obs1 = res1["observationId"] - obs2 = res2["observationId"] - - # Manually clear the lookup index (simulating legacy data) - kv.delete(KV.obs_lookup, obs1) - kv.delete(KV.obs_lookup, obs2) - assert kv.get(KV.obs_lookup, obs1) is None - assert kv.get(KV.obs_lookup, obs2) is None - - # Run backfill - backfill_obs_lookup_if_needed(kv) - - # Verify populated - lookup1 = kv.get(KV.obs_lookup, obs1) - lookup2 = kv.get(KV.obs_lookup, obs2) - assert lookup1 is not None - assert lookup2 is not None - assert lookup1["folderPath"] == "home/user/proj" - assert lookup2["folderPath"] == "home/user/proj" - - def test_verify_index_sync_detects_mismatch(self, tmp_path): - kv = make_kv(tmp_path) - - # Clear indexes - functions._bm25_index.clear() - if functions._vector_index: - functions._vector_index.clear() - - # Ingest one observation - res = folder_observe(kv, base_payload()) - obs_id = res["observationId"] - - # BM25 size should be 1 - assert functions._bm25_index.size == 1 - - # verify_index_sync_on_boot should return True (in sync) - assert verify_index_sync_on_boot(kv) is True - - # Manually remove from BM25 (simulate dirty restart) - functions._bm25_index.remove(obs_id) - assert functions._bm25_index.size == 0 - - # verify_index_sync_on_boot should detect mismatch and return False - assert verify_index_sync_on_boot(kv) is False diff --git a/tests/test_observation_store.py b/tests/test_observation_store.py index 7c45bd97201642452d5e13e62511c9116df8ce39..f456bcabcc16d4e8bddbb69da0429862f8a21523 100644 --- a/tests/test_observation_store.py +++ b/tests/test_observation_store.py @@ -1,10 +1,9 @@ -import pytest -import os import json -from agentcache.core import KV, ObservationStore, ObservationEvents, SearchService -from agentcache.db import StateKV + +import pytest + +from agentcache.core import KV, ObservationEvents, ObservationStore, SearchService from agentcache.search import SearchIndex -from agentcache.app import create_app def test_observation_events_dataclass(): @@ -72,26 +71,32 @@ def test_observation_store_max_cap(tmp_db, monkeypatch): kv = tmp_db store = ObservationStore(kv=kv) - store.ingest({ - "folderPath": "src/cap", - "agentId": "agent_beta", - "text": "First item", - "timestamp": "2026-07-24T12:00:00Z", - }) - store.ingest({ - "folderPath": "src/cap", - "agentId": "agent_beta", - "text": "Second item", - "timestamp": "2026-07-24T12:01:00Z", - }) - - with pytest.raises(ValueError, match="Folder observation limit reached"): - store.ingest({ + store.ingest( + { "folderPath": "src/cap", "agentId": "agent_beta", - "text": "Third item", - "timestamp": "2026-07-24T12:02:00Z", - }) + "text": "First item", + "timestamp": "2026-07-24T12:00:00Z", + } + ) + store.ingest( + { + "folderPath": "src/cap", + "agentId": "agent_beta", + "text": "Second item", + "timestamp": "2026-07-24T12:01:00Z", + } + ) + + with pytest.raises(ValueError, match="Folder observation limit reached"): + store.ingest( + { + "folderPath": "src/cap", + "agentId": "agent_beta", + "text": "Third item", + "timestamp": "2026-07-24T12:02:00Z", + } + ) def test_observation_store_manual_dedup(tmp_db): @@ -114,11 +119,23 @@ def test_observation_store_manual_dedup(tmp_db): "timestamp": "2026-07-24T11:00:00Z", } - kv.set(KV.folders, "src/dedup:agent_gamma", {"folderPath": "src/dedup", "agentId": "agent_gamma"}) + kv.set( + KV.folders, + "src/dedup:agent_gamma", + {"folderPath": "src/dedup", "agentId": "agent_gamma"}, + ) kv.set(KV.folder_obs("src/dedup", "agent_gamma"), "fobs_dup_1", obs1) kv.set(KV.folder_obs("src/dedup", "agent_gamma"), "fobs_dup_2", obs2) - kv.set(KV.obs_lookup, "fobs_dup_1", {"folderPath": "src/dedup", "agentId": "agent_gamma"}) - kv.set(KV.obs_lookup, "fobs_dup_2", {"folderPath": "src/dedup", "agentId": "agent_gamma"}) + kv.set( + KV.obs_lookup, + "fobs_dup_1", + {"folderPath": "src/dedup", "agentId": "agent_gamma"}, + ) + kv.set( + KV.obs_lookup, + "fobs_dup_2", + {"folderPath": "src/dedup", "agentId": "agent_gamma"}, + ) res = store.dedup("src/dedup", "agent_gamma") assert res["success"] is True @@ -145,35 +162,43 @@ def test_observation_store_forget_full_and_partial(tmp_db): store = ObservationStore(kv=kv, search_service=search_svc, events=events) - obs1_id = store.ingest({ - "folderPath": "src/forget", - "agentId": "agent_forget", - "text": "First item to forget", - "timestamp": "2026-07-24T10:00:00Z", - })["observationId"] - - obs2_id = store.ingest({ - "folderPath": "src/forget", - "agentId": "agent_forget", - "text": "Second item to keep initially", - "timestamp": "2026-07-24T11:00:00Z", - })["observationId"] + obs1_id = store.ingest( + { + "folderPath": "src/forget", + "agentId": "agent_forget", + "text": "First item to forget", + "timestamp": "2026-07-24T10:00:00Z", + } + )["observationId"] + + store.ingest( + { + "folderPath": "src/forget", + "agentId": "agent_forget", + "text": "Second item to keep initially", + "timestamp": "2026-07-24T11:00:00Z", + } + )["observationId"] # Explicitly empty observationIds list -> graceful handling of no-op partial delete - res_noop = store.forget({ - "folderPath": "src/forget", - "agentId": "agent_forget", - "observationIds": [], - }) + res_noop = store.forget( + { + "folderPath": "src/forget", + "agentId": "agent_forget", + "observationIds": [], + } + ) assert res_noop["success"] is True assert res_noop["deleted"] == 0 # 1. Partial deletion - res_part = store.forget({ - "folderPath": "src/forget", - "agentId": "agent_forget", - "observationIds": [obs1_id], - }) + res_part = store.forget( + { + "folderPath": "src/forget", + "agentId": "agent_forget", + "observationIds": [obs1_id], + } + ) assert res_part["success"] is True assert res_part["deleted"] == 1 assert deleted_obs == [obs1_id] @@ -182,10 +207,12 @@ def test_observation_store_forget_full_and_partial(tmp_db): assert meta["obsCount"] == 1 # 2. Full folder pair deletion - res_full = store.forget({ - "folderPath": "src/forget", - "agentId": "agent_forget", - }) + res_full = store.forget( + { + "folderPath": "src/forget", + "agentId": "agent_forget", + } + ) assert res_full["success"] is True assert res_full["deleted"] == 1 assert len(deleted_folders) == 1 @@ -200,24 +227,30 @@ def test_observation_store_timeline_sorting_and_filtering(tmp_db): kv = tmp_db store = ObservationStore(kv=kv) - store.ingest({ - "folderPath": "src/t1", - "agentId": "agent_t", - "text": "Oldest observation", - "timestamp": "2026-07-24T10:00:00Z", - }) - store.ingest({ - "folderPath": "src/t1", - "agentId": "agent_t", - "text": "Middle observation", - "timestamp": "2026-07-24T12:00:00Z", - }) - store.ingest({ - "folderPath": "src/t2", - "agentId": "agent_t", - "text": "Newest observation", - "timestamp": "2026-07-24T14:00:00Z", - }) + store.ingest( + { + "folderPath": "src/t1", + "agentId": "agent_t", + "text": "Oldest observation", + "timestamp": "2026-07-24T10:00:00Z", + } + ) + store.ingest( + { + "folderPath": "src/t1", + "agentId": "agent_t", + "text": "Middle observation", + "timestamp": "2026-07-24T12:00:00Z", + } + ) + store.ingest( + { + "folderPath": "src/t2", + "agentId": "agent_t", + "text": "Newest observation", + "timestamp": "2026-07-24T14:00:00Z", + } + ) # Timeline all tl_all = store.timeline(limit=10) @@ -248,14 +281,22 @@ def test_observation_store_rebuild_index_and_backfill_lookup(tmp_db): store = ObservationStore(kv=kv, search_service=search_svc) # 1. Backfill test: insert raw observation without lookup entry - kv.set(KV.folders, "src/bf:agent_bf", {"folderPath": "src/bf", "agentId": "agent_bf", "obsCount": 1}) - kv.set(KV.folder_obs("src/bf", "agent_bf"), "fobs_bf1", { - "id": "fobs_bf1", - "folderPath": "src/bf", - "agentId": "agent_bf", - "text": "Unindexed backfill observation", - "timestamp": "2026-07-24T10:00:00Z", - }) + kv.set( + KV.folders, + "src/bf:agent_bf", + {"folderPath": "src/bf", "agentId": "agent_bf", "obsCount": 1}, + ) + kv.set( + KV.folder_obs("src/bf", "agent_bf"), + "fobs_bf1", + { + "id": "fobs_bf1", + "folderPath": "src/bf", + "agentId": "agent_bf", + "text": "Unindexed backfill observation", + "timestamp": "2026-07-24T10:00:00Z", + }, + ) kv.delete(KV.obs_lookup, "fobs_bf1") assert kv.get(KV.obs_lookup, "fobs_bf1") is None @@ -355,7 +396,11 @@ def test_full_lifecycle_e2e_pass(app_client): # 2. SEARCH via HTTP search_resp = client.post( "/agentcache/search", - json={"query": "observation store", "folderPath": "src/e2e", "agentId": "agent_e2e"}, + json={ + "query": "observation store", + "folderPath": "src/e2e", + "agentId": "agent_e2e", + }, ) assert search_resp.status_code == 200 search_data = search_resp.get_json() @@ -371,10 +416,13 @@ def test_full_lifecycle_e2e_pass(app_client): }, ) assert timeline_resp.status_code == 200 - assert "Refactored observation store" in timeline_resp.get_json()["content"][0]["text"] + assert ( + "Refactored observation store" in timeline_resp.get_json()["content"][0]["text"] + ) # 4. REBUILD via ObservationStore import agentcache.app as app_mod + obs_store = app_mod.observation_store count = obs_store.rebuild_index() assert count >= 1 @@ -388,5 +436,3 @@ def test_full_lifecycle_e2e_pass(app_client): }, ) assert forget_resp.status_code == 200 - - diff --git a/tests/test_observe_core.py b/tests/test_observe_core.py deleted file mode 100644 index d1200545966988eaf6bf3c36a66d8ec3d7043d95..0000000000000000000000000000000000000000 --- a/tests/test_observe_core.py +++ /dev/null @@ -1,211 +0,0 @@ -""" -C1.1 — Unit tests for observe(), strip_private_data(), and folder_observe(). -""" - -import datetime - -import pytest - -from agentcache.db import StateKV -from agentcache.functions import KV, folder_observe, observe, strip_private_data - -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - - -def make_kv(tmp_path): - return StateKV(db_path=str(tmp_path / "test.db")) - - -def _now() -> str: - return ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - - -def valid_observe_payload(**overrides): - p = { - "sessionId": "sess_test_001", - "hookType": "post_tool_use", - "timestamp": _now(), - "data": {"tool_name": "read_file", "tool_input": {"path": "/src/app.py"}}, - } - p.update(overrides) - return p - - -def valid_folder_payload(**overrides): - p = { - "folderPath": "/home/user/projects/myapp", - "agentId": "test-agent", - "text": "Edited the authentication module", - "timestamp": _now(), - } - p.update(overrides) - return p - - -# --------------------------------------------------------------------------- -# observe() — valid payload -# --------------------------------------------------------------------------- - - -class TestObserveValid: - def test_returns_observation_id(self, tmp_path): - kv = make_kv(tmp_path) - result = observe(kv, valid_observe_payload()) - assert "observationId" in result - assert isinstance(result["observationId"], str) - assert len(result["observationId"]) > 0 - - def test_observation_id_has_obs_prefix(self, tmp_path): - kv = make_kv(tmp_path) - result = observe(kv, valid_observe_payload()) - assert result["observationId"].startswith("obs_") - - def test_observation_stored_in_kv(self, tmp_path): - kv = make_kv(tmp_path) - result = observe(kv, valid_observe_payload(sessionId="sess_store_check")) - obs_id = result["observationId"] - stored = kv.get(KV.observations("sess_store_check"), obs_id) - assert stored is not None - assert stored["id"] == obs_id - - -# --------------------------------------------------------------------------- -# observe() — missing required fields -# --------------------------------------------------------------------------- - - -class TestObserveMissingFields: - def test_missing_session_id_raises(self, tmp_path): - kv = make_kv(tmp_path) - with pytest.raises(ValueError): - observe(kv, valid_observe_payload(sessionId="")) - - def test_missing_session_id_none_raises(self, tmp_path): - kv = make_kv(tmp_path) - payload = valid_observe_payload() - del payload["sessionId"] - with pytest.raises(ValueError): - observe(kv, payload) - - def test_missing_hook_type_raises(self, tmp_path): - kv = make_kv(tmp_path) - with pytest.raises(ValueError): - observe(kv, valid_observe_payload(hookType="")) - - def test_missing_hook_type_none_raises(self, tmp_path): - kv = make_kv(tmp_path) - payload = valid_observe_payload() - del payload["hookType"] - with pytest.raises(ValueError): - observe(kv, payload) - - def test_missing_timestamp_raises(self, tmp_path): - kv = make_kv(tmp_path) - with pytest.raises(ValueError): - observe(kv, valid_observe_payload(timestamp="")) - - def test_missing_timestamp_none_raises(self, tmp_path): - kv = make_kv(tmp_path) - payload = valid_observe_payload() - del payload["timestamp"] - with pytest.raises(ValueError): - observe(kv, payload) - - -# --------------------------------------------------------------------------- -# strip_private_data() — redaction -# --------------------------------------------------------------------------- - - -class TestStripPrivateData: - def test_redacts_api_key_assignment(self): - text = "api_key = sk-proj-abc123LONGKEY456789012345678901234567890" - result = strip_private_data(text) - assert "[REDACTED_SECRET]" in result - # Raw key value must not appear - assert "sk-proj-abc123" not in result - - def test_redacts_bearer_token(self): - text = "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c" - result = strip_private_data(text) - assert "[REDACTED_SECRET]" in result - assert "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9" not in result - - def test_redacts_secret_key_pair(self): - text = "secret=mysupersecretvalue12345678901234567890" - result = strip_private_data(text) - assert "[REDACTED_SECRET]" in result - - def test_non_sensitive_text_unchanged(self): - text = "Edited authentication module in src/app.py" - result = strip_private_data(text) - assert result == text - - def test_redacts_private_xml_tags(self): - text = "prefix confidential info suffix" - result = strip_private_data(text) - assert "[REDACTED]" in result - assert "confidential info" not in result - - def test_redacts_google_api_key(self): - # AIza prefix (Google API key pattern) - text = "key = AIzaSyABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789" - result = strip_private_data(text) - assert "[REDACTED_SECRET]" in result - assert "AIzaSy" not in result - - -# --------------------------------------------------------------------------- -# MAX_OBS_PER_SESSION cap -# --------------------------------------------------------------------------- - - -class TestObserveSessionCap: - def test_cap_raises_on_fourth_observation(self, tmp_path, monkeypatch): - monkeypatch.setenv("MAX_OBS_PER_SESSION", "3") - kv = make_kv(tmp_path) - sess = "sess_cap_test" - for _ in range(3): - observe(kv, valid_observe_payload(sessionId=sess)) - with pytest.raises(ValueError, match="limit"): - observe(kv, valid_observe_payload(sessionId=sess)) - - def test_cap_not_triggered_at_limit(self, tmp_path, monkeypatch): - monkeypatch.setenv("MAX_OBS_PER_SESSION", "3") - kv = make_kv(tmp_path) - sess = "sess_cap_ok" - # Exactly 3 should succeed - for _ in range(3): - result = observe(kv, valid_observe_payload(sessionId=sess)) - assert "observationId" in result - - -# --------------------------------------------------------------------------- -# folder_observe() — valid payload / fobs_ prefix -# --------------------------------------------------------------------------- - - -class TestFolderObserveCore: - def test_returns_observation_id(self, tmp_path): - kv = make_kv(tmp_path) - result = folder_observe(kv, valid_folder_payload()) - assert "observationId" in result - - def test_observation_id_has_fobs_prefix(self, tmp_path): - kv = make_kv(tmp_path) - result = folder_observe(kv, valid_folder_payload()) - assert result["observationId"].startswith("fobs_") - - def test_observation_stored_in_kv(self, tmp_path): - kv = make_kv(tmp_path) - result = folder_observe(kv, valid_folder_payload()) - obs_id = result["observationId"] - # normalized: "home/user/projects/myapp" - normalized = "home/user/projects/myapp" - stored = kv.get(KV.folder_obs(normalized, "test-agent"), obs_id) - assert stored is not None - assert stored["id"] == obs_id diff --git a/tests/test_properties.py b/tests/test_properties.py deleted file mode 100644 index f8ea633168f695d54a1679e04d2f7ec6d84e178a..0000000000000000000000000000000000000000 --- a/tests/test_properties.py +++ /dev/null @@ -1,350 +0,0 @@ -""" -tests/test_properties.py — C2.1 - -Hypothesis property-based tests for the folder-based memory system. -All 8 properties from the spec. - -Note: Each property test creates a fresh isolated SQLite DB per hypothesis -example using a shared counter, avoiding state accumulation between examples. -""" - -import datetime -import os -import tempfile - -import pytest - -try: - from hypothesis import HealthCheck, assume, given, settings - from hypothesis import strategies as st - - HYPOTHESIS_AVAILABLE = True -except ImportError: - HYPOTHESIS_AVAILABLE = False - -pytestmark = pytest.mark.skipif( - not HYPOTHESIS_AVAILABLE, - reason="hypothesis not installed — run: pip install hypothesis", -) - -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - -_counter = [0] - - -def _fresh_kv(): - """Create a brand-new isolated StateKV in a temp directory.""" - from agentcache.db import StateKV - - os.environ.pop("AGENTCACHE_SECRET", None) - _counter[0] += 1 - d = tempfile.mkdtemp(prefix=f"agmem_prop_{_counter[0]}_") - return StateKV(db_path=os.path.join(d, "test.db")) - - -def _now(): - return ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - - -def _safe_path(): - """Strategy for valid, non-traversal folder paths.""" - return st.from_regex( - r"[a-zA-Z][a-zA-Z0-9_-]{0,20}/[a-zA-Z][a-zA-Z0-9_-]{0,20}", - fullmatch=True, - ) - - -def _safe_agent(): - return st.from_regex(r"[a-z][a-z0-9_-]{1,12}", fullmatch=True) - - -def _safe_text(): - return st.text( - alphabet=st.characters( - whitelist_categories=("Lu", "Ll", "Nd", "Zs"), - whitelist_characters="_-.,()", - ), - min_size=5, - max_size=200, - ) - - -# --------------------------------------------------------------------------- -# Property 1: Pair Isolation -# Two distinct (folderPath, agentId) pairs never share observations. -# --------------------------------------------------------------------------- - - -@settings(max_examples=50, deadline=None) -@given( - path1=_safe_path(), - agent1=_safe_agent(), - path2=_safe_path(), - agent2=_safe_agent(), - text=_safe_text(), -) -def test_property_1_pair_isolation(path1, agent1, path2, agent2, text): - assume((path1, agent1) != (path2, agent2)) - - from agentcache.functions import KV, folder_observe - - kv = _fresh_kv() - - folder_observe( - kv, {"folderPath": path1, "agentId": agent1, "text": text, "timestamp": _now()} - ) - - scope1 = KV.folder_obs(path1, agent1) - scope2 = KV.folder_obs(path2, agent2) - - obs1_ids = {o["id"] for o in kv.list(scope1)} - obs2_ids = {o["id"] for o in kv.list(scope2)} - - assert obs1_ids.isdisjoint(obs2_ids) - - -# --------------------------------------------------------------------------- -# Property 2: Observation Count Consistency -# meta.obsCount == len(kv.list(folder_obs_scope)) -# --------------------------------------------------------------------------- - - -@settings(max_examples=50, deadline=None) -@given( - path=_safe_path(), - agent=_safe_agent(), - texts=st.lists(_safe_text(), min_size=1, max_size=10), -) -def test_property_2_obs_count_consistency(path, agent, texts): - from agentcache.functions import KV, folder_observe - - kv = _fresh_kv() - - for text in texts: - folder_observe( - kv, - {"folderPath": path, "agentId": agent, "text": text, "timestamp": _now()}, - ) - - meta_scope = KV.folder_meta(path, agent) - meta = kv.get(meta_scope, "meta") - assert meta is not None - - actual_obs = kv.list(KV.folder_obs(path, agent)) - assert meta["obsCount"] == len(actual_obs) - - -# --------------------------------------------------------------------------- -# Property 3: Index Coverage -# Every written pair has a KV.folders entry. -# --------------------------------------------------------------------------- - - -@settings(max_examples=50, deadline=None) -@given( - path=_safe_path(), - agent=_safe_agent(), - text=_safe_text(), -) -def test_property_3_index_coverage(path, agent, text): - from agentcache.functions import KV, folder_observe - - kv = _fresh_kv() - - folder_observe( - kv, {"folderPath": path, "agentId": agent, "text": text, "timestamp": _now()} - ) - - index_entries = kv.list(KV.folders) - normalized_path = path.replace("\\", "/").strip("/") - normalized_agent = agent.strip() - - matching = [ - e - for e in index_entries - if e.get("folderPath") == normalized_path - and e.get("agentId") == normalized_agent - ] - assert len(matching) >= 1 - - -# --------------------------------------------------------------------------- -# Property 4: Privacy Invariant -# No stored obs text contains raw secrets after folder_observe(). -# --------------------------------------------------------------------------- - - -@settings(max_examples=30, deadline=None) -@given( - path=_safe_path(), - agent=_safe_agent(), - prefix=st.text(alphabet="abcdefghijklmnop", min_size=3, max_size=10), -) -def test_property_4_privacy_invariant(path, agent, prefix): - from agentcache.functions import KV, folder_observe - - kv = _fresh_kv() - - secret_text = f"My api_key = sk-proj-{prefix}abc123def456ghi789jkl012 in production" - - folder_observe( - kv, - { - "folderPath": path, - "agentId": agent, - "text": secret_text, - "timestamp": _now(), - }, - ) - - obs_list = kv.list(KV.folder_obs(path, agent)) - for obs in obs_list: - stored_text = obs.get("text", "") - assert "sk-proj-" not in stored_text - - -# --------------------------------------------------------------------------- -# Property 5: Timeline Ordering -# folder_timeline() always returns results sorted newest-first. -# --------------------------------------------------------------------------- - - -@settings(max_examples=40, deadline=None) -@given( - path=_safe_path(), - agent=_safe_agent(), - n=st.integers(min_value=2, max_value=8), -) -def test_property_5_timeline_ordering(path, agent, n): - from agentcache.functions import folder_observe, folder_timeline - - kv = _fresh_kv() - - base_ts = datetime.datetime(2025, 1, 1, 0, 0, 0) - for i in range(n): - ts = (base_ts + datetime.timedelta(minutes=i)).isoformat() + "Z" - folder_observe( - kv, - { - "folderPath": path, - "agentId": agent, - "text": f"Observation number {i}", - "timestamp": ts, - }, - ) - - results = folder_timeline(kv, limit=100, folder_path=path, agent_id=agent) - timestamps = [r["timestamp"] for r in results] - assert timestamps == sorted(timestamps, reverse=True) - - -# --------------------------------------------------------------------------- -# Property 6: Forget Completeness -# After forget({folderPath, agentId}), all three scopes are empty. -# --------------------------------------------------------------------------- - - -@settings(max_examples=40, deadline=None) -@given( - path=_safe_path(), - agent=_safe_agent(), - texts=st.lists(_safe_text(), min_size=1, max_size=5), -) -def test_property_6_forget_completeness(path, agent, texts): - from agentcache.functions import KV, folder_observe, forget - - kv = _fresh_kv() - - for text in texts: - folder_observe( - kv, - {"folderPath": path, "agentId": agent, "text": text, "timestamp": _now()}, - ) - - assert len(kv.list(KV.folder_obs(path, agent))) > 0 - - forget(kv, {"folderPath": path, "agentId": agent}) - - normalized_path = path.replace("\\", "/").strip("/") - normalized_agent = agent.strip() - index_key = f"{normalized_path}:{normalized_agent}" - - assert kv.list(KV.folder_obs(normalized_path, normalized_agent)) == [] - assert kv.get(KV.folder_meta(normalized_path, normalized_agent), "meta") is None - assert kv.get(KV.folders, index_key) is None - - -# --------------------------------------------------------------------------- -# Property 7: Memory Version Uniqueness -# Superseded memories have parentId; at least one memory is always latest. -# --------------------------------------------------------------------------- - - -@settings( - max_examples=30, suppress_health_check=[HealthCheck.filter_too_much], deadline=None -) -@given( - base_content=st.text( - alphabet="abcdefghijklmnopqrstuvwxyz ", - min_size=30, - max_size=80, - ).filter(lambda s: len(s.split()) >= 6), - n_variants=st.integers(min_value=2, max_value=4), -) -def test_property_7_memory_version_uniqueness(base_content, n_variants): - from agentcache.functions import KV, remember - - kv = _fresh_kv() - - for i in range(n_variants): - content = base_content + f" variant {i}" - remember(kv, {"content": content}) - - all_mems = kv.list(KV.memories) - - # Build sets for validation - {m["id"] for m in all_mems} - {m["id"] for m in all_mems if m.get("isLatest") is False} - # Every superseded memory must be referenced by exactly one newer memory via parentId - for m in all_mems: - pid = m.get("parentId") - if pid: - # The parentId must point to a memory that exists and is marked isLatest=False - parent = next((x for x in all_mems if x["id"] == pid), None) - assert parent is not None, f"parentId {pid} not found in memories" - assert parent.get("isLatest") is False, ( - "Parent of superseding memory must be isLatest=False" - ) - - # At least one memory must be latest - latest_count = sum(1 for m in all_mems if m.get("isLatest") is True) - assert latest_count >= 1 - - -# --------------------------------------------------------------------------- -# Property 8: Path Normalization Idempotency -# normalize(normalize(p)) == normalize(p) for all valid inputs. -# --------------------------------------------------------------------------- - - -@settings(max_examples=100, deadline=None) -@given( - path=st.text( - alphabet="abcdefghijklmnopqrstuvwxyz0123456789/_-.", - min_size=2, - max_size=100, - ).filter(lambda p: ".." not in p and p.strip("/") != ""), -) -def test_property_8_path_normalization_idempotency(path): - from agentcache.functions import normalize_folder_path - - try: - normalized_once = normalize_folder_path(path) - normalized_twice = normalize_folder_path(normalized_once) - assert normalized_once == normalized_twice - except ValueError: - pass diff --git a/tests/test_remember.py b/tests/test_remember.py deleted file mode 100644 index 2337203e93855038c9c089d3b1adf8c3b5ac944b..0000000000000000000000000000000000000000 --- a/tests/test_remember.py +++ /dev/null @@ -1,245 +0,0 @@ -""" -tests/test_remember.py — C1.2 - -Tests for remember(), forget(), and jaccard_similarity(). -""" - -import datetime -import os - -import pytest - -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - - -def _make_kv(tmp_path): - from agentcache.db import StateKV - - os.environ.pop("AGENTCACHE_SECRET", None) - return StateKV(db_path=str(tmp_path / "test.db")) - - -def _now(): - return ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - - -# --------------------------------------------------------------------------- -# jaccard_similarity -# --------------------------------------------------------------------------- - - -class TestJaccardSimilarity: - def test_identical_strings(self): - from agentcache.functions import jaccard_similarity - - assert jaccard_similarity("hello world foo", "hello world foo") == 1.0 - - def test_completely_different(self): - from agentcache.functions import jaccard_similarity - - score = jaccard_similarity("apple banana cherry", "xyz uvw qrs") - assert score == 0.0 - - def test_partial_overlap(self): - from agentcache.functions import jaccard_similarity - - score = jaccard_similarity( - "authentication security token", "authentication bearer token" - ) - assert 0.0 < score < 1.0 - - def test_empty_strings(self): - from agentcache.functions import jaccard_similarity - - assert jaccard_similarity("", "") == 1.0 - - def test_high_similarity_above_threshold(self): - from agentcache.functions import jaccard_similarity - - # These two should meet or exceed the 0.7 threshold used in remember() - a = "Use parameterised queries to prevent SQL injection in all database calls" - b = "Use parameterised queries to prevent SQL injection in database operations" - assert jaccard_similarity(a, b) >= 0.7 - - def test_low_similarity_below_threshold(self): - from agentcache.functions import jaccard_similarity - - a = "Configure Redis as the session cache backend" - b = "Deploy the React frontend to Vercel using GitHub Actions CI" - assert jaccard_similarity(a, b) < 0.7 - - -# --------------------------------------------------------------------------- -# remember() -# --------------------------------------------------------------------------- - - -class TestRemember: - def test_creates_memory_with_is_latest_true(self, tmp_path): - from agentcache.functions import remember - - kv = _make_kv(tmp_path) - result = remember(kv, {"content": "Always use type hints in Python functions"}) - assert result["success"] is True - mem = result["memory"] - assert mem["isLatest"] is True - assert mem["id"].startswith("mem_") - assert "Always use type hints" in mem["content"] - - def test_memory_has_required_fields(self, tmp_path): - from agentcache.functions import remember - - kv = _make_kv(tmp_path) - result = remember( - kv, - { - "content": "Prefer composition over inheritance", - "type": "architecture", - "concepts": ["design", "patterns"], - }, - ) - mem = result["memory"] - assert "id" in mem - assert "content" in mem - assert "createdAt" in mem - assert mem["type"] == "architecture" - assert "design" in mem["concepts"] - - def test_supersedes_memory_with_high_jaccard_similarity(self, tmp_path): - from agentcache.functions import KV, remember - - kv = _make_kv(tmp_path) - - first = remember( - kv, - { - "content": "Always use parameterised SQL queries to prevent injection attacks in every database call" - }, - ) - first_id = first["memory"]["id"] - - # Highly similar content — should supersede the first - second = remember( - kv, - { - "content": "Always use parameterised SQL queries to prevent injection attacks in every database operation" - }, - ) - second_mem = second["memory"] - - # Old memory should be marked as not latest - old_mem = kv.get(KV.memories, first_id) - assert old_mem is not None - assert old_mem.get("isLatest") is False - - # New memory should be latest and point to old via parentId - assert second_mem["isLatest"] is True - assert second_mem.get("parentId") == first_id - - def test_independent_memory_with_low_jaccard_similarity(self, tmp_path): - from agentcache.functions import KV, remember - - kv = _make_kv(tmp_path) - - first = remember( - kv, - { - "content": "Configure Redis as the session cache backend for high throughput" - }, - ) - first_id = first["memory"]["id"] - - # Very different content — should be independent - second = remember( - kv, - { - "content": "Deploy the React frontend to Vercel using GitHub Actions continuous deployment" - }, - ) - second_mem = second["memory"] - - # Old memory should remain latest - old_mem = kv.get(KV.memories, first_id) - assert old_mem is not None - assert old_mem.get("isLatest") is True - - # New memory has no parentId - assert second_mem["isLatest"] is True - assert second_mem.get("parentId") is None - - # Both exist independently - all_mems = kv.list(KV.memories) - assert len(all_mems) == 2 - - def test_remember_raises_on_empty_content(self, tmp_path): - from agentcache.functions import remember - - kv = _make_kv(tmp_path) - with pytest.raises(ValueError, match="content is required"): - remember(kv, {"content": ""}) - - def test_remember_strips_private_data(self, tmp_path): - from agentcache.functions import remember - - kv = _make_kv(tmp_path) - result = remember( - kv, - { - "content": "API key is sk-proj-abc123def456ghi789jkl012mno345pqr678 for production" - }, - ) - assert "sk-proj-" not in result["memory"]["content"] - assert "[REDACTED" in result["memory"]["content"] - - def test_remember_with_project_scoping(self, tmp_path): - from agentcache.functions import remember - - kv = _make_kv(tmp_path) - # Two very similar memories for different projects should not supersede each other - first = remember( - kv, - { - "content": "Always use parameterised queries in all database operations", - "project": "project-alpha", - }, - ) - second = remember( - kv, - { - "content": "Always use parameterised queries in all database operations", - "project": "project-beta", - }, - ) - # Both should remain independent (different projects) - assert first["memory"]["isLatest"] is True - assert second["memory"]["isLatest"] is True - - -# --------------------------------------------------------------------------- -# forget() -# --------------------------------------------------------------------------- - - -class TestForget: - def test_forget_memory_by_id(self, tmp_path): - from agentcache.functions import KV, forget, remember - - kv = _make_kv(tmp_path) - result = remember(kv, {"content": "This memory will be forgotten"}) - mem_id = result["memory"]["id"] - - forget_result = forget(kv, {"memoryId": mem_id}) - assert forget_result["deleted"] >= 1 - assert kv.get(KV.memories, mem_id) is None - - def test_forget_returns_zero_for_nonexistent_memory(self, tmp_path): - from agentcache.functions import forget - - kv = _make_kv(tmp_path) - result = forget(kv, {"memoryId": "mem_nonexistent_id"}) - # Should still succeed (memory was already gone) — deleted may be 0 or 1 - assert "deleted" in result diff --git a/tests/test_route_regressions.py b/tests/test_route_regressions.py deleted file mode 100644 index 6e241181a8ce264102d6ac77f9e3b228731cf4ac..0000000000000000000000000000000000000000 --- a/tests/test_route_regressions.py +++ /dev/null @@ -1,741 +0,0 @@ -""" -Route regression tests for the blueprint split (Task A1.3). - -Validates that every endpoint from all 7 blueprints is reachable and returns -the expected HTTP status codes after the monolithic app.py was split into -src/routes/{observations, memories, search, graph, health, mcp, migration}.py. - -Tests use the Flask test client — no running server required. -The AGENTCACHE_SECRET env var is NOT set so all auth checks pass through. -""" - -import datetime -import json -import os - -import pytest - -# Ensure src/ is on the path - - -# --------------------------------------------------------------------------- -# Fixtures -# --------------------------------------------------------------------------- - - -@pytest.fixture(scope="module") -def flask_app(tmp_path_factory): - """ - Create a fully-configured Flask application using create_app(). - - Uses a temp SQLite database so the test suite never touches the real DB. - Background workers are started but are daemon threads — they exit when - the process exits; this does not affect test correctness. - """ - tmp_dir = tmp_path_factory.mktemp("route_regression_db") - db_path = str(tmp_dir / "test.db") - - # Point the server at an isolated DB - os.environ["AGENTCACHE_DB_PATH"] = db_path - # Ensure no auth requirement during tests - os.environ.pop("AGENTCACHE_SECRET", None) - os.environ.pop("AGENTMEMORY_SECRET", None) - - import agentcache.app as app_module - - os.environ.pop("AGENTCACHE_SECRET", None) - os.environ.pop("AGENTMEMORY_SECRET", None) - from agentcache.db import StateKV - - # Patch StateKV to use tmp db before create_app() initialises it - original_init = StateKV.__init__ - - def patched_init(self, db_path=None, **kwargs): - original_init(self, db_path=str(tmp_dir / "test.db"), **kwargs) - - StateKV.__init__ = patched_init - flask_application = app_module.create_app() - StateKV.__init__ = original_init - - flask_application.config["TESTING"] = True - return flask_application - - -@pytest.fixture(scope="module") -def client(flask_app): - """Return a Flask test client bound to the module-scoped app.""" - return flask_app.test_client() - - -# --------------------------------------------------------------------------- -# Helper -# --------------------------------------------------------------------------- - - -def _now_iso() -> str: - return ( - datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z") - ) - - -def _post_json(client, url, payload): - return client.post( - url, - data=json.dumps(payload), - content_type="application/json", - ) - - -# =========================================================================== -# Blueprint 1: health.py -# GET /agentcache/livez -# GET /agentcache/health -# GET /agentcache/audit -# GET /agentcache/config/flags -# =========================================================================== - - -class TestHealthBlueprint: - def test_livez_no_auth_required(self, client): - """GET /livez must respond 200 without any auth token (always open).""" - resp = client.get("/agentcache/livez") - assert resp.status_code == 200 - data = resp.get_json() - assert data["status"] == "ok" - assert "service" in data - - def test_livez_returns_service_name(self, client): - resp = client.get("/agentcache/livez") - data = resp.get_json() - assert data["service"] == "agentcache" - - def test_health_returns_200(self, client): - resp = client.get("/agentcache/health") - assert resp.status_code == 200 - data = resp.get_json() - # Folder-based health check fields (REQ-047) - assert "folderCount" in data - assert "observationCount" in data - assert "memoryCount" in data - - def test_audit_returns_200(self, client): - resp = client.get("/agentcache/audit") - assert resp.status_code == 200 - data = resp.get_json() - assert "entries" in data - - def test_config_flags_returns_200(self, client): - resp = client.get("/agentcache/config/flags") - assert resp.status_code == 200 - data = resp.get_json() - assert "flags" in data - assert "version" in data - - -# =========================================================================== -# Blueprint 2: observations.py -# POST /agentcache/observe -# POST /agentcache/agent/observe -# GET /agentcache/folders -# GET /agentcache/folder/observations -# =========================================================================== - - -class TestObservationsBlueprint: - def test_agent_observe_valid_payload(self, client): - """POST /agent/observe with valid payload returns 201 with observationId.""" - payload = { - "folderPath": "/home/user/proj-test", - "agentId": "kiro", - "text": "Added a new feature to the app", - "timestamp": _now_iso(), - } - resp = _post_json(client, "/agentcache/agent/observe", payload) - assert resp.status_code == 201 - data = resp.get_json() - assert "observationId" in data - assert data["observationId"].startswith("fobs_") - - def test_agent_observe_missing_folder_path_returns_400(self, client): - """POST /agent/observe missing folderPath returns 400.""" - payload = {"agentId": "kiro", "text": "some work", "timestamp": _now_iso()} - resp = _post_json(client, "/agentcache/agent/observe", payload) - assert resp.status_code == 400 - - def test_agent_observe_missing_agent_id_returns_400(self, client): - """POST /agent/observe missing agentId returns 400.""" - payload = { - "folderPath": "/home/user/proj", - "text": "some work", - "timestamp": _now_iso(), - } - resp = _post_json(client, "/agentcache/agent/observe", payload) - assert resp.status_code == 400 - - def test_agent_observe_missing_text_returns_400(self, client): - """POST /agent/observe missing text returns 400.""" - payload = { - "folderPath": "/home/user/proj", - "agentId": "kiro", - "timestamp": _now_iso(), - } - resp = _post_json(client, "/agentcache/agent/observe", payload) - assert resp.status_code == 400 - - def test_folders_list_returns_200(self, client): - """GET /folders returns 200 with a folders list.""" - # Seed at least one observation first - _post_json( - client, - "/agentcache/agent/observe", - { - "folderPath": "/home/user/proj-folders-test", - "agentId": "kiro", - "text": "Folders test observation", - "timestamp": _now_iso(), - }, - ) - resp = client.get("/agentcache/folders") - assert resp.status_code == 200 - data = resp.get_json() - assert "folders" in data - assert isinstance(data["folders"], list) - - def test_folder_observations_returns_200(self, client): - """GET /folder/observations with valid params returns 200.""" - # Seed an observation first - fp = "/home/user/proj-obs-test" - _post_json( - client, - "/agentcache/agent/observe", - { - "folderPath": fp, - "agentId": "kiro", - "text": "Obs for folder/observations test", - "timestamp": _now_iso(), - }, - ) - resp = client.get( - "/agentcache/folder/observations?folderPath=home/user/proj-obs-test&agentId=kiro" - ) - assert resp.status_code == 200 - data = resp.get_json() - assert "observations" in data - - def test_folder_observations_missing_params_returns_400(self, client): - """GET /folder/observations without required params returns 400.""" - resp = client.get("/agentcache/folder/observations") - assert resp.status_code == 400 - - def test_legacy_observe_endpoint_returns_400_or_201(self, client): - """POST /observe endpoint exists (legacy hook) — doesn't 404.""" - payload = { - "folderPath": "/home/user/proj", - "agentId": "kiro", - "text": "legacy observe call", - "timestamp": _now_iso(), - } - resp = _post_json(client, "/agentcache/observe", payload) - # The legacy endpoint exists — expect either success or a controlled error, never 404 - assert resp.status_code != 404 - - -# =========================================================================== -# Blueprint 3: memories.py -# POST /agentcache/remember -# POST /agentcache/agent/remember -# GET /agentcache/memories -# POST /agentcache/forget -# =========================================================================== - - -class TestMemoriesBlueprint: - def test_remember_valid_payload(self, client): - """POST /remember with valid payload returns 201.""" - payload = { - "content": "Always use parameterised queries for SQL", - "type": "fact", - "concepts": ["sql", "security"], - } - resp = _post_json(client, "/agentcache/remember", payload) - assert resp.status_code == 201 - data = resp.get_json() - assert "memory" in data - - def test_agent_cache_valid_payload(self, client): - """POST /agent/remember with content returns 201.""" - payload = { - "content": "The project uses SQLite with WAL mode", - "agentId": "kiro", - "type": "architecture", - "concepts": ["sqlite", "wal"], - } - resp = _post_json(client, "/agentcache/agent/remember", payload) - assert resp.status_code == 201 - - def test_agent_cache_missing_content_returns_400(self, client): - """POST /agent/remember without content returns 400.""" - resp = _post_json(client, "/agentcache/agent/remember", {"agentId": "kiro"}) - assert resp.status_code == 400 - - def test_memories_list_returns_200(self, client): - """GET /memories returns 200 with memories list.""" - resp = client.get("/agentcache/memories") - assert resp.status_code == 200 - data = resp.get_json() - assert "memories" in data - assert isinstance(data["memories"], list) - - def test_memories_list_latest_only(self, client): - """GET /memories?latest=true filters to only latest memories.""" - resp = client.get("/agentcache/memories?latest=true") - assert resp.status_code == 200 - data = resp.get_json() - # All returned memories should be latest=True or isLatest not False - for mem in data["memories"]: - assert mem.get("isLatest") is not False - - def test_forget_memory_by_id(self, client): - """POST /forget with memoryId deletes a global memory.""" - # Create a memory first - create_resp = _post_json( - client, - "/agentcache/remember", - { - "content": "This memory will be forgotten", - "type": "fact", - }, - ) - mem_id = create_resp.get_json()["memory"]["id"] - # Forget it - resp = _post_json(client, "/agentcache/forget", {"memoryId": mem_id}) - assert resp.status_code == 200 - data = resp.get_json() - assert data["deleted"] >= 1 - - def test_forget_folder_pair(self, client): - """POST /forget with folderPath+agentId clears folder observations.""" - # Seed an observation - fp = "/home/user/proj-to-forget" - _post_json( - client, - "/agentcache/agent/observe", - { - "folderPath": fp, - "agentId": "kiro", - "text": "Some work that will be forgotten", - "timestamp": _now_iso(), - }, - ) - resp = _post_json( - client, - "/agentcache/forget", - { - "folderPath": fp, - "agentId": "kiro", - }, - ) - assert resp.status_code == 200 - data = resp.get_json() - assert "deleted" in data - assert data["deleted"] >= 1 - - -# =========================================================================== -# Blueprint 4: search.py -# POST /agentcache/search -# POST /agentcache/timeline -# =========================================================================== - - -class TestSearchBlueprint: - def test_search_with_query_returns_200(self, client): - """POST /search with a query returns 200.""" - # Seed something searchable first - _post_json( - client, - "/agentcache/agent/observe", - { - "folderPath": "/home/user/proj-search-test", - "agentId": "kiro", - "text": "Implemented authentication middleware for the app", - "timestamp": _now_iso(), - }, - ) - resp = _post_json(client, "/agentcache/search", {"query": "authentication"}) - assert resp.status_code == 200 - - def test_search_missing_query_returns_400(self, client): - """POST /search without query returns 400.""" - resp = _post_json(client, "/agentcache/search", {}) - assert resp.status_code == 400 - - def test_search_with_folder_filter(self, client): - """POST /search with folderPath filter returns 200.""" - resp = _post_json( - client, - "/agentcache/search", - { - "query": "test", - "folderPath": "/home/user/proj-search-test", - "agentId": "kiro", - }, - ) - assert resp.status_code == 200 - - def test_timeline_returns_200(self, client): - """POST /timeline returns 200 with observations list.""" - resp = _post_json(client, "/agentcache/timeline", {}) - assert resp.status_code == 200 - data = resp.get_json() - assert "observations" in data - assert isinstance(data["observations"], list) - - def test_timeline_with_filters_returns_200(self, client): - """POST /timeline with folder/agent filters returns 200.""" - resp = _post_json( - client, - "/agentcache/timeline", - { - "folderPath": "/home/user/proj-search-test", - "agentId": "kiro", - "limit": 10, - }, - ) - assert resp.status_code == 200 - - def test_timeline_results_sorted_descending(self, client): - """Timeline results are sorted newest-first (REQ-071).""" - fp = "/home/user/proj-timeline-order" - # Seed observations with distinct timestamps - for i in range(3): - ts = datetime.datetime(2025, 6, 1, 10, i, 0).isoformat() + "Z" - _post_json( - client, - "/agentcache/agent/observe", - { - "folderPath": fp, - "agentId": "kiro", - "text": f"Observation {i}", - "timestamp": ts, - }, - ) - resp = _post_json( - client, - "/agentcache/timeline", - { - "folderPath": fp, - "agentId": "kiro", - }, - ) - assert resp.status_code == 200 - data = resp.get_json() - timestamps = [o["timestamp"] for o in data["observations"]] - assert timestamps == sorted(timestamps, reverse=True) - - -# =========================================================================== -# Blueprint 5: graph.py -# GET /agentcache/graph -# GET /agentcache/graph/stats -# POST /agentcache/graph/query -# POST /agentcache/graph/build -# =========================================================================== - - -class TestGraphBlueprint: - def test_graph_returns_200(self, client): - """GET /graph returns 200 with nodes and edges.""" - resp = client.get("/agentcache/graph") - assert resp.status_code == 200 - data = resp.get_json() - assert "nodes" in data - assert "edges" in data - - def test_graph_stats_returns_200(self, client): - """GET /graph/stats returns 200.""" - resp = client.get("/agentcache/graph/stats") - assert resp.status_code == 200 - data = resp.get_json() - assert "nodes" in data - assert "edges" in data - - def test_graph_query_returns_200(self, client): - """POST /graph/query returns 200.""" - resp = _post_json(client, "/agentcache/graph/query", {}) - assert resp.status_code == 200 - - def test_graph_build_returns_200(self, client): - """POST /graph/build returns 200.""" - resp = _post_json(client, "/agentcache/graph/build", {}) - assert resp.status_code == 200 - - def test_graph_nodes_have_required_fields(self, client): - """Graph nodes contain id, label, folderPath, agentIds, obsCount, color (REQ-024).""" - # Seed some data - _post_json( - client, - "/agentcache/agent/observe", - { - "folderPath": "/home/user/proj-graph-check", - "agentId": "kiro", - "text": "Graph node field check", - "timestamp": _now_iso(), - }, - ) - resp = client.get("/agentcache/graph") - data = resp.get_json() - if data["nodes"]: - node = data["nodes"][0] - for field in ("id", "label", "folderPath", "agentIds", "obsCount", "color"): - assert field in node, f"Missing field '{field}' on graph node" - - -# =========================================================================== -# Blueprint 6: mcp.py -# GET /agentcache/mcp/tools -# POST /agentcache/mcp/tools -# =========================================================================== - - -class TestMcpBlueprint: - def test_mcp_tools_list_returns_200(self, client): - """GET /mcp/tools returns 200 with a tools list.""" - resp = client.get("/agentcache/mcp/tools") - assert resp.status_code == 200 - data = resp.get_json() - assert "tools" in data - assert isinstance(data["tools"], list) - assert len(data["tools"]) > 0 - - def test_mcp_tools_contains_required_tools(self, client): - """The expected MCP tools are all present in the schema.""" - resp = client.get("/agentcache/mcp/tools") - tool_names = {t["name"] for t in resp.get_json()["tools"]} - required_tools = { - "agent_observe", - "cache_recall", - "cache_save", - "agent_cache", - "cache_diagnose", - "cache_forget", - "cache_export", - "cache_smart_search", - "cache_folders", - "cache_folder_observations", - "cache_timeline", - } - assert required_tools.issubset(tool_names), ( - f"Missing tools: {required_tools - tool_names}" - ) - - def test_mcp_tool_call_agent_observe(self, client): - """POST /mcp/tools agent_observe returns 200.""" - payload = { - "name": "agent_observe", - "arguments": { - "folderPath": "/home/user/proj-mcp-test", - "agentId": "kiro", - "text": "MCP observe test", - "timestamp": _now_iso(), - }, - } - resp = _post_json(client, "/agentcache/mcp/tools", payload) - assert resp.status_code == 200 - data = resp.get_json() - assert "content" in data - - def test_mcp_tool_call_cache_recall(self, client): - """POST /mcp/tools cache_recall returns 200.""" - resp = _post_json( - client, - "/agentcache/mcp/tools", - { - "name": "cache_recall", - "arguments": {"query": "authentication"}, - }, - ) - assert resp.status_code == 200 - - def test_mcp_tool_call_cache_diagnose(self, client): - """POST /mcp/tools cache_diagnose returns folderCount etc.""" - resp = _post_json( - client, - "/agentcache/mcp/tools", - { - "name": "cache_diagnose", - "arguments": {}, - }, - ) - assert resp.status_code == 200 - data = resp.get_json() - result = json.loads(data["content"][0]["text"]) - assert "folderCount" in result - assert "observationCount" in result - - def test_mcp_tool_call_cache_folders(self, client): - """POST /mcp/tools cache_folders returns list.""" - resp = _post_json( - client, - "/agentcache/mcp/tools", - { - "name": "cache_folders", - "arguments": {}, - }, - ) - assert resp.status_code == 200 - - def test_mcp_tool_call_cache_timeline(self, client): - """POST /mcp/tools cache_timeline returns list.""" - resp = _post_json( - client, - "/agentcache/mcp/tools", - { - "name": "cache_timeline", - "arguments": {"limit": 10}, - }, - ) - assert resp.status_code == 200 - - def test_mcp_tool_unknown_name_returns_400(self, client): - """POST /mcp/tools with unknown tool name returns 400.""" - resp = _post_json( - client, - "/agentcache/mcp/tools", - { - "name": "nonexistent_tool_xyz", - "arguments": {}, - }, - ) - assert resp.status_code == 400 - - def test_mcp_tool_missing_name_returns_400(self, client): - """POST /mcp/tools without name returns 400.""" - resp = _post_json(client, "/agentcache/mcp/tools", {"arguments": {}}) - assert resp.status_code == 400 - - -# =========================================================================== -# Blueprint 7: migration.py -# POST /agentcache/migrate -# =========================================================================== - - -class TestMigrationBlueprint: - def test_migrate_dry_run_returns_200(self, client): - """POST /migrate with dry_run=true returns 200 with counts.""" - resp = _post_json(client, "/agentcache/migrate", {"dry_run": True}) - assert resp.status_code == 200 - data = resp.get_json() - assert "migrated_sessions" in data - assert "migrated_observations" in data - assert "errors" in data - - def test_migrate_without_body_defaults_to_no_dry_run(self, client): - """POST /migrate with empty body returns 200.""" - resp = _post_json(client, "/agentcache/migrate", {}) - assert resp.status_code == 200 - - -# =========================================================================== -# Blueprint registration completeness -# =========================================================================== - - -class TestBlueprintRegistration: - def test_all_expected_endpoints_registered(self, flask_app): - """All 7 blueprints contribute at least one route each.""" - registered_rules = {rule.rule for rule in flask_app.url_map.iter_rules()} - - # observations.py - assert "/agentcache/agent/observe" in registered_rules - assert "/agentcache/folders" in registered_rules - assert "/agentcache/folder/observations" in registered_rules - - # memories.py - assert "/agentcache/remember" in registered_rules - assert "/agentcache/memories" in registered_rules - assert "/agentcache/forget" in registered_rules - - # search.py - assert "/agentcache/search" in registered_rules - assert "/agentcache/timeline" in registered_rules - - # graph.py - assert "/agentcache/graph" in registered_rules - assert "/agentcache/graph/stats" in registered_rules - assert "/agentcache/graph/query" in registered_rules - - # health.py - assert "/agentcache/livez" in registered_rules - assert "/agentcache/health" in registered_rules - assert "/agentcache/audit" in registered_rules - assert "/agentcache/config/flags" in registered_rules - - # mcp.py - assert "/agentcache/mcp/tools" in registered_rules - - # migration.py - assert "/agentcache/migrate" in registered_rules - - def test_no_routes_return_404_from_blueprints(self, client): - """None of the expected blueprint routes return 404.""" - get_routes = [ - "/agentcache/livez", - "/agentcache/health", - "/agentcache/folders", - "/agentcache/memories", - "/agentcache/graph", - "/agentcache/graph/stats", - "/agentcache/audit", - "/agentcache/config/flags", - "/agentcache/mcp/tools", - ] - for route in get_routes: - resp = client.get(route) - assert resp.status_code != 404, f"Route {route} returned 404" - - def test_cors_headers_present_on_responses(self, client): - """CORS after_request hook adds the expected headers.""" - resp = client.get( - "/agentcache/livez", - headers={"Origin": "http://localhost:3000"}, - ) - assert resp.status_code == 200 - # The CORS hook in create_app() should have added these headers - assert "Access-Control-Allow-Headers" in resp.headers - assert "Access-Control-Allow-Methods" in resp.headers - - def test_auth_returns_401_when_secret_set(self, client, flask_app): - """When AGENTCACHE_SECRET is configured, missing token returns 401 on protected routes. - - /agentcache/audit requires auth; /agentcache/livez and /agentcache/health are open. - """ - os.environ["AGENTCACHE_SECRET"] = "test-secret-token" - try: - resp = client.get("/agentcache/audit") - assert resp.status_code == 401 - finally: - del os.environ["AGENTCACHE_SECRET"] - - def test_auth_passes_with_correct_bearer_token(self, client, flask_app): - """Correct Bearer token passes auth on protected endpoints.""" - secret = "test-secret-token-correct" - os.environ["AGENTCACHE_SECRET"] = secret - try: - resp = client.get( - "/agentcache/audit", - headers={"Authorization": f"Bearer {secret}"}, - ) - assert resp.status_code == 200 - finally: - del os.environ["AGENTCACHE_SECRET"] - - def test_livez_always_open_regardless_of_secret(self, client): - """GET /livez is always unauthenticated, even when secret is set (REQ-057).""" - os.environ["AGENTCACHE_SECRET"] = "some-secret" - try: - resp = client.get("/agentcache/livez") - assert resp.status_code == 200 - finally: - del os.environ["AGENTCACHE_SECRET"] diff --git a/tests/test_search.py b/tests/test_search.py deleted file mode 100644 index 4a4c0bdd3e7207566600ccc9115cda22a2154040..0000000000000000000000000000000000000000 --- a/tests/test_search.py +++ /dev/null @@ -1,268 +0,0 @@ -""" -tests/test_search.py — C1.3 - -Tests for SearchIndex, HybridSearch, and synonym expansion. -""" - - -# --------------------------------------------------------------------------- -# SearchIndex unit tests -# --------------------------------------------------------------------------- - - -class TestSearchIndex: - def _make_obs(self, obs_id, title, narrative="", concepts=None, obs_type="other"): - return { - "id": obs_id, - "sessionId": "sess_test", - "title": title, - "narrative": narrative, - "concepts": concepts or [], - "files": [], - "type": obs_type, - } - - def test_add_and_exact_match_returns_rank_one(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - obs = self._make_obs("obs_001", "authentication middleware refactor") - idx.add(obs) - results = idx.search("authentication middleware") - assert len(results) > 0 - assert results[0]["obsId"] == "obs_001" - - def test_prefix_matching(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - obs = self._make_obs("obs_002", "authentication token validation") - idx.add(obs) - results = idx.search("authen") - assert any(r["obsId"] == "obs_002" for r in results) - - def test_synonym_expansion_db_conn(self): - """'db conn' should find document indexed with 'database connection'.""" - from agentcache.search import SearchIndex - - idx = SearchIndex() - obs = self._make_obs( - "obs_003", - "database connection pooling setup", - "configure database connection pool", - ) - idx.add(obs) - results = idx.search("db conn") - assert any(r["obsId"] == "obs_003" for r in results) - - def test_remove_document(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - obs = self._make_obs("obs_004", "deploy kubernetes service mesh") - idx.add(obs) - idx.remove("obs_004") - results = idx.search("kubernetes") - assert not any(r["obsId"] == "obs_004" for r in results) - - def test_empty_index_returns_empty(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - results = idx.search("anything") - assert results == [] - - def test_multiple_docs_rank_order(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - idx.add( - self._make_obs( - "obs_a", "authentication login system", "user authentication flow" - ) - ) - idx.add( - self._make_obs( - "obs_b", "database migration script", "run database migration" - ) - ) - idx.add( - self._make_obs( - "obs_c", "deployment pipeline CI", "CI CD pipeline deployment" - ) - ) - - results = idx.search("authentication") - assert results[0]["obsId"] == "obs_a" - - def test_size_property(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - assert idx.size == 0 - idx.add(self._make_obs("x1", "title one")) - idx.add(self._make_obs("x2", "title two")) - assert idx.size == 2 - idx.remove("x1") - assert idx.size == 1 - - def test_clear(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - idx.add(self._make_obs("x1", "something")) - idx.clear() - assert idx.size == 0 - assert idx.search("something") == [] - - def test_dirty_flag_set_on_add(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - assert idx._dirty is False - idx.add(self._make_obs("x1", "test dirty flag")) - assert idx._dirty is True - - def test_dirty_flag_set_on_remove(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - idx.add(self._make_obs("x1", "test remove dirty")) - idx._dirty = False # reset manually - idx.remove("x1") - assert idx._dirty is True - - def test_dirty_flag_reset_after_restore(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - idx.add(self._make_obs("x1", "test restore")) - data = idx.serialize_data() - idx2 = SearchIndex() - idx2.restore_from_data(data) - assert idx2._dirty is False - - def test_has_method(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - obs = self._make_obs("obs_has", "has method test") - assert not idx.has("obs_has") - idx.add(obs) - assert idx.has("obs_has") - idx.remove("obs_has") - assert not idx.has("obs_has") - - -# --------------------------------------------------------------------------- -# VectorIndex unit tests -# --------------------------------------------------------------------------- - - -class TestVectorIndex: - def test_dirty_flag_on_add(self): - from agentcache.search import VectorIndex - - vi = VectorIndex() - assert vi._dirty is False - vi.add("v1", "sess", [0.1, 0.2, 0.3]) - assert vi._dirty is True - - def test_dirty_flag_on_remove(self): - from agentcache.search import VectorIndex - - vi = VectorIndex() - vi.add("v1", "sess", [0.1, 0.2, 0.3]) - vi._dirty = False - vi.remove("v1") - assert vi._dirty is True - - def test_dirty_flag_reset_after_restore(self): - from agentcache.search import VectorIndex - - vi = VectorIndex() - vi.add("v1", "sess", [0.1, 0.2, 0.3]) - data = vi.serialize_data() - vi2 = VectorIndex() - vi2.restore_from_data(data) - assert vi2._dirty is False - - -# --------------------------------------------------------------------------- -# HybridSearch in BM25-only mode -# --------------------------------------------------------------------------- - - -class TestHybridSearchBM25Only: - def _make_obs(self, obs_id, title, narrative=""): - return { - "id": obs_id, - "sessionId": "sess_hybrid", - "title": title, - "narrative": narrative, - "concepts": [], - "files": [], - "type": "other", - } - - def test_hybrid_bm25_only_returns_same_results_as_search_index(self): - from agentcache.search import HybridSearch, SearchIndex, VectorIndex - - bm25 = SearchIndex() - vector = VectorIndex() - - docs = [ - self._make_obs("h1", "authentication middleware implementation"), - self._make_obs("h2", "database migration scripts"), - self._make_obs("h3", "deployment kubernetes configuration"), - ] - for d in docs: - bm25.add(d) - - # HybridSearch with no embedding provider — BM25 only - hybrid = HybridSearch(bm25, vector, None, None) - bm25_direct = bm25.search("authentication", 10) - hybrid_results = hybrid.search("authentication", 10) - - bm25_ids = [r["obsId"] for r in bm25_direct] - hybrid_ids = [r["obsId"] for r in hybrid_results] - - # The same document should appear at the top in both - assert bm25_ids[0] == hybrid_ids[0] - - def test_hybrid_returns_empty_for_no_matches(self): - from agentcache.search import HybridSearch, SearchIndex, VectorIndex - - bm25 = SearchIndex() - hybrid = HybridSearch(bm25, VectorIndex(), None, None) - assert hybrid.search("zzznomatch", 10) == [] - - -# --------------------------------------------------------------------------- -# Serialization round-trip -# --------------------------------------------------------------------------- - - -class TestSearchIndexSerialization: - def test_roundtrip_preserves_search_results(self): - from agentcache.search import SearchIndex - - idx = SearchIndex() - idx.add( - { - "id": "rt_001", - "sessionId": "sess_rt", - "title": "serialization round trip test", - "narrative": "verify that index survives serialize/restore", - "concepts": ["test"], - "files": [], - "type": "other", - } - ) - - data = idx.serialize_data() - idx2 = SearchIndex() - idx2.restore_from_data(data) - - results = idx2.search("serialization round trip") - assert any(r["obsId"] == "rt_001" for r in results) diff --git a/tests/test_search_service.py b/tests/test_search_service.py index e6f1f02f08a4aa7121dec48f08170b4ae6a1ed63..acc1a1126e6e724ce93f7ca0bbbd3f920855413b 100644 --- a/tests/test_search_service.py +++ b/tests/test_search_service.py @@ -1,9 +1,5 @@ -import os -import pytest -from agentcache.core import KV, SearchService, IndexPersistence -from agentcache.db import StateKV +from agentcache.core import KV, SearchService from agentcache.search import SearchIndex -from agentcache.app import create_app, init_services, search_service as global_search_service def test_kv_scopes_import(): @@ -37,11 +33,19 @@ def test_search_service_index_remove_and_search(tmp_db): } # Populate KV store for hydration - kv.set(KV.folders, "src/auth:agent_alpha", {"folderPath": "src/auth", "agentId": "agent_alpha"}) + kv.set( + KV.folders, + "src/auth:agent_alpha", + {"folderPath": "src/auth", "agentId": "agent_alpha"}, + ) kv.set(KV.folder_obs("src/auth", "agent_alpha"), "obs_1", obs1) kv.set(KV.obs_lookup, "obs_1", {"folderPath": "src/auth", "agentId": "agent_alpha"}) - kv.set(KV.folders, "src/db:agent_beta", {"folderPath": "src/db", "agentId": "agent_beta"}) + kv.set( + KV.folders, + "src/db:agent_beta", + {"folderPath": "src/db", "agentId": "agent_beta"}, + ) kv.set(KV.folder_obs("src/db", "agent_beta"), "obs_2", obs2) kv.set(KV.obs_lookup, "obs_2", {"folderPath": "src/db", "agentId": "agent_beta"}) @@ -115,6 +119,7 @@ def test_http_search_routes_end_to_end(app_client): client = app_client import agentcache.app as app_mod + kv = app_mod.kv search_svc = app_mod.search_service @@ -126,9 +131,15 @@ def test_http_search_routes_end_to_end(app_client): "agentId": "agent_gamma", } - kv.set(KV.folders, "src/api:agent_gamma", {"folderPath": "src/api", "agentId": "agent_gamma"}) + kv.set( + KV.folders, + "src/api:agent_gamma", + {"folderPath": "src/api", "agentId": "agent_gamma"}, + ) kv.set(KV.folder_obs("src/api", "agent_gamma"), "obs_http_1", obs) - kv.set(KV.obs_lookup, "obs_http_1", {"folderPath": "src/api", "agentId": "agent_gamma"}) + kv.set( + KV.obs_lookup, "obs_http_1", {"folderPath": "src/api", "agentId": "agent_gamma"} + ) search_svc.index(obs) @@ -142,7 +153,9 @@ def test_http_search_routes_end_to_end(app_client): assert data[0]["folderPath"] == "src/api" # POST /agentmemory/search (alias endpoint) - res_alias = client.post("/agentmemory/search", json={"query": "OAuth2 bearer token", "limit": 5}) + res_alias = client.post( + "/agentmemory/search", json={"query": "OAuth2 bearer token", "limit": 5} + ) assert res_alias.status_code == 200 data_alias = res_alias.get_json() assert len(data_alias) == 1 @@ -157,4 +170,3 @@ def test_http_search_routes_end_to_end(app_client): res_alias_no_query = client.post("/agentmemory/search", json={}) assert res_alias_no_query.status_code == 400 - diff --git a/tests/test_security.py b/tests/test_security.py deleted file mode 100644 index 3ddccb49a6ea6549181fb6d093b74e7106bbc796..0000000000000000000000000000000000000000 --- a/tests/test_security.py +++ /dev/null @@ -1,84 +0,0 @@ -""" -tests/test_security.py - -Advanced security and injection checks for the agentcache-python project (A2.3, A3.1). -""" - -import os -import sys - -sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src")) - -import pytest - -from agentcache.db import StateKV -from agentcache.storage.paths import normalize_folder_path - - -@pytest.fixture -def temp_db(tmp_path): - db_file = tmp_path / "security_test.db" - return StateKV(db_path=str(db_file)) - - -def test_sql_injection_get_set_delete(temp_db): - """Verify that parameterized SQL queries prevent SQL injection payloads from hijacking queries.""" - # List of classic SQL injection strings - payloads = [ - "' OR '1'='1", - "'; DROP TABLE kv_store; --", - "UNION SELECT 'a', 'b', 'c'", - '" OR ""="', - "'; INSERT INTO kv_store VALUES ('abc', 'xyz', '123'); --", - "'; DELETE FROM kv_store; --", - ] - - # Store a benign test key to monitor database state integrity - temp_db.set("mem:folders", "benign_key", {"name": "test_folder"}) - - for payload in payloads: - # 1. Check get operations - # Malicious key/scope should simply return None (no SQL exception, no data leaks) - res = temp_db.get("mem:folders", payload) - assert res is None - - # 2. Check set operations - # Malicious key/scope should be safely stored as a literal string value without executing SQL commands - temp_db.set(payload, "malicious_key", {"data": payload}) - retrieved = temp_db.get(payload, "malicious_key") - assert retrieved == {"data": payload, "id": "malicious_key"} - - # 3. Check delete operations - # Malicious deletion should not delete the benign data - temp_db.delete("mem:folders", payload) - - # Verify database state integrity remains untouched - benign_data = temp_db.get("mem:folders", "benign_key") - assert benign_data == {"name": "test_folder", "id": "benign_key"} - - -def test_path_traversal_payloads(): - """Verify that normalize_folder_path rejects all variations of directory traversal payloads.""" - dangerous_paths = [ - "../../etc/passwd", - "projects/../../etc/shadow", - "C:\\..\\Windows\\System32\\cmd.exe", - "projects/myapp/../../..", - "../", - "..", - "/../", - "\\\\server\\share\\..\\file", - "a/b/c/../../../../d", - ] - - for path in dangerous_paths: - with pytest.raises(ValueError, match="path traversal segment '..'"): - normalize_folder_path(path) - - -def test_empty_and_invalid_slash_rejections(): - """Verify that empty inputs and paths consisting only of slashes raise ValueError.""" - invalid_paths = ["", "///", "////"] - for path in invalid_paths: - with pytest.raises(ValueError): - normalize_folder_path(path) diff --git a/tests/test_timeline.py b/tests/test_timeline.py deleted file mode 100644 index f12514b37007cc600c01d1aad28ecb1f68bbbe17..0000000000000000000000000000000000000000 --- a/tests/test_timeline.py +++ /dev/null @@ -1,99 +0,0 @@ -"""Unit tests for folder_timeline (REQ-020, REQ-021, REQ-022, REQ-071).""" - -import datetime -import os - -from agentcache.db import StateKV -from agentcache.functions import folder_observe, folder_timeline - - -def make_kv(tmp_path): - db_path = os.path.join(str(tmp_path), "test.db") - return StateKV(db_path=db_path) - - -def ts(offset_seconds=0): - dt = datetime.datetime(2025, 1, 15, 10, 0, 0) + datetime.timedelta( - seconds=offset_seconds - ) - return dt.isoformat() + "Z" - - -def add_obs(kv, folder="/home/user/proj", agent="kiro", timestamp=None, text="obs"): - return folder_observe( - kv, - { - "folderPath": folder, - "agentId": agent, - "text": text, - "timestamp": timestamp or ts(), - }, - ) - - -class TestTimelineOrdering: - def test_results_sorted_desc(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, timestamp=ts(0)) - add_obs(kv, timestamp=ts(60)) - add_obs(kv, timestamp=ts(30)) - results = folder_timeline(kv) - timestamps = [r["timestamp"] for r in results] - assert timestamps == sorted(timestamps, reverse=True) - - def test_empty_returns_empty(self, tmp_path): - kv = make_kv(tmp_path) - results = folder_timeline(kv) - assert results == [] - - -class TestTimelineLimit: - def test_limit_respected(self, tmp_path): - kv = make_kv(tmp_path) - for i in range(10): - add_obs(kv, timestamp=ts(i), text=f"obs {i}") - results = folder_timeline(kv, limit=5) - assert len(results) == 5 - - def test_default_limit_100(self, tmp_path): - kv = make_kv(tmp_path) - for i in range(150): - add_obs(kv, timestamp=ts(i), text=f"obs {i}") - results = folder_timeline(kv) - assert len(results) == 100 - - -class TestTimelineFilters: - def test_folder_filter(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, folder="/home/user/proj-a", timestamp=ts(0)) - add_obs(kv, folder="/home/user/proj-b", timestamp=ts(1)) - results = folder_timeline(kv, folder_path="home/user/proj-a") - assert all(r["folderPath"] == "home/user/proj-a" for r in results) - assert len(results) == 1 - - def test_agent_filter(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, agent="kiro", timestamp=ts(0)) - add_obs(kv, agent="claude", timestamp=ts(1)) - results = folder_timeline(kv, agent_id="kiro") - assert all(r["agentId"] == "kiro" for r in results) - - def test_before_filter(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, timestamp=ts(0)) # 10:00:00 - add_obs(kv, timestamp=ts(60)) # 10:01:00 - add_obs(kv, timestamp=ts(120)) # 10:02:00 - results = folder_timeline(kv, before=ts(90)) - # Should only include obs before 10:01:30 - for r in results: - assert r["timestamp"] < ts(90) - - def test_after_filter(self, tmp_path): - kv = make_kv(tmp_path) - add_obs(kv, timestamp=ts(0)) - add_obs(kv, timestamp=ts(60)) - add_obs(kv, timestamp=ts(120)) - results = folder_timeline(kv, after=ts(30)) - for r in results: - assert r["timestamp"] > ts(30)