agharsallah commited on
Commit
c3f5c19
Β·
1 Parent(s): 9dd6dab

feat: add optional mem0 cloud backend for memory index and update documentation

Browse files
.env.example CHANGED
@@ -55,5 +55,18 @@ DATABASE_URL=
55
  MEMORY_INDEX=
56
  MEMORY_INDEX_CONFIG=
57
 
 
 
 
 
 
 
 
 
 
 
 
 
 
58
  # Gradio server port (auto-detects a free port in 7960-8059 if unset).
59
  GRADIO_SERVER_PORT=
 
55
  MEMORY_INDEX=
56
  MEMORY_INDEX_CONFIG=
57
 
58
+ # Hosted mem0 platform backend (ADR-0020), opt-in. Set MEMORY_INDEX=cloud (or
59
+ # MEMORY_INDEX_BACKEND=cloud) to use mem0's managed service instead of the local
60
+ # embedder. NOTE: this sends ledger event text to mem0's servers β€” a deliberate
61
+ # departure from the off-the-grid default; the local backend stays the default.
62
+ # MEMORY_INDEX=cloud
63
+ # MEM0_API_KEY=m0-... # required for the hosted backend
64
+ MEM0_API_KEY=
65
+ # Optional hosted scoping:
66
+ # MEM0_ORG_ID=
67
+ # MEM0_PROJECT_ID=
68
+ # MEM0_HOST=
69
+ # MEMORY_INDEX_BACKEND= # set to "cloud" to force the hosted backend
70
+
71
  # Gradio server port (auto-detects a free port in 7960-8059 if unset).
72
  GRADIO_SERVER_PORT=
.gitignore CHANGED
@@ -8,4 +8,5 @@ __pycache__/
8
  scripts/
9
  runs/
10
  *.db
11
- CLAUDE.md
 
 
8
  scripts/
9
  runs/
10
  *.db
11
+ CLAUDE.md
12
+ ui/raw
docs/adr/0020-mem0-hosted-cloud-memory-index.md ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ADR-0020: Optional mem0 Hosted-Cloud Backend for the Memory Index
2
+
3
+ ## Status
4
+
5
+ Accepted (extends [ADR-0018](0018-layered-semantic-memory-index.md); relates to
6
+ [ADR-0019](0019-single-model-catalogue-no-cloud-path.md))
7
+
8
+ ## Context
9
+
10
+ ADR-0018 added an optional **semantic memory index** as a *derived, rebuildable
11
+ lens* over the append-only ledger, behind a two-method `MemoryIndex` protocol
12
+ (`index(events)` / `search(query, k)`). ADR-0018 explicitly anticipated
13
+ **alternative backends behind the same protocol**, and ADR-0019 set the default
14
+ embedder to a **local sentence-transformers** model so an activated index stays
15
+ *off the grid* β€” no API key, nothing leaves the machine.
16
+
17
+ The local backend (`Mem0MemoryIndex`, wrapping the OSS `mem0.Memory`) is the right
18
+ default, but some deployments want mem0's **managed platform** (api.mem0.ai)
19
+ instead:
20
+
21
+ - no local embedder or vector store to host (mem0 runs both);
22
+ - memory that **persists across processes and runs** in one managed store, rather
23
+ than an in-process index rebuilt per boot;
24
+ - access to mem0's hosted retrieval features and org/project scoping.
25
+
26
+ The request is "can we use the mem0 hosted cloud version." The protocol already
27
+ makes room for it; the only real design question is how to add it **without
28
+ compromising the off-the-grid default** or the ledger-is-truth invariant.
29
+
30
+ ## Decision
31
+
32
+ Add a second backend, **`Mem0CloudIndex`**, that wraps the mem0 platform client
33
+ (`mem0.MemoryClient`) behind the *same* `MemoryIndex` protocol. It is **strictly
34
+ opt-in and never the default**.
35
+
36
+ **Shared base, two thin backends.** Factor the protocol surface β€” idempotent
37
+ upsert keyed by `event.id`, verbatim storage (`infer=False`), and search-hit β†’
38
+ `Event` reconstruction from metadata β€” into a private `_Mem0BackendBase`. The two
39
+ backends differ only in three variation points: how the client is built
40
+ (`Memory.from_config` vs `MemoryClient(...)`), how one event is stored, and how a
41
+ query is run. The invariants from ADR-0018 (derived, rebuildable, ledger is truth,
42
+ visibility filter unchanged) hold identically for both β€” only *where* the
43
+ embedding and retrieval happen differs.
44
+
45
+ **Same contract, verbatim storage.** The cloud backend stores each event as one
46
+ chat-style user turn with `infer=False`, so mem0's generative LLM never rewrites
47
+ or extracts β€” the ledger text is what is embedded, and the full event rides in
48
+ `metadata` for reconstruction. Dedup is process-local by `event.id`, *before* the
49
+ client is built, so re-indexing the same ledger slice never re-embeds and an
50
+ already-indexed id needs no network call.
51
+
52
+ **Selection by env, local stays default.** `memory_index_from_env()`:
53
+
54
+ | `MEMORY_INDEX` | `MEMORY_INDEX_BACKEND` | Result |
55
+ |---|---|---|
56
+ | unset / falsey | β€” | `None` β€” offline keyword path (default; suite default) |
57
+ | `1` / `true` / `local` / `mem0` / `on` | β€” | `Mem0MemoryIndex` (local, off the grid) |
58
+ | `cloud` / `mem0-cloud` / `platform` / `hosted` | β€” | `Mem0CloudIndex` (hosted) |
59
+ | any | `cloud` | `Mem0CloudIndex` (explicit backend wins) |
60
+
61
+ The cloud backend reads `MEM0_API_KEY` (required), and optional `MEM0_ORG_ID` /
62
+ `MEM0_PROJECT_ID` / `MEM0_HOST`, from the environment. `mem0` is lazy-imported
63
+ inside the backend, so the offline path needs neither the package nor a key, and a
64
+ missing key fails loudly on first use β€” not at import.
65
+
66
+ **No new dependency.** `MemoryClient` ships in the same `mem0ai` package already
67
+ declared by the optional `memory` extra; the cloud path does not need
68
+ `sentence-transformers` (embeddings are server-side).
69
+
70
+ ## Off-the-grid reconciliation (ADR-0019)
71
+
72
+ ADR-0019 made the engine off the grid *by default* β€” local embeddings, no cloud
73
+ key path for inference or memory. This ADR does not reverse that: it adds an
74
+ **opt-in** that is dormant unless explicitly selected. **Activating
75
+ `Mem0CloudIndex` sends ledger event text to mem0's servers** β€” a deliberate,
76
+ clearly-flagged departure from the default, documented in the backend docstring,
77
+ `config` comments, and `memory-stack.md`. With the gate unset or set to a local
78
+ spelling, behaviour is byte-for-byte what ADR-0019 specified.
79
+
80
+ ## Consequences
81
+
82
+ - **Same protocol, no blast radius.** `SalienceMemory`, `ManifestAgent._recall`,
83
+ the registry wiring, and the agent contract are untouched β€” they still see a
84
+ `MemoryIndex`. Swapping local ↔ cloud is one env var.
85
+ - **Tradeoffs the operator opts into.** Cloud means data egress (ledger text
86
+ leaves the machine), a network dependency on api.mem0.ai, and per-call latency
87
+ and cost on mem0's side. In exchange: managed embeddings + vector store, and
88
+ memory that persists across processes/runs rather than rebuilding in-process.
89
+ - **Tests mirror the local tiers, all green offline.** Protocol conformance, env
90
+ selection (each cloud spelling + backend override + credential plumbing), and
91
+ idempotent dedup are exercised with `mem0` absent and no key. A real hosted
92
+ round-trip is guarded behind `MEM0_API_KEY` + `MEM0_CLOUD_E2E` and skipped
93
+ otherwise. The suite stays green with the extra uninstalled.
94
+ - **Known follow-ups.** Surface the active backend (keyword / local / cloud) on the
95
+ stats panel; reconcile mem0's own platform memory-id with our `event.id` (we key
96
+ dedup and reconstruction on `event.id` in metadata, not mem0's id); consider
97
+ scoping cloud entries by `run_id` via mem0 filters for multi-run isolation
98
+ (mirrors the ADR-0014 single-store caveat); the `infer=False` + chat-message
99
+ `add` shape assumes a mem0 client version with platform `infer` support β€” pin it
100
+ in the `memory` extra if a future release changes the contract.
docs/architecture/memory-stack.md CHANGED
@@ -155,6 +155,16 @@ Postgres+pgvector, ADR-0014) via `MEMORY_INDEX_CONFIG` (a JSON blob forwarded
155
  verbatim to the backend's `from_config`). Install the `memory` extra (`mem0ai` +
156
  `sentence-transformers`). See ADR-0019.
157
 
 
 
 
 
 
 
 
 
 
 
158
  **Alternative backends**: the two-method protocol can wrap any retrieval store β€”
159
  a stateful agent-memory service (e.g. a Letta-style memory server) could be a
160
  `MemoryIndex` too, as long as it stays derived from and rebuildable from the
 
155
  verbatim to the backend's `from_config`). Install the `memory` extra (`mem0ai` +
156
  `sentence-transformers`). See ADR-0019.
157
 
158
+ **Hosted backend (opt-in, ADR-0020)**: set `MEMORY_INDEX=cloud` (or
159
+ `MEMORY_INDEX_BACKEND=cloud`) to use mem0's managed platform (`MemoryClient`,
160
+ api.mem0.ai) instead of the local embedder. `Mem0CloudIndex` satisfies the *same*
161
+ `MemoryIndex` protocol β€” derived, idempotent by `event.id`, ledger-is-truth,
162
+ verbatim `infer=False` storage β€” so nothing downstream changes; only *where* the
163
+ embedding and retrieval run differs. It needs `MEM0_API_KEY` (plus optional
164
+ `MEM0_ORG_ID` / `MEM0_PROJECT_ID` / `MEM0_HOST`). **Caveat:** activating it sends
165
+ ledger event text to mem0's servers β€” a deliberate departure from the
166
+ off-the-grid default, which is why the local backend remains the default.
167
+
168
  **Alternative backends**: the two-method protocol can wrap any retrieval store β€”
169
  a stateful agent-memory service (e.g. a Letta-style memory server) could be a
170
  `MemoryIndex` too, as long as it stays derived from and rebuildable from the
src/core/memory_index.py CHANGED
@@ -14,15 +14,22 @@ Two pieces:
14
  events back. Any backend that satisfies this protocol can supply semantic
15
  relevance β€” a vector service, a local embedding store, or a fake in tests.
16
 
17
- * :class:`Mem0MemoryIndex` β€” a concrete backend. It is **lazy-imported and
18
- env-gated**: with the backend not installed or not configured, nothing here
19
- is imported and :class:`~src.core.memory.SalienceMemory` falls back to its
20
- keyword-Jaccard relevance exactly as before. The backend activates only when
21
- :func:`memory_index_from_env` finds it configured.
22
-
23
- Because the index is derived, ``index()`` upserts each event under its
24
- ``event.id`` so re-indexing the same events is a no-op (no duplicates) β€” this is
25
- what makes the index rebuildable rather than authoritative.
 
 
 
 
 
 
 
26
  """
27
 
28
  from __future__ import annotations
@@ -33,14 +40,22 @@ from typing import TYPE_CHECKING, Protocol, runtime_checkable
33
  from src.core.events import Event
34
 
35
  if TYPE_CHECKING: # pragma: no cover - typing only
36
- from mem0 import Memory
37
 
38
- #: Env gate. Set to a truthy value to activate the semantic index; unset (the
 
39
  #: default) keeps memory on the offline keyword path with nothing imported.
40
  INDEX_ENV = "MEMORY_INDEX"
41
 
42
- #: Truthy spellings accepted for the gate and boolean sub-options.
43
- _TRUTHY: frozenset[str] = frozenset({"1", "true", "yes", "on", "mem0"})
 
 
 
 
 
 
 
44
 
45
  #: Default mem0 config when ``MEMORY_INDEX_CONFIG`` is unset: embed LOCALLY with
46
  #: sentence-transformers (no API key; fully offline once the model is cached), so
@@ -79,7 +94,7 @@ class MemoryIndex(Protocol):
79
  ...
80
 
81
 
82
- # ── mem0 backend ────────────────────────────────────────────────────────────
83
 
84
 
85
  def _event_text(event: Event) -> str:
@@ -87,77 +102,60 @@ def _event_text(event: Event) -> str:
87
  return str(event.payload.get("text") or event.payload.get("summary") or event.payload)
88
 
89
 
90
- class Mem0MemoryIndex:
91
- """Semantic :class:`MemoryIndex` backed by the ``mem0`` vector memory.
92
 
93
- Derived, not authoritative. Each ledger event is upserted as one raw memory
94
- (``infer=False`` β€” text is stored verbatim, **no model extraction**, so
95
- indexing is deterministic and the ledger stays the source of truth) carrying
96
- the full event in ``metadata`` so a search hit reconstructs the original
97
- :class:`Event` without a second lookup. The entry id is the ``event.id``, so
98
- re-indexing the same event updates in place rather than duplicating β€” the
99
- index is rebuildable from the ledger.
100
 
101
- Configuration (env, read by :func:`memory_index_from_env`):
 
102
 
103
- * ``MEMORY_INDEX`` β€” gate; truthy activates the index, unset disables it.
104
- * Embeddings β€” run LOCALLY by default via sentence-transformers (no API
105
- key, fully offline once the model is cached); see :data:`_LOCAL_INDEX_CONFIG`.
106
- Install the backend with ``uv sync --extra memory``.
107
- * ``MEMORY_INDEX_CONFIG`` β€” optional JSON config forwarded verbatim to
108
- ``mem0.Memory.from_config``, replacing the local default. Use it to pick a
109
- different embedder or to persist vectors in the project's own
110
- Postgres/pgvector (the durable store from ADR-0014) instead of the default
111
- in-process vector store, so the index lives beside the ledger it derives
112
- from.
113
-
114
- ``mem0`` is imported lazily inside :meth:`_memory` so ``import src.*`` and
115
- ``import app`` work with the package not installed.
116
  """
117
 
118
- #: mem0 scopes memories to a session id; the index is engine-wide, so a fixed
119
  #: namespace keeps every event in one searchable space.
120
  _NAMESPACE = "ledger"
121
 
122
- def __init__(self, config: dict | None = None) -> None:
123
- self._config = config
124
- self._mem: "Memory | None" = None
125
  self._indexed: set[str] = set()
126
 
127
- # ── lazy construction ─────────────────────────────────────────────────────
128
 
129
- def _memory(self) -> "Memory":
130
- """Construct (once) and return the underlying ``mem0`` memory.
 
131
 
132
- With no explicit config, the local sentence-transformers default
133
- (:data:`_LOCAL_INDEX_CONFIG`) is used β€” never mem0's cloud-keyed default β€”
134
- so an activated index stays fully offline.
135
- """
136
- if self._mem is None:
137
- from mem0 import Memory # lazy: offline import must not require mem0
 
138
 
139
- self._mem = Memory.from_config(self._config or _LOCAL_INDEX_CONFIG)
 
 
 
 
140
  return self._mem
141
 
142
  # ── MemoryIndex protocol ──────────────────────────────────────────────────
143
 
144
  def index(self, events: tuple[Event, ...]) -> None:
145
- """Upsert *events* into the vector store, keyed by ``event.id``.
146
 
147
- Idempotent: an ``event.id`` already indexed in this process is skipped, so
148
- re-indexing the same ledger slice each turn does not duplicate entries.
149
- """
150
  fresh = [e for e in events if e.id not in self._indexed]
151
  if not fresh:
152
  return
153
  mem = self._memory()
154
  for event in fresh:
155
- mem.add(
156
- _event_text(event),
157
- user_id=self._NAMESPACE,
158
- metadata=_event_metadata(event),
159
- infer=False, # store verbatim; the ledger, not a model, is truth
160
- )
161
  self._indexed.add(event.id)
162
 
163
  def search(self, query: str, k: int) -> list[Event]:
@@ -165,15 +163,133 @@ class Mem0MemoryIndex:
165
  if not query or k <= 0:
166
  return []
167
  mem = self._memory()
168
- hits = mem.search(query, top_k=k, filters={"user_id": self._NAMESPACE})
169
  events: list[Event] = []
170
- for hit in _result_items(hits):
171
  event = _event_from_metadata(hit.get("metadata"))
172
  if event is not None:
173
  events.append(event)
174
  return events
175
 
176
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
177
  # ── metadata round-trip (event ⇄ vector entry) ────────────────────────────────
178
 
179
 
@@ -210,10 +326,11 @@ def _event_from_metadata(metadata: dict | None) -> Event | None:
210
 
211
 
212
  def _result_items(hits: object) -> list[dict]:
213
- """Normalise ``mem0.search`` output to a list of hit dicts.
214
 
215
- ``mem0`` returns either ``{"results": [...]}`` (v1.1+) or a bare list,
216
- depending on version/config; accept both so the backend is version-tolerant.
 
217
  """
218
  if isinstance(hits, dict):
219
  results = hits.get("results", [])
@@ -230,16 +347,34 @@ def _is_truthy(value: str | None) -> bool:
230
 
231
 
232
  def memory_index_from_env(env: dict[str, str] | None = None) -> MemoryIndex | None:
233
- """Build a :class:`Mem0MemoryIndex` from the env gate, or ``None`` if unset.
 
 
234
 
235
- Returns ``None`` (the offline default the suite exercises) unless
236
- ``MEMORY_INDEX`` is truthy. ``mem0`` is only imported later, on first use, so
237
- a truthy gate without the package installed still imports cleanly and fails
238
- loudly only when the index is actually exercised.
 
 
 
239
  """
240
  source = os.environ if env is None else env
241
- if not _is_truthy(source.get(INDEX_ENV)):
 
 
 
 
242
  return None
 
 
 
 
 
 
 
 
 
243
  raw_config = (source.get("MEMORY_INDEX_CONFIG") or "").strip()
244
  config: dict | None = None
245
  if raw_config:
 
14
  events back. Any backend that satisfies this protocol can supply semantic
15
  relevance β€” a vector service, a local embedding store, or a fake in tests.
16
 
17
+ * Two concrete backends behind that protocol, both **lazy-imported and
18
+ env-gated** so nothing is imported and :class:`~src.core.memory.SalienceMemory`
19
+ stays on its keyword-Jaccard relevance unless an index is configured:
20
+
21
+ - :class:`Mem0MemoryIndex` β€” the default, **off the grid**. Wraps the
22
+ ``mem0`` OSS ``Memory`` with a local sentence-transformers embedder; no
23
+ API key, nothing leaves the machine (ADR-0019).
24
+ - :class:`Mem0CloudIndex` β€” opt-in hosted backend. Wraps the ``mem0``
25
+ platform ``MemoryClient`` (api.mem0.ai). **Activating it sends ledger
26
+ event text to mem0's servers** and needs ``MEM0_API_KEY`` β€” a deliberate
27
+ departure from the off-the-grid default, so it is never the default
28
+ (ADR-0020).
29
+
30
+ Because the index is derived, both backends upsert each event under its
31
+ ``event.id`` (process-local dedup) so re-indexing the same events is a no-op (no
32
+ duplicates) β€” this is what makes the index rebuildable rather than authoritative.
33
  """
34
 
35
  from __future__ import annotations
 
40
  from src.core.events import Event
41
 
42
  if TYPE_CHECKING: # pragma: no cover - typing only
43
+ from mem0 import Memory, MemoryClient
44
 
45
+ #: Env gate. Set to a truthy value to activate the local semantic index, or to a
46
+ #: cloud spelling (see :data:`_CLOUD_VALUES`) for the hosted backend; unset (the
47
  #: default) keeps memory on the offline keyword path with nothing imported.
48
  INDEX_ENV = "MEMORY_INDEX"
49
 
50
+ #: Optional explicit backend selector (``local`` | ``cloud``). Takes precedence
51
+ #: over the spelling of ``MEMORY_INDEX`` when set.
52
+ BACKEND_ENV = "MEMORY_INDEX_BACKEND"
53
+
54
+ #: Truthy spellings accepted for the gate and boolean sub-options (β†’ local backend).
55
+ _TRUTHY: frozenset[str] = frozenset({"1", "true", "yes", "on", "mem0", "local"})
56
+
57
+ #: Spellings of ``MEMORY_INDEX`` that select the hosted mem0 platform backend.
58
+ _CLOUD_VALUES: frozenset[str] = frozenset({"cloud", "mem0-cloud", "platform", "hosted"})
59
 
60
  #: Default mem0 config when ``MEMORY_INDEX_CONFIG`` is unset: embed LOCALLY with
61
  #: sentence-transformers (no API key; fully offline once the model is cached), so
 
94
  ...
95
 
96
 
97
+ # ── shared event ⇄ entry helpers ──────────────────────────────────────────────
98
 
99
 
100
  def _event_text(event: Event) -> str:
 
102
  return str(event.payload.get("text") or event.payload.get("summary") or event.payload)
103
 
104
 
105
+ # ── shared mem0 backend base ──────────────────────────────────────────────────
 
106
 
 
 
 
 
 
 
 
107
 
108
+ class _Mem0BackendBase:
109
+ """Shared :class:`MemoryIndex` machinery for the mem0-backed indexes.
110
 
111
+ Subclasses supply the three variation points β€” how the client is built and
112
+ how a single event is stored / queried β€” while this base owns the protocol
113
+ surface that keeps the index *derived*: idempotent upsert keyed by
114
+ ``event.id`` and search-hit β†’ :class:`Event` reconstruction from metadata.
 
 
 
 
 
 
 
 
 
115
  """
116
 
117
+ #: mem0 scopes memories to an id; the index is engine-wide, so a fixed
118
  #: namespace keeps every event in one searchable space.
119
  _NAMESPACE = "ledger"
120
 
121
+ def __init__(self) -> None:
122
+ self._mem: object | None = None
 
123
  self._indexed: set[str] = set()
124
 
125
+ # ── variation points (subclass) ───────────────────────────────────────────
126
 
127
+ def _build_memory(self) -> object:
128
+ """Construct the underlying mem0 client (lazy-imported by the subclass)."""
129
+ raise NotImplementedError
130
 
131
+ def _store(self, mem: object, event: Event) -> None:
132
+ """Upsert one event verbatim into *mem* (``infer=False``; ledger is truth)."""
133
+ raise NotImplementedError
134
+
135
+ def _query(self, mem: object, query: str, k: int) -> list[dict]:
136
+ """Run semantic search on *mem*; return raw hit dicts (carrying metadata)."""
137
+ raise NotImplementedError
138
 
139
+ # ── lazy construction ─────────────────────────────────────────────────────
140
+
141
+ def _memory(self) -> object:
142
+ if self._mem is None:
143
+ self._mem = self._build_memory()
144
  return self._mem
145
 
146
  # ── MemoryIndex protocol ──────────────────────────────────────────────────
147
 
148
  def index(self, events: tuple[Event, ...]) -> None:
149
+ """Upsert *events*, keyed by ``event.id`` β€” idempotent within the process.
150
 
151
+ Dedup happens *before* the client is built, so re-indexing the same
152
+ ledger slice each turn never re-embeds and never forces a mem0 import."""
 
153
  fresh = [e for e in events if e.id not in self._indexed]
154
  if not fresh:
155
  return
156
  mem = self._memory()
157
  for event in fresh:
158
+ self._store(mem, event)
 
 
 
 
 
159
  self._indexed.add(event.id)
160
 
161
  def search(self, query: str, k: int) -> list[Event]:
 
163
  if not query or k <= 0:
164
  return []
165
  mem = self._memory()
 
166
  events: list[Event] = []
167
+ for hit in self._query(mem, query, k):
168
  event = _event_from_metadata(hit.get("metadata"))
169
  if event is not None:
170
  events.append(event)
171
  return events
172
 
173
 
174
+ # ── local (off-the-grid) backend ─────���────────────────────────────────────────
175
+
176
+
177
+ class Mem0MemoryIndex(_Mem0BackendBase):
178
+ """Local semantic :class:`MemoryIndex` backed by the ``mem0`` OSS ``Memory``.
179
+
180
+ Derived, not authoritative, and **off the grid**: each ledger event is
181
+ upserted as one raw memory (``infer=False`` β€” text stored verbatim, **no
182
+ model extraction**) carrying the full event in ``metadata`` so a search hit
183
+ reconstructs the :class:`Event` without a second lookup. Embeddings run
184
+ locally via sentence-transformers by default (:data:`_LOCAL_INDEX_CONFIG`).
185
+
186
+ Configuration (env, read by :func:`memory_index_from_env`):
187
+
188
+ * ``MEMORY_INDEX`` β€” gate; truthy (``1``/``true``/``local``/…) activates this
189
+ backend, unset disables it.
190
+ * ``MEMORY_INDEX_CONFIG`` β€” optional JSON config forwarded verbatim to
191
+ ``mem0.Memory.from_config``, replacing the local default (pick a different
192
+ embedder, or persist vectors in the project's Postgres/pgvector, ADR-0014).
193
+
194
+ ``mem0`` is imported lazily inside :meth:`_build_memory` so ``import src.*`` and
195
+ ``import app`` work with the package not installed.
196
+ """
197
+
198
+ def __init__(self, config: dict | None = None) -> None:
199
+ super().__init__()
200
+ self._config = config
201
+
202
+ def _build_memory(self) -> "Memory":
203
+ from mem0 import Memory # lazy: offline import must not require mem0
204
+
205
+ return Memory.from_config(self._config or _LOCAL_INDEX_CONFIG)
206
+
207
+ def _store(self, mem: object, event: Event) -> None:
208
+ mem.add( # type: ignore[attr-defined]
209
+ _event_text(event),
210
+ user_id=self._NAMESPACE,
211
+ metadata=_event_metadata(event),
212
+ infer=False, # store verbatim; the ledger, not a model, is truth
213
+ )
214
+
215
+ def _query(self, mem: object, query: str, k: int) -> list[dict]:
216
+ return _result_items(mem.search(query, top_k=k, filters={"user_id": self._NAMESPACE})) # type: ignore[attr-defined]
217
+
218
+
219
+ # ── hosted (opt-in) backend ────────────────────────────────────────────────────
220
+
221
+
222
+ class Mem0CloudIndex(_Mem0BackendBase):
223
+ """Hosted semantic :class:`MemoryIndex` backed by the ``mem0`` platform.
224
+
225
+ Wraps ``mem0.MemoryClient`` (api.mem0.ai): embeddings, the vector store, and
226
+ retrieval all live in mem0's managed service. The :class:`MemoryIndex`
227
+ contract is identical to the local backend β€” derived, idempotent, ledger is
228
+ truth β€” and events are still stored verbatim (``infer=False``) with the full
229
+ event in ``metadata`` for reconstruction. The only difference is *where* the
230
+ work happens.
231
+
232
+ **Off-the-grid caveat (ADR-0019/0020).** Activating this backend sends ledger
233
+ event text to mem0's servers and requires a ``MEM0_API_KEY``. It is therefore
234
+ strictly opt-in and never the default; the local backend remains the engine's
235
+ off-the-grid default.
236
+
237
+ Configuration (env, read by :func:`memory_index_from_env`):
238
+
239
+ * ``MEMORY_INDEX=cloud`` (or ``MEMORY_INDEX_BACKEND=cloud``) β€” selects this
240
+ backend.
241
+ * ``MEM0_API_KEY`` β€” required platform key (falls back to the client's own
242
+ ``MEM0_API_KEY`` env read if not passed explicitly).
243
+ * ``MEM0_ORG_ID`` / ``MEM0_PROJECT_ID`` / ``MEM0_HOST`` β€” optional scoping.
244
+
245
+ ``mem0`` is imported lazily inside :meth:`_build_memory`, so the offline path
246
+ needs neither the package nor a key.
247
+ """
248
+
249
+ def __init__(
250
+ self,
251
+ api_key: str | None = None,
252
+ org_id: str | None = None,
253
+ project_id: str | None = None,
254
+ host: str | None = None,
255
+ ) -> None:
256
+ super().__init__()
257
+ self._api_key = api_key
258
+ self._org_id = org_id
259
+ self._project_id = project_id
260
+ self._host = host
261
+
262
+ def _build_memory(self) -> "MemoryClient":
263
+ from mem0 import MemoryClient # lazy: offline import must not require mem0
264
+
265
+ # Pass only what is set; MemoryClient falls back to MEM0_API_KEY from the
266
+ # environment and raises loudly here (not at import) if no key is found.
267
+ kwargs = {
268
+ k: v
269
+ for k, v in {
270
+ "api_key": self._api_key,
271
+ "org_id": self._org_id,
272
+ "project_id": self._project_id,
273
+ "host": self._host,
274
+ }.items()
275
+ if v
276
+ }
277
+ return MemoryClient(**kwargs)
278
+
279
+ def _store(self, mem: object, event: Event) -> None:
280
+ # The platform `add` takes chat-style messages; one verbatim user turn per
281
+ # event, inference disabled so nothing but the ledger text is stored.
282
+ mem.add( # type: ignore[attr-defined]
283
+ [{"role": "user", "content": _event_text(event)}],
284
+ user_id=self._NAMESPACE,
285
+ metadata=_event_metadata(event),
286
+ infer=False,
287
+ )
288
+
289
+ def _query(self, mem: object, query: str, k: int) -> list[dict]:
290
+ return _result_items(mem.search(query, user_id=self._NAMESPACE, top_k=k)) # type: ignore[attr-defined]
291
+
292
+
293
  # ── metadata round-trip (event ⇄ vector entry) ────────────────────────────────
294
 
295
 
 
326
 
327
 
328
  def _result_items(hits: object) -> list[dict]:
329
+ """Normalise mem0 ``search`` output to a list of hit dicts.
330
 
331
+ Both the OSS ``Memory`` and the platform ``MemoryClient`` return either
332
+ ``{"results": [...]}`` or a bare list depending on version/config; accept both
333
+ so the backends are version-tolerant.
334
  """
335
  if isinstance(hits, dict):
336
  results = hits.get("results", [])
 
347
 
348
 
349
  def memory_index_from_env(env: dict[str, str] | None = None) -> MemoryIndex | None:
350
+ """Build a mem0-backed :class:`MemoryIndex` from the env, or ``None`` if unset.
351
+
352
+ Selection (``mem0`` is only imported later, on first use):
353
 
354
+ * gate unset / falsey β†’ ``None`` (the offline keyword path the suite exercises).
355
+ * ``MEMORY_INDEX`` truthy (``1``/``true``/``local``/…) β†’ :class:`Mem0MemoryIndex`
356
+ (local sentence-transformers; off the grid).
357
+ * ``MEMORY_INDEX`` ∈ {cloud, mem0-cloud, platform, hosted}, or
358
+ ``MEMORY_INDEX_BACKEND=cloud`` β†’ :class:`Mem0CloudIndex` (hosted; sends
359
+ ledger text to mem0). An explicit ``MEMORY_INDEX_BACKEND`` wins over the
360
+ ``MEMORY_INDEX`` spelling.
361
  """
362
  source = os.environ if env is None else env
363
+ gate = (source.get(INDEX_ENV) or "").strip().lower()
364
+ backend = (source.get(BACKEND_ENV) or "").strip().lower()
365
+
366
+ is_cloud = backend == "cloud" or gate in _CLOUD_VALUES
367
+ if not (is_cloud or _is_truthy(gate)):
368
  return None
369
+
370
+ if is_cloud:
371
+ return Mem0CloudIndex(
372
+ api_key=source.get("MEM0_API_KEY") or None,
373
+ org_id=source.get("MEM0_ORG_ID") or None,
374
+ project_id=source.get("MEM0_PROJECT_ID") or None,
375
+ host=source.get("MEM0_HOST") or None,
376
+ )
377
+
378
  raw_config = (source.get("MEMORY_INDEX_CONFIG") or "").strip()
379
  config: dict | None = None
380
  if raw_config:
tests/test_memory_index.py CHANGED
@@ -22,6 +22,7 @@ from src.core.events import Event
22
  from src.core.manifest import AgentManifest, MemoryConfig
23
  from src.core.memory import SalienceMemory
24
  from src.core.memory_index import (
 
25
  Mem0MemoryIndex,
26
  MemoryIndex,
27
  memory_index_from_env,
@@ -72,6 +73,10 @@ class TestProtocol:
72
  # No mem0 import needed: the backend is constructed lazily.
73
  assert isinstance(Mem0MemoryIndex(), MemoryIndex)
74
 
 
 
 
 
75
 
76
  # ── layering: index drives the relevance term, recency/importance intact ────────
77
 
@@ -155,6 +160,13 @@ class TestIdempotentIndexing:
155
  # raise (mem0 may be absent), so reaching it on a dup would surface here.
156
  backend.index((_event("world.observed", eid="e1"),)) # no-op, no import
157
 
 
 
 
 
 
 
 
158
 
159
  # ── env gate (no mem0 required) ──────────────────────────────────────────────────
160
 
@@ -175,6 +187,31 @@ class TestEnvGate:
175
  assert isinstance(idx, Mem0MemoryIndex)
176
  assert idx._config == {"version": "v1.1"}
177
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
178
 
179
  # ── agent wiring: _recall threads the index into salience ────────────────────────
180
 
@@ -237,3 +274,28 @@ class TestMem0RoundTrip:
237
  hit = next(h for h in hits if h.id == "rt1")
238
  assert hit.kind == "world.observed"
239
  assert hit.payload.get("text", "").startswith("golden spores")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22
  from src.core.manifest import AgentManifest, MemoryConfig
23
  from src.core.memory import SalienceMemory
24
  from src.core.memory_index import (
25
+ Mem0CloudIndex,
26
  Mem0MemoryIndex,
27
  MemoryIndex,
28
  memory_index_from_env,
 
73
  # No mem0 import needed: the backend is constructed lazily.
74
  assert isinstance(Mem0MemoryIndex(), MemoryIndex)
75
 
76
+ def test_cloud_backend_is_memory_index(self):
77
+ # No mem0 import needed: the platform client is constructed lazily.
78
+ assert isinstance(Mem0CloudIndex(), MemoryIndex)
79
+
80
 
81
  # ── layering: index drives the relevance term, recency/importance intact ────────
82
 
 
160
  # raise (mem0 may be absent), so reaching it on a dup would surface here.
161
  backend.index((_event("world.observed", eid="e1"),)) # no-op, no import
162
 
163
+ def test_cloud_backend_skips_already_indexed_ids(self):
164
+ """Cloud dedup is identical and happens before the client is built β€” so a
165
+ repeat id is a no-op with mem0 absent and no MEM0_API_KEY set."""
166
+ backend = Mem0CloudIndex()
167
+ backend._indexed.add("e1")
168
+ backend.index((_event("world.observed", eid="e1"),)) # no-op, no import/key
169
+
170
 
171
  # ── env gate (no mem0 required) ──────────────────────────────────────────────────
172
 
 
187
  assert isinstance(idx, Mem0MemoryIndex)
188
  assert idx._config == {"version": "v1.1"}
189
 
190
+ def test_truthy_gate_selects_local_not_cloud(self):
191
+ assert isinstance(memory_index_from_env({"MEMORY_INDEX": "1"}), Mem0MemoryIndex)
192
+
193
+ def test_cloud_spelling_selects_hosted_backend(self):
194
+ for spelling in ("cloud", "mem0-cloud", "platform", "hosted"):
195
+ idx = memory_index_from_env({"MEMORY_INDEX": spelling})
196
+ assert isinstance(idx, Mem0CloudIndex), spelling
197
+
198
+ def test_backend_env_overrides_local_gate(self):
199
+ # Explicit MEMORY_INDEX_BACKEND=cloud wins even when the gate spells local.
200
+ idx = memory_index_from_env({"MEMORY_INDEX": "1", "MEMORY_INDEX_BACKEND": "cloud"})
201
+ assert isinstance(idx, Mem0CloudIndex)
202
+
203
+ def test_cloud_reads_credentials_from_env(self):
204
+ idx = memory_index_from_env(
205
+ {
206
+ "MEMORY_INDEX": "cloud",
207
+ "MEM0_API_KEY": "k-123",
208
+ "MEM0_ORG_ID": "org-1",
209
+ "MEM0_PROJECT_ID": "proj-1",
210
+ }
211
+ )
212
+ assert isinstance(idx, Mem0CloudIndex)
213
+ assert (idx._api_key, idx._org_id, idx._project_id) == ("k-123", "org-1", "proj-1")
214
+
215
 
216
  # ── agent wiring: _recall threads the index into salience ────────────────────────
217
 
 
274
  hit = next(h for h in hits if h.id == "rt1")
275
  assert hit.kind == "world.observed"
276
  assert hit.payload.get("text", "").startswith("golden spores")
277
+
278
+
279
+ # ── guarded real-mem0-CLOUD round-trip (requires mem0 + MEM0_API_KEY) ────────────
280
+
281
+
282
+ class TestMem0CloudRoundTrip:
283
+ def test_index_then_search_recovers_event(self):
284
+ pytest.importorskip("mem0")
285
+ # Opt-in: this hits the hosted mem0 platform and sends text off-machine, so
286
+ # it only runs when explicitly enabled with a real key.
287
+ if not (os.getenv("MEM0_API_KEY") and os.getenv("MEM0_CLOUD_E2E")):
288
+ pytest.skip("set MEM0_API_KEY and MEM0_CLOUD_E2E=1 to run the hosted round-trip")
289
+
290
+ backend = Mem0CloudIndex() # reads MEM0_API_KEY from the environment
291
+ ev = _event("world.observed", turn=1, text="golden spores drift over the glass forest", eid="rtc1")
292
+ try:
293
+ backend.index((ev,))
294
+ hits = backend.search("golden spores", k=5)
295
+ except Exception as exc: # pragma: no cover - environment dependent
296
+ pytest.skip(f"mem0 cloud unavailable: {exc}")
297
+
298
+ assert any(h.id == "rtc1" for h in hits)
299
+ hit = next(h for h in hits if h.id == "rtc1")
300
+ assert hit.kind == "world.observed"
301
+ assert hit.payload.get("text", "").startswith("golden spores")