picassodocs
docs/15-memory.md

15 - Memory

The persistent memory layer. The agent does not start cold every session. It remembers the codebase's architecture, the developer's coding style, and the decisions that were made and why. Inspired by Supermemory (https://supermemory.ai); built native so it composes with every other surface — plan mode, subagents, /goal, the dashboard. Retrieval, never training. Per-account isolated. User-deletable at any time. This is the second-biggest differentiator after the ad-supported price (13-ads-monetization.md).

The memory layer's visual identity is locked to --rose. The Memory Recall band on Scene 01 (First Light) uses a rose-tinted background with a glowing rose dot, the MEMORY RECALL eyebrow in rose ALL-CAPS, and a fact-count summary on the right. The full memory inspector lives in Scene 06 with class-grouped columns for facts, style, decisions, failures, workflows, repo rules, and model performance, a filter input, keyboard-action hints, and a mini relationship graph showing how decisions connect to files and modules. The /memory slash command opens this scene in the TUI; the Mac Memory screen shows the same records. See design/scenes-b.jsx for the layout.

Why it exists

Every coding-agent CLI today starts each session from zero. Claude Code reads CLAUDE.md. Kimi reads AGENTS.md. Tabnine has lightweight save_memory for to-dos. None of them retain what they learned about the codebase last session, what coding patterns the developer prefers, or what decisions were taken on prior turns. Developers compensate by re-pasting context, re-explaining conventions, and accepting the same wrong suggestions over and over.

The memory layer fixes this. After the first session in a repo, the agent has a profile of the codebase. After ten sessions, the agent knows the developer's style. After a /goal run, the agent knows what was decided and why. Tokens drop. Quality rises. Re-onboarding goes away.

Source of truth

The structured memory store at ~/.picasso/memory.json is the source of truth: one user-level file for every surface (PICASSO_MEMORY_ROOT overrides the directory; otherwise PICASSO_HOME, otherwise the OS account's home, never $HOME). CLI, TUI, local helper, and Mac all read and write it through the same backend: Node writers through @picasso/memory and @picasso/shared-home, the engine plugin through picasso-memory's byte-compatible lock, and the Mac through the helper relay (it never writes the file from Swift). Repo scoping lives in each record's namespace (account:<id>/projects/workspace:<fingerprint>/…), not in the file path. The Mac app must not keep a separate UserDefaults-only memory store; UserDefaults may only cache view state or recently fetched records.

The file is schema v2: { "schemaVersion": 2, "records": VectorRecord[] }. Mac-only fields (scope, source, stale, tags, title) live in metadata.attributes, and a deleted record stays as a tombstone (attributes.deletedAt, empty body) for 30 days so an older copy cannot bring it back. Older shapes migrate forward on read: v0 (a bare array, the old Mac and plugin file) and v1 ({ "version": 1 }, the old CLI file). The first write after a migration leaves memory.json.v<n>.bak. A file from a newer Picasso is refused with "…written by a newer Picasso (schema N…). Update Picasso." It is never read as empty, and the engine injects nothing. Every write holds memory.json.lock and re-reads the file, then merges per id (the newest metadata.lastObservedAt wins).

The old per-repo <repo>/.picasso/memory.json and the Mac's old Application Support/Picasso/memory.json are imported once (importRepoMemoryIntoHome, importLegacyMacMemory). Each is merged by record id under the lock and keeps its namespaces. The source is left in place with a <file>.migrated-to-home marker beside it.

Human-readable files under ~/.picasso/memory/*.md are generated projections. They help users inspect what Picasso remembers, but they are not the backend and must not become the authority for retrieval, write-back, sync, or namespace isolation.

Local projection layout:

~/.picasso/
  memory.json
  memory/
    GLOBAL.md
    REPO.md
    DECISIONS.md
    FAILURES.md

Cloud sync is optional, off by default, and layered over the local source of truth. It never creates a parallel memory model; it syncs structured records, cursors, provenance, and projection invalidation state from the same runtime backend through /v1/memory/sync, and needs a signed-in account (signed out exits 3 with "Memory sync needs a signed-in Picasso account. Run picasso auth login.").

Synced records are encrypted on the device with a random 256-bit account memory key (AES-256-GCM); the API stores only ciphertext. The key is created on the first device that turns on sync and is stored in the login Keychain (service com.picassohq.memory-key, account <accountId>:<keyId>, plus <accountId>:active naming the current key), with the same ACL as the account session so the CLI and the Mac app on one machine share it. Another machine gets the key in one of two ways:

deterministicAccountMemoryKey exists for tests only and throws outside a test runner.

What gets remembered

Seven memory classes, with distinct lifecycles and namespaces:

Codebase facts

Architectural and structural truth about the repo. Examples:

Codebase facts are derived automatically from the indexer (crates/indexer) and from observations during sessions. They are bound to the workspace fingerprint, so switching repos switches profiles. They go stale — fact entries carry a last_observed timestamp and a staleness_score that increases with workspace drift.

Style preferences

How the developer codes. Examples:

Style preferences are learned from accepted edits, rejected suggestions, and explicit corrections the developer makes to agent output. They are scoped to the account (cross-project — your style is your style) but namespaced by language and framework when relevant.

Decision history

Why the codebase looks the way it does. Examples:

Decisions are written explicitly during REFLECT steps when the model decides something non-obvious, or when the user accepts a plan that contains a rationale. Decisions are bound to the workspace fingerprint and carry rich provenance (session id, prompt that led to the decision, plan document if relevant).

Failure patterns

Actionable records of failed attempts, rejected fixes, repeated loops, test failures, tool errors, and runtime wiring mistakes. Examples:

Failure memory is scoped to the repo or goal when possible. It carries preventNextTime guidance so future runs can avoid the same mistake.

Workflow patterns

Repeatable operational procedures that help the agent choose the right commands and verification order. Examples:

Repo rules

Project-specific instructions that should be treated like durable local policy. Examples:

Repo rules are stronger than ordinary facts and should be surfaced prominently during prompt injection.

Model performance notes

Provider/model behavior observed during real runs. Examples:

Model performance memories must not include sponsor metadata, provider credentials, raw prompts, secrets, or private user content.

Storage architecture

V1 local storage starts with ~/.picasso/memory.json plus vector-shaped record fields. Production retrieval uses provider-routed embeddings when configured, with deterministic hash embeddings only as a local/test fallback. The long-term architecture remains hybrid vector + graph storage: vectors for semantic retrieval; graph edges for typed relationships between memories.

┌─────────────────────────────────────────────────────┐
│           Memory Store (per account)                │
│                                                     │
│  ┌─────────────────┐    ┌──────────────────┐        │
│  │  Vector Index   │◄──►│   Graph Store    │        │
│  │  (Qdrant /      │    │   (memdb crate)  │        │
│  │   Lance)        │    │                  │        │
│  └─────────────────┘    └──────────────────┘        │
│         ▲                       ▲                   │
│         │                       │                   │
│         └──────┬────────────────┘                   │
│                │                                    │
│         ┌──────┴──────┐                             │
│         │  Embedder   │ (provider-routed)           │
│         └─────────────┘                             │
└─────────────────────────────────────────────────────┘

Namespaces

Memory is namespaced for isolation and selective injection:

account:<account-id>
  ├── style/                  ← cross-project, account-wide
  │     ├── typescript/
  │     ├── python/
  │     └── go/
  └── projects/
        ├── workspace:<fingerprint-A>
        │     ├── facts/
        │     └── decisions/
        ├── workspace:<fingerprint-B>
        │     ├── facts/
        │     └── decisions/
        └── orgs/
              └── org:<org-id>/
                    └── shared/            ← team-shared, opt-in

Default retrieval queries style/ for the active language plus projects/workspace:<current>/ for the current workspace. Org-shared namespaces are opt-in per project and visible to all org members.

Retrieval pipeline

Memory queries fire before planning/model execution in every normal CLI run, TUI prompt run, and /goal iteration. Pipeline:

  1. Build query — extract intent from the prompt and the current plan-mode draft (if any). Multi-vector: the prompt itself, file paths mentioned, symbol names referenced.
  2. Vector search — top-K (default 8) similarity match across style/ and projects/workspace:<current>/.
  3. Graph expansion — for each hit, traverse 1-hop outgoing edges to find related decisions and facts.
  4. Rank and filter — score combining similarity, recency, staleness, and explicit user pins.
  5. Inject — retrieved memories are formatted as a system-prompt block and prepended to the loop's next call.
<system>
You are a coding agent operating in workspace acme-corp/api.

Memory context for this session:

Codebase facts:
- The auth module owns session tokens and JWT verification. (last observed: 2 days ago)
- HTTP handlers live in src/handlers/, wired through src/server/router.ts.
- Test runner is Vitest. Tests live alongside source in *.test.ts files.

Style preferences (TypeScript):
- Explicit return types on exported functions; inferred internally.
- async/await preferred over .then() chains.

Recent decisions:
- 2026-04: chose argon2id over bcrypt for password hashing.
- 2026-05: rolled back Redis session cache; using in-memory LRU.

Failure patterns:
- Do not treat manual `/memory` CRUD as proof that runtime memory is wired.

Repo rules:
- Verify normal agent run paths, not only package-level primitives.

(... agent's actual prompt continues ...)
</system>

Retrieval latency target: <100ms p95 from query construction to injected memory block ready.

Write pipeline

Writes happen at the REFLECT step after successful and failed runs. A classifier examines the loop's prompt, actions, tool results, changed files, verification output, and final reflection, then decides what (if anything) to persist:

The normal runtime flow is:

user prompt
  -> retrieve relevant memory
  -> inject memory into planning/model context
  -> execute tools/edits/tests
  -> reflect on outcome
  -> classify useful facts/style/decisions/failures/workflows/repo rules/model performance
  -> write structured memory
  -> invalidate projections and refresh CLI/TUI/Mac views
  -> optionally sync through /v1/memory/sync

Every memory entry has:

type MemoryEntry = {
  id: string;
  class:
    | "fact"
    | "style"
    | "decision"
    | "failure"
    | "workflow"
    | "repo_rule"
    | "model_performance";
  namespace: string; // e.g., "projects/workspace:abc123/facts"
  body: string; // natural-language form, model-friendly
  embedding: number[]; // computed at write time
  edges: GraphEdge[]; // typed relationships
  preventNextTime?: string[]; // failure/workflow guidance
  provenance: {
    sessionId: string;
    eventIndex: number; // event within the session that produced this
    sourceFiles?: string[]; // files involved
    sourcePlan?: string; // plan id if from plan-mode approval
  };
  createdAt: ISO8601;
  lastObservedAt: ISO8601; // updated on re-observation
  stalenessScore: number; // 0-1, increases with code drift
  pinned: boolean; // user-pinned entries are never auto-evicted
};

Memory write-back must redact secrets, provider credentials, private keys, env values, sponsor/ad metadata, raw tool output, and sensitive terminal output. The write pipeline emits explicit events when retrieval, injection, write-back, sync, projection, deletion, or denial occurs.

User-facing controls

picasso memory list                       # show entries for current workspace
picasso memory list --namespace style     # account-wide style preferences
picasso memory init                       # initialize profile and projections
picasso memory inspect <entry-id>         # full detail of an entry
picasso memory add <note...>              # save a new memory (same as /memory add)
picasso memory edit <entry-id>            # edit in $EDITOR
picasso memory delete <entry-id>          # remove an entry
picasso memory pin <entry-id>             # never auto-evict
picasso memory unpin <entry-id>
picasso memory export > memory.json       # full export, {schemaVersion: 2, records}
picasso memory import memory.json         # merge an export back in by record id
picasso memory sync                       # encrypted cloud sync (signed in)
picasso memory key status                 # does this device have the account memory key
picasso memory key export                 # print the pmk1-… recovery key (a secret)
picasso memory key import --stdin         # store a recovery key (stdin or hidden prompt, never argv)
picasso memory key link --request         # new device: ask for the key, shows six words
picasso memory key link --approve --code "<six words>"   # device with the key: send it
picasso memory recover-sync-lock          # inspect (then quarantine) a stuck sync cursor lock after stopping all writers
picasso memory clear --workspace <fingerprint> --confirm clear-memories   # wipe one workspace's memories
picasso memory clear --confirm clear-memories                             # wipe everything (asks for the phrase first)

In the TUI:

/memory                       # opens the memory inspector panel
/memory add <note>            # save a workspace memory note
/memory delete <entry-id>     # remove a visible memory row
/memory pin <entry-id>        # keep a memory from auto-eviction
/memory unpin <entry-id>      # return a pinned memory to normal retention

Normal picasso run "..." and normal TUI prompt execution must use the same memory retrieval/write-back path. No user should need to run /memory add for memory to become useful.

Runtime tools:

Project profile generation

On the first session in a new workspace, the CLI offers to generate a project profile:

This looks like a new workspace. Generate a project profile so the agent
remembers structure, conventions, and dependencies?

  [Y] Yes (runs codebase_investigator, ~30 seconds)
  [N] Not now (you can run `picasso memory init` later)
  [Skip] Don't ask again for this workspace

The profile generator dispatches a codebase_investigator subagent (see 03-subagents.md) in read-only mode. The subagent walks the repo, calls the indexer, writes high-quality structured entries into ~/.picasso/memory.json, and regenerates ~/.picasso/memory/*.md projections. It scans README files, package manifests, workspace config, docs, tests, .picasso config, recent git history, and agent rules. Total time under one minute for medium repos; longer for monorepos.

Shared CLI, TUI, and Mac behavior

CLI, TUI, Mac, and the desktop helper must present one memory system:

Companion dashboard surface

Not hosted at launch. The account dashboard at https://picassohq.com/account shows account sessions, privacy choices and telemetry state only; it does not show memory, and the API stores synced memory only as ciphertext it cannot read. Memory is inspected in the CLI (picasso memory list|inspect), the TUI /memory panel, the Mac Memory screen, and the ~/.picasso/memory/*.md projections. A hosted memory inspector would need the memory key in the browser and is post-launch.

Eviction and retention

Without bounds, memory grows. Eviction policy:

Eviction runs nightly per-account on the API; users can trigger manual cleanup with picasso memory gc.

Privacy and integrity guarantees

Hard rules, written in code, audited per release:

  1. Memory is never training data. The memory layer is retrieval-only. No model is trained on user code or memory entries. This is enforced by infrastructure separation — memory storage has no path to training pipelines.
  2. Memory is per-account isolated. Cross-account access is impossible at the database layer. No row can be read by a query that does not present the owning account's credential.
  3. Memory is never shared with ad networks. The ad SDK (13-ads-monetization.md) has no API to query memory. Memory content is excluded from every telemetry event by schema design.
  4. Memory is user-deletable. picasso memory clear --confirm clear-memories purges everything within 60 seconds, including from cold storage. Account deletion (11-auth-and-accounts.md) is a superset.
  5. Memory export is full-fidelity. Users can extract every memory entry as one JSON file ({schemaVersion: 2, records}). There is no proprietary lock-in.
  6. Memory never stores secrets. The write pipeline rejects or redacts secrets, env var values, provider credentials, private keys, sponsor metadata, ad metadata, and sensitive raw command output before persistence.
  7. Memory is namespace-isolated. Retrieval never crosses account, workspace, org, or repo boundaries unless explicit org/team shared memory is enabled and policy allows it.

Integration with /goal mode

Goal runs write extensively to memory as they progress. Every plan-mode iteration during a /goal produces a decision entry. Every accepted change reinforces or creates style entries. Every observed codebase fact gets recorded. By the end of a 14-hour /goal run, the project profile is rich and current. See 16-goals.md.

Cache harness selection

V2 cache harnesses treat memory as an input to context packing, not as a separate cache. Pinned decisions and stable project facts can appear in cacheable prompt sections when privacy policy allows it. Recent tool observations, fresh corrections, and volatile reflections stay in the dynamic suffix. Memory context must fit a configured token budget and record omitted entries with reasons.

Comparison to alternatives

Tabnine save_memoryDevin WikiCLAUDE.md filesPicasso memory
Persistent across sessionsYesYesYes (manual)Yes
Auto-populatedPartialYesNoYes
Semantic retrievalNoNoNoYes
Typed relationshipsNoNoNoYes
User-inspectablePartialCloud onlyYesYes (CLI + dashboard)
User-editablePartialCloud onlyYesYes
Per-namespace isolationNoNoPer-fileYes (account / workspace / org)
Style learningNoNoManualYes
Decision historyNoPartialManualYes

Acceptance criteria

Dependencies

Open questions