MemPalace: An AI Memory System That Recalls Everything in ~170 Tokens

This is my summary and take on MemPalace, a long-term memory system for AI agents created and written up by recca0120. All credit for the design and benchmarks belongs to the original author — I'm just sharing why I think it's clever. The full write-up is linked at the end.

Anyone who uses an AI assistant every day runs into the same wall: months of conversations, millions of tokens of history, and yet every new session starts from nothing. The usual patch is to stuff important notes into a file the AI reads on startup — but that file keeps growing until it eats your context window before you've even asked a question.

MemPalace attacks this from a different angle, and the headline claim is striking: recall everything from a ~170-token startup, with reported 96.6% accuracy on a long-term memory benchmark, running fully offline with no API calls.

The core idea: query memory, don't paste it

Instead of loading all your memories into the prompt, MemPalace keeps them in a structured vault that the AI searches on demand. Wake-up costs almost nothing; the assistant only pulls the specific memories it needs, only when a topic actually comes up. Memory becomes a database you query, not a wall of text you carry everywhere.

Structure borrowed from a memory palace

The organisation uses the ancient "memory palace" metaphor — remembering things by where they live. In MemPalace that becomes:

• Wings — one per project, person, or topic.
• Rooms — sub-topics inside a wing (auth, billing, deploy).
• Halls — corridors shared across wings for a type of memory: facts, events, discoveries, preferences, advice.
• Closets — compressed summaries that point back to the originals.
• Drawers — the verbatim original notes, kept losslessly.
• Tunnels — links between wings when the same topic shows up in more than one place.

The neat finding is that the structure itself does a lot of the work. According to the author's benchmarks, narrowing a search from "everything" down to the right wing and room lifts recall from about 61% to 95% — a ~34-point jump with no fancy algorithm, just knowing where to look.

AAAK: shorthand any model can read

The part I found most interesting is the compression format, called AAAK. It squeezes memories down roughly 30× by writing them as terse, structured abbreviations — think entity codes, pipes, and star-ratings instead of full sentences. A paragraph about a team and its decisions collapses into a few dense lines.

Crucially, there's no decoder. As the author puts it, any large language model reads it natively — it's structured English abbreviations, not binary encoding. So the same compressed note works whether you're on Claude, GPT, Llama, or Mistral.

Loading memory in layers

Memory is split into layers that load only as deep as needed:

• L0 — Identity (~50 tokens): who the assistant is. Always loaded.
• L1 — Critical facts (~120 tokens, in AAAK): team, projects, preferences. Always loaded.
• L2 — Room recall: recent sessions, pulled in when a topic surfaces.
• L3 — Deep search: a semantic sweep across everything, only when explicitly asked.

Startup is just L0 + L1 — roughly 170 tokens. Everything else is fetched on demand.

Why the token count matters

The efficiency isn't just elegant, it's a cost story. The author lays out the contrast: pasting a full 19.5M-token history is impossible; keeping rolling LLM summaries might run hundreds of dollars a year; MemPalace's wake-up is a fraction of a cent, and even with heavy searching it stays in single-digit dollars. When every session pays the startup cost, shrinking it from hundreds of thousands of tokens to ~170 changes the economics entirely.

My take

What I like about MemPalace is that it treats memory as an architecture problem, not a model problem. Two of its ideas are the kind you can borrow even without adopting the whole system: give your memory a shape so the AI knows where to look, and store facts in a compact form any model can read back. It lines up with something I keep re-learning — the hard part of AI memory isn't storing things, it's structuring them so recall is cheap and accurate.

If the topic interests you, read the original — it goes far deeper than this summary, with the real benchmarks and the reasoning behind each design choice.

Source & full credit: MemPalace: 170 Tokens to Recall Everything — A Long-Term Memory System for AI Agents by recca0120.