AI agents have a goldfish memory. Here's how to fix that.
AI agents have a terrible memory. As bad as a goldfish. Past a certain point, they just forget. That's not a criticism, it's a fact you have to design around if you're building an agentic system yourself rather than working inside someone else's harness. This is for the builders, not the harness users.
The idea to hold onto
Every message you send an agent, and every reply it gives you, gets fed back in on the next turn. That running history, plus the system instructions, is your context. It has a hard limit, 200,000 tokens on some models, a million on others, but a limit all the same. Once you hit it, something has to give. The normal answer is compaction, summarise the conversation and start again from the summary. Do that a few times and the agent can't remember something you did last week. Sometimes it can't remember something from yesterday. This isn't rare. It happens day to day.
So you need a deliberate answer to "how does this agent remember", not just "however much fits in the window this time". There are four techniques that I've used (there are many more), roughly basic to complicated. Each has a real tradeoff. None of them is free.
Fix one: a memory file
The simplest option. Tell the agent, in its instructions, that whenever something important happens, a decision gets made, or it works out a way to do something, it writes it to a file. If you're running an online system that's probably a database row rather than a literal file, but the mechanic is the same.
The placement matters more than it sounds like it should. You append to the end of the message history, near the last message, not near the system instructions at the start. Putting it up near the system prompt means it changes every time something new gets written, and a changing prefix busts your prompt cache. Put it at the end instead, and the earlier part of your context stays stable and cache-cheap.
The gotcha: the file only grows. Eventually it gets too big and starts eating into your context on its own account. You'll need to trim it.
Fix two: RAG
Instead of passing everything in on every turn, RAG (retrieval augmented generation) retrieves what's needed only when it's needed. You put your information in a vector database and search it by meaning rather than by keyword. Ask "what was that article we talked about last week on context limits" and a semantic search over timestamped entries pulls back the likely matches.
The tradeoff: you now have a database to maintain, and once it gets big, querying it well gets harder. It's also a bit clunky mechanically. Every lookup is a tool call, which means more context added to the history for the tool call itself, which is more tokens spent, just to save tokens elsewhere.
Fix three: the file system
If your agent has file system access, you can lay memory out as folders, ideas here, decisions there, contacts somewhere else, all plain markdown files. You pass in an index every turn, essentially a table of contents describing what each file is and what it's about. When the agent needs something, it looks up the index, finds the right file, and loads it in that turn with a tool.
Works well for a local agent. Doesn't really work if you're running on a server, you can make it work, but it's not the most optimal way of doing it. Good option for local tooling, weak option for anything server-side.
Fix four: observational memory
This is the one that sits between "cram it all in" and "fetch it on demand", and it's the one I run in my own agent system. Credit to Mastra, first people I saw do it.
The mechanic: watch the conversation history. Once it hits a certain token size, summarise it into observations, a trimmed-down version of what happened. Pass the observations through instead of the raw messages, and carry on the conversation from there. Once the observations themselves get too big, distil the older ones down again into reflections, a further layer of compression. Cap the observations, cap the reflections, and you can keep the live messages at a genuinely finite number of tokens indefinitely.
Back that with a vector database as well, pointers into the specific things that happened, and you've effectively got limitless memory. In the system I built this for, I added one more layer on top, an era summary sitting above the reflections, so the stack is observations, reflections, era summary, plus the vector database underneath for exact recall.
It works. My agents remember things that happened since they were "born" (I think of their persona creation date as a birthday, which says something about how much time I spend around them). No cache-busting, no unbounded growth, and no forgetting last week just because this week got long.
The takeaway
There's no single right answer, only a right answer for your constraints. A memory file is cheap to build and fine until it isn't. RAG buys you scale at the cost of a database and an extra tool round-trip. The file system is elegant locally and awkward on a server. Observational memory, capped observations plus capped reflections plus a vector database, is the most work to build but the closest thing to memory that doesn't run out. Pick based on what you're actually building, not on which one sounds most impressive.