The importance of memory for AI

AI systems are the ultimate amnesiacs. Despite an impressive ability to generate text, code, music, and more, they’re limited by the prompt immediately in front of them. Ask ChatGPT about a recipe it recommended last week, and you’ll likely get a confused response or even a hallucinated answer. Large language models (LLMs) are fundamentally stateless: They process each query as if it’s brand new, with no accumulated learning or personalization.

This, however, is changing. All the leading LLM providers are exploring ways to bring memory to AI, which promises to dramatically change the impact AI can have. According to Richmond Alake, a leading developer voice in AI (and my former colleague at MongoDB), “Memory in AI isn’t entirely new … but its application within modern AI agents is … revolutionary.” Why? Because “true personalization and long-term utility depend on an agent’s ability to remember, learn, and adapt.” In other words, real intelligence isn’t just about crunching billions of words in a neural network—it’s also about remembering relevant information at the right time.

As such, memory is emerging as the missing piece for AI, the factor that could turn today’s forgetful bots into adaptive companions. The big question now is how to give our AI systems this much-needed memory. It turns out the solution is not very glamorous: databases.

Databases fuel AI’s external memory

Yes, databases. It’s true that databases don’t show up anywhere on the lists of top 10 buzzwords powering our industry conversations about AI today. MCP servers! GANs! We’ve already moved past retrieval-augmented generation (RAG) (so 2024!) and are neck deep in agentic systems. Never mind that “literally nobody knows what an agent is,” as Santiago Valdarrama argues. But beneath the shiny allure of all these rapidly shifting trends in AI is data. And databases hold that data.

In traditional software, databases have always been the source of truth, the long-term storage for state and data. Now, in the era of generative AI, databases are taking on a new role as the memory layer of the AI stack. In fact, vector databases have become an integral part of the genAI technology stack because they address key LLM limitations such as hallucinations and the lack of persistent memory. By storing knowledge in a database that the AI can query, we effectively give these models an external brain to complement their built-in intelligence.

As Alake outlines in an informative video, there are a few key ways to think about (and use) memory for AI:

• Persona memory stores the agent’s identity, personality traits, roles, expertise, and communication style.
• Toolbox memory contains tool definitions, metadata, parameter schemas, and embeddings for the agent’s capabilities.
• Conversation memory stores the history of exchanges between the user and the agent.
• Workflow memory tracks the state of multistep processes.
• Episodic memory stores specific events or experiences the agent has encountered.
• Long-term memory (knowledge base) provides the agent with a persistent store of background knowledge.
• Agent registry is a repository for facts and information about entities the agent interacts with, such as humans, other agents, or APIs.
• Entity memory stores facts and data associated with the various entities an agent interacts with during its operation.
• Working memory serves as a temporary, active processing space, which is implemented through the large language model’s context window.

That’s a lot of “memories,” but how do we bring them to life? The industry is still figuring that out, but for most enterprises today, RAG is the most common way of improving an AI application’s memory. In RAG, the AI pulls in relevant facts from a knowledge base (database) to ground its answers. Instead of relying solely on what’s packed in the model’s training (which may be outdated or too general), the AI performs a search in an external store, often a vector database, to retrieve up-to-date or detailed information. This allows the system to “remember” things it was never explicitly trained on, for example, a company’s internal documents or a specific user’s history, which it can then incorporate into its response.

By augmenting prompts with data fetched from a database, AI systems can hold a coherent conversation over time and answer domain-specific questions accurately, essentially gaining state and long-term memory beyond their fixed model parameters. It’s a way to ensure that AI doesn’t start from zero every time; it can recall what was said earlier and tap into facts beyond its training cutoff. In short, databases (particularly vector stores) are proving essential to AI’s long-term memory.

Vectors, graphs, and hybrid memories

Not all memories are created equal, of course, and not all databases work the same way. As an industry, we’re currently experimenting with different database technologies to serve as AI memory, each with strengths and trade-offs. As mentioned, vector databases are the poster child of AI memory. They excel at semantic similarity search, finding pieces of information that are related in meaning, not just by keywords. This makes them ideal for unstructured data like chunks of text: Ask a question, and find the passage that best answers it.

As has become the norm in AI, we had a brief fling with standalone vector databases (Weaviate, Pinecone, etc.). That fling didn’t last long, as every major database vendor (including my previous and current employers, MongoDB and Oracle) added vector search capabilities to their core database. Back in 2023 AWS announced plans to add “vector capabilities to all our database services in the fullness of time.” Today, most of its database services include vector capabilities. At AWS, Oracle, MongoDB, and others, this vector addition enables developers to store vector embeddings alongside operational data.

In other words, the line between application database and AI memory store is blurring.

Still, vector search alone isn’t a silver bullet for all memory problems. One limitation is that pure semantic similarity can miss context, such as timing or relationships. A vector query might surface a months-old fact that’s technically similar but contextually stale or out of date. This is where other data stores such as graph databases enter the picture. Knowledge graph techniques store information as nodes and edges. Think of it as a web of facts linked by relationships (who is CEO of what company, when a document was created, etc.). Such structured memory can help an AI distinguish when something happened or how facts connect. For example, if you ask “What restaurant did you recommend to me yesterday?” a graph-based memory can filter results by the explicit date of recommendation, not just semantic similarity. Graphs can thus provide a form of temporal and contextual awareness that vector search alone cannot.

They also offer auditability. You can trace why the AI retrieved a certain fact via the relationships, which is useful for debugging and trust. Startups like Zep are exploring hybrid approaches that combine vectors with graphlike linkages to get the best of both worlds. The downside? Graph-based memory requires defining a schema and maintaining structured data, which can be complex and may not capture every nuance of unstructured text. For many applications, a simple vector store (or a vector-enhanced document database) offers a happy medium between ease and effectiveness.

We’re also seeing hybrid search approaches: combining traditional keyword queries with vector similarity. This can filter results by metadata (date ranges, user ID, or tags) before doing the semantic match, ensuring that what the AI “remembers” is not only relevant in meaning but also in context. In practice, AI developers often use a blend of techniques: a short-term memory buffer for recent interactions, a vector database for long-term semantic recall, and sometimes a relational or document database for explicit facts and user-specific data. These pieces together form a rudimentary memory hierarchy: fast ephemeral memory (context window) plus persistent searchable memory (database). The database essentially acts as the AI’s hippocampus, storing experiences and knowledge that can be retrieved when needed to inform future reasoning.

Ending AI’s amnesia

For all the buzz about neural networks and model sizes, it’s the humble database—the technology of records and transactions—that is quietly redefining what AI can do. By plugging in a database, we give AI working memory and long-term memory. It can now maintain state, learn new information on the fly, and retrieve past knowledge to inform future decisions. It’s not sexy, but it is essential.

Challenges remain, of course. Engineers are figuring out how to manage AI memories at scale, deciding what to store or forget to prevent information overload, ensuring relevant facts win out over stale data, and guarding against “memory poisoning” where bad data corrupts the AI’s knowledge. These are classic data management problems wearing a new AI costume. Solutions will no doubt borrow from database science (transactions, indexing, caching) and new techniques (smarter context pruning and embedding models). The AI stack is consolidating around the idea that models, data, and memory all have to work in concert. All of this means the next time an AI assistant smoothly recalls your last conversation or adapts its answers based on a quirk you mentioned weeks ago, a database is behind the scenes, quietly serving as the memory bank for the machine’s synthetic mind.