Long-term memory design for AI agents refers to the architectural design that enables agents to retain and utilize business context across conversation sessions.

Standard RAG and context windows alone make it difficult to continuously retain the flow of work spanning weeks or months. This article explains design patterns for achieving long-term context retention by combining MemGPT from UC Berkeley and GraphRAG from Microsoft Research.

The target audience is engineers involved in the development and operation of AI systems, and architects who feel challenged by agent context management. By the end of this article, you will have learned the following.

Have you ever felt uneasy about using AI agents for business purposes upon hearing that they forget everything once a session ends?

Context windows have a physical upper limit, and as conversations grow longer, older information is pushed out first. For simple one-off question-and-answer interactions, this is sufficient—but for tasks such as multi-day project management or making proposals based on per-customer interaction histories, an agent that cannot carry over "what was discussed last time" quickly reveals its limitations.

Designing long-term memory means building a mechanism that accumulates information outside this window and retrieves it when needed. Without careful design around what to store, where to store it, and when to reference it, memory simply grows into unusable noise. The following sections explore three perspectives in turn: the structural differences from short-term memory, how to apply this to real business scenarios, and the classification of memory types.

The context window is the upper limit of text that an LLM can reference in a single inference. According to the MemGPT paper (Table 1), the limits at the time were approximately 4,000 tokens for llama-2 and 8,000 tokens for GPT, and any information exceeding these limits cannot be referenced within that session.

It is tempting at first to think that "making the context longer will solve the problem," but simply extending the window length cannot retain the history of work spanning weeks or months. This is because memory is reset when a conversation session ends, and the next session begins from a blank slate.

The difference between short-term and long-term memory is not merely a matter of retention duration. Short-term memory is constrained by a token limit, whereas long-term memory is written to external storage, making its capacity virtually unlimited and allowing it to persist across sessions. The update mechanisms also differ: while short-term memory is automatically overwritten by the flow of conversation, long-term memory can be controlled through explicit triggers for writing and updating. In terms of retrieval as well, unlike short-term memory which references the entire content within the window at once, long-term memory selectively retrieves only the necessary fragments.

In business systems, past interactions with customers and the history behind decisions directly affect subsequent judgments. To leverage these across sessions, a long-term memory mechanism integrated with external storage—rather than an extension of short-term memory—is essential.

The necessity of long-term memory varies depending on the continuity and complexity of the work. Standard RAG is sufficient for one-off question-and-answer interactions, but tracking the same project over weeks or months requires context retention across sessions.

Long-term memory design becomes especially important in scenarios such as the following:

• Customer support handoffs: If an agent does not retain the bugs a customer previously reported or their interaction history, the customer will be asked to repeat the same explanations every time, significantly degrading the customer experience.
• Project management assistants: When tracking the history of decisions and reasons for changes across sprints, consistent advice cannot be provided without the context of past discussions.
• Internal knowledge accumulation: In use cases where an agent continuously learns and accumulates operational know-how known only to specific personnel through ongoing dialogue, memory continuity between sessions is a prerequisite.
• Legal and compliance responses: In situations where new cases must be handled while referencing the history of contract negotiations and the basis for past decisions, simple document retrieval is insufficient.

When business complexity is low and inquiries are independent, the context window is adequate—but for work involving multiple stakeholders where decisions depend on past history, a long-term memory architecture is necessary.

What these business scenarios have in common is that "past context directly affects the quality of present decisions."

The question "I'm not sure what I should have the agent remember"—is one that often arises in the early stages of design. Referring to memory classifications from cognitive science reveals the structure needed for long-term memory design in AI agents.

The memory that an agent should retain can be broadly classified into three types:

• Episodic memory: A chronological record of events—"when, with whom, and what was discussed." Examples include the background behind a specification change decided at last week's meeting, or the priorities of requirements a user indicated in the past. It forms the foundation supporting conversational continuity across sessions.

• Semantic memory: A knowledge system of concepts, rules, and terminology related to the business domain. It is a collection of facts that hold true independent of context—such as "this customer prefers end-of-month billing" or "Product A is exempt from a specific regulation." This is also an area where GraphRAG particularly excels.

• Procedural memory: Task execution procedures and workflow patterns. It is formalized knowledge of repeatedly performed operations—such as "the approval flow goes in the order of direct supervisor → legal → accounting" or "when an error occurs, first check the logs before retrying."

If all three types are stored together in a single vector store without separation, there is an increased risk that memory types cannot be distinguished at retrieval time, causing irrelevant context to be surfaced. It is important to separate storage at the design stage and assign an indexing strategy appropriate to each memory type. The limitations of standard RAG covered in the next section will also be easier to understand when kept in mind alongside this classification.

"Do you remember what we talked about last week?"——A phrase that humans understand naturally, yet AI agents sometimes don't. Standard RAG works by using vector search to retrieve relevant documents in response to a query. While it functions well enough for one-off lookups, it is simply not designed to retain context that spans a timeline—such as "last week's discussion" or "the background behind a previous decision."

When a session ends, memory resets. In the next conversation, you have to explain the situation all over again from scratch. When you try to use this in a business context, this limitation quickly becomes a wall. In the sections that follow, we will dig into three specific challenges that RAG alone cannot fully address.

Implement vector search and the long-term memory problem is solved——or so the thinking goes. But push forward with that approach and you may soon hit a wall: "I'm retrieving semantically similar fragments, yet the conversation still doesn't add up."

The core of the problem with standard RAG is that its scoring looks only at chunk-level similarity. Because each chunk is evaluated independently, causal relationships and chronological flows that span multiple sessions cannot be represented. Temporal dependencies—such as "the policy decided in last week's meeting" or "the approval workflow changed several months ago"—often do not appear as neighbors in vector space, causing a break in context. On top of that, retrieving a large number of relevant chunks quickly consumes the LLM's context window. As the MemGPT paper (arXiv: 2310.08560) points out, the context lengths of leading models at the time were limited to a few thousand tokens (e.g., llama‑1 = 2,000 tokens, llama‑2 = 4,000 tokens, gpt‑3.5‑turbo = 4,000 tokens, gpt‑4 = 8,000 tokens), making a design that passes large numbers of chunks all at once fundamentally impractical.

There is also a more fundamental issue: vector search only decides what to retrieve—it has no logic for when to update memory or how to handle contradictory information. As a result, old and new information can appear mixed together in search results, creating the risk that the agent responds based on incorrect assumptions.

When business logs reach the scale of tens of thousands of entries, vector search accuracy tends to degrade faster than expected.

One cause is the co-mingling of entries that are semantically similar but contextually different. For example, the phrase "pending approval" appears in both budget-request approval workflows and system access-rights approvals. Because the two are close in vector space, unrelated documents easily surface near the top of results.

Organizing the noise sources specific to large-scale logs yields the following:

• Accumulation of duplicate and near-duplicate entries: Logs generated repeatedly in a fixed format produce large numbers of nearly identical vectors, homogenizing search results.
• Persistence of stale information: When old logs are not deleted after organizational or process changes, contradictory contexts hit simultaneously.
• Uneven granularity: When detailed procedure documents and single-line notes coexist in the same index, chunk weights become unbalanced.

The severity of accuracy degradation varies with the nature of the logs. When structured logs (CSV or DB records) are the primary source, the issue can be addressed by tuning chunk-splitting granularity; when unstructured text (emails, meeting minutes, chat histories) is mixed in, a noise-removal preprocessing step is indispensable.

These problems cannot be solved simply by increasing the index size. In fact, the larger the set of retrieval targets, the greater the risk that irrelevant entries float to the top of the scores. The combination of MemGPT and GraphRAG explained in the next section is an approach that addresses this structural problem at the architecture level.

Have you ever felt: "I integrated standard RAG, so why doesn't the agent remember what was discussed in last week's meeting?" That question is precisely the core of what MemGPT and GraphRAG set out to solve.

The two complement each other's respective weaknesses.

The domain MemGPT complements

• It provides a mechanism for offloading and retrieving information that exceeds the physical upper limit of the context window (8,000 tokens for GPT-4 in Table 1 of the paper) to and from external storage.
• It excels at retaining "episodic memory" that persists across sessions.
• Function calls allow the agent itself to control the timing of memory reads and writes.

The domain GraphRAG complements

• It holds, in a graph structure, the entities and relations extracted from large volumes of business documents—capturing "who is involved with what, and in which project."
• Relationships that span multiple documents and are difficult for vector search to capture can be retrieved cross-sectionally via graph queries.
• GraphRAG, published by Microsoft Research in February 2024, is reported to significantly improve answer accuracy on complex reasoning queries, even though graph extraction accounts for approximately 75% of the index-construction cost.

The effect produced by combining them

MemGPT manages the chronological context of "when, with whom, and what was discussed," while GraphRAG provides the structural knowledge of "how that person or concept is connected within the organization."

Conclusion: MemGPT mimics the memory hierarchy of an operating system and is designed to achieve long-term memory that transcends the context constraints of LLMs.

MemGPT is a framework published in 2023 by a research team including UC Berkeley, and it manages memory by separating a main context from external storage. The H3 sections that follow explain, in order, the architecture's structure, its initial configuration, and the procedure for controlling function calls.

When first encountering MemGPT, it is easy to dismiss it as "just a memory extension plugin." In reality, however, it is a hierarchical architecture inspired by OS memory management, and understanding its design philosophy is central to implementing long-term memory.

The MemGPT paper (UC Berkeley et al., October 2023) proposes a structure that treats the LLM as an OS, the context window as "main memory (RAM)," and external storage as a "disk." According to a comparison table in the paper, the context lengths of major models at the time were limited to 2,000 tokens for llama-1 and 8,000 tokens for GPT-4, making it structurally difficult to fit long-term business context into a single window.

The architecture consists of the following three main layers:

• In-Context Storage: The active working area directly accessible by the LLM. Holds the system prompt, recent conversations, and key summaries.
• External Storage: Long-term memory stored in a vector DB or KV store.

When setting up MemGPT, note that the procedure diverges depending on whether you use a local environment or a cloud API. If you use a local LLM (such as LM Studio), you will additionally need to download a model and configure the API endpoint. If you use an OpenAI API key, you only need to set an environment variable and can proceed to verification in the shortest number of steps.

Installation Steps (Using the OpenAI API)

First, prepare an environment with Python 3.10 or higher.

After installation, run the following command to launch the interactive initial configuration wizard.

The wizard walks you through the following settings in order:

• LLM provider: Select openai (or local for a local LLM)
• Model name: Specify the model to use
• Embedding model: The default OpenAI embedding model is recommended
• External storage backend: Select chroma (local) or postgres

Creating the Initial Agent

Once configuration is complete, create an agent with the following command:

On first launch, you will be prompted to enter the system prompt (persona/character settings) and the initial values for personal memory.

A common complaint in development environments is: "I've set up MemGPT, but I can't control when the agent writes to memory and when it reads from it." In MemGPT, this control is achieved through the function calling mechanism.

A MemGPT agent manipulates memory by calling dedicated functions alongside ordinary text responses as LLM output:

• core_memory_append: Appends information to the fixed memory area within the main context
• core_memory_replace: Overwrites and updates an existing memory entry
• archival_memory_insert: Saves information to long-term external storage
• archival_memory_search: Retrieves information from external storage by keyword or semantic similarity

These functions are embedded as definitions in the system prompt, and the LLM itself reasons about and decides which function to call. For example, if a user says "Going forward, hand off this project to person A," the agent will call core_memory_replace to update the responsible party information.

There are three key points to keep in mind during implementation:

Conclusion: GraphRAG extracts entities and relations from business documents and indexes them as a knowledge graph, enabling deep retrieval across contexts.

GraphRAG is a project published by Microsoft Research in February 2024. Its distinguishing feature is the ability to convert text into a graph structure and perform searches while preserving the relationships between entities.

It is tempting to think that "extracting as many entities as possible will improve accuracy," but in practice, narrowing the granularity of entities and limiting them to units that are meaningful in a business context yields higher search quality. Over-extraction increases graph noise and raises the cost of path traversal at query time.

Basic Principles for Entity Design

When designing entity extraction for business documents, deciding the following three points in advance will smooth out downstream processes:

• Enumerate the target domain: Identify noun classes that appear repeatedly in business operations, such as "customer," "project," "person in charge," "product," and "date"
• Unify granularity: Define normalization rules in advance so that, for example, "ABC Co., Ltd." and "ABC Corp." are treated as the same entity
• Set exclusion rules: Exclude common nouns and frequently occurring stop words from entity candidates to prevent graph bloat

Key Points for Relation Definition

Relations between entities are fundamentally defined on the basis of verb phrases. For example, using verbs aligned with business workflows—such as "is responsible for," "approved," and "depends on"—makes it easier to retrieve intended paths in subsequent graph queries.

The GraphRAG documentation shows examples of setting the text chunk size to 50–100 tokens; note that while smaller chunks make it easier to capture local co-occurrence relationships between entities, they also increase the cost of graph extraction.

Explicitly specifying whether each relation is bidirectional or unidirectional also helps maintain consistency in query design.

When feeding business documents into GraphRAG, pipeline design has a significant impact on quality. It is important to structure the processing steps appropriately according to the type and volume of documents.

The basic pipeline flow is as follows:

1. Preprocessing (Chunking): The GraphRAG documentation shows example text chunk size settings of 50–100 tokens. For documents with clear contextual boundaries, such as meeting minutes and specifications, smaller chunks are effective; for long-form reports and contracts where inter-paragraph dependencies are strong, setting slightly larger chunks tends to improve entity extraction accuracy.
2. Entity and Relation Extraction: Entities and relationships are extracted from chunks using an LLM. The documentation states that this step accounts for approximately 75% of the graph index creation cost, making it important to estimate the balance between processing volume and cost in advance.
3. Graph Construction and Storage: Extraction results are stored in a graph DB (such as Neo4j) or GraphRAG's internal storage.

As a decision framework by document type, for internal knowledge bases (FAQs, procedure manuals) with low update frequency, a design that periodically re-indexes via batch processing is rational. On the other hand, for rapidly changing data such as daily business logs and CRM notes, a configuration that prepares a differential extraction pipeline to incrementally update the graph is more appropriate.

"We built the graph, but how do we actually query it to retrieve business context?"——this is a question frequently heard in the field.

In GraphRAG, two primary search modes are used selectively against the constructed knowledge graph.

• Local Search: A localized search that starts from a specific entity or relationship and traverses neighboring nodes. It is well-suited for retrieving concrete context, such as "this customer's past inquiry history."
• Global Search: A search that aggregates community summaries to grasp broad-level trends. It is suited for bird's-eye-view questions such as "what were the overall challenges in last month's operations?"

The basic implementation flow is as follows:

1. Convert the query string into entity candidates and identify the corresponding nodes in the graph
2. Expand adjacent edges and related communities from the nodes to retrieve a subgraph
3. Insert a text representation of the subgraph into the LLM's context to generate a response

The documentation's official example settings show chunk sizes of 50–100 tokens; finer segmentation increases node granularity and tends to improve accuracy, while also increasing graph extraction costs. Note that the documentation states graph extraction accounts for approximately 75% of the total index creation cost, so chunk size settings must be determined with awareness of the cost-accuracy tradeoff.

Conclusion: Integrating MemGPT and GraphRAG enables the realization of a robust long-term memory foundation that combines cross-session memory management with knowledge graph search.

Clearly defining the roles of each component and appropriately configuring write triggers and search scoring are the keys to a successful integration.

When combining MemGPT and GraphRAG, it is tempting at first to think "just call both in parallel," but in practice, a hierarchical design with clearly separated roles leads to more stable behavior.

In the integrated architecture, the following responsibilities are assigned to each component:

• MemGPT (Session Memory Layer): Manages recent conversation history, user work state, and short-term instructions in the main context, offloading to external storage when capacity is exceeded
• GraphRAG (Knowledge Graph Layer): Holds relationships between entities, business rules, and structured information from past projects, making them searchable via semantic connections
• Orchestrator: Receives user input, references MemGPT's context, and issues queries to GraphRAG as needed to integrate the results

Data flows are processed in the following order:

1. User input → the orchestrator classifies intent
2. Check MemGPT's main context and determine whether the query can be resolved within the session
3. If not resolvable, issue an entity search to GraphRAG
4. Inject the retrieved knowledge into the main context, and the LLM generates a response
5. Important information is written to MemGPT's external storage and carried over to the next session

What is particularly important in this design is limiting queries to GraphRAG to "only when necessary."

When and how to write to memory is a core design decision that determines the quality of a long-term memory system.

Write triggers can be broadly classified into the following three types:

• Event-driven: Writing occurs when the user explicitly instructs "remember this," or when a business event such as task completion or approval takes place
• Inference-based: The agent evaluates the conversation content and autonomously writes when an importance score exceeds a threshold
• Scheduled batch: At the end of a session or at fixed intervals, a summary of that period is generated and saved

In designing update policies, it is important to explicitly define conditional branching. When new information about the same entity does not conflict with existing memory, it should be appended (Append); when a conflict or obsolescence is confirmed, it should be overwritten (Overwrite)—these policies must be clearly distinguished.

Have you ever felt that "even though GraphRAG retrieved the relevant nodes, the agent's answers are somehow off"? In many cases, the root cause is that the search scoring design does not align with the importance levels of actual business operations.

In an environment integrating MemGPT and GraphRAG, it is effective to combine the following elements in search result scoring:

• Semantic similarity: Base score derived from vector distance
• Graph proximity: Number of hops from the target node to referenced entities (fewer hops = higher score)
• Recency weight: Gradually decay the scores of older memory entries based on timestamps
• Access frequency: Add a bonus for the number of past references, prioritizing memories with a proven track record

By linearly combining these four factors and tuning the weights per task type, you can achieve search rankings that reflect your business context.

A fallback strategy is equally important. If a GraphRAG graph query returns empty results and is passed directly to the agent, answer quality drops sharply. The recommended three-tier fallback is as follows:

1. Graph query — Prioritizes entity relationship search
2. Vector search (standard RAG) — Executed when the graph yields no results
3. MemGPT summary memory — Referenced when the top two produce low scores

When a fallback is triggered, flag it in the logs and use it as a trigger for subsequent graph updates. This allows you to continuously fill gaps in memory.

When you bring an integrated system into a production environment, no matter how carefully you review the design beforehand, unexpected problems begin to surface once real business data starts flowing through. In the following sections, we take a concrete look at two failure patterns that are frequently encountered in the field, along with specific strategies to avoid each of them.

Immediately after introducing long-term memory, many teams tend to adopt the policy of "let's save everything we can record." In practice, however, it has been reported that as the number of entries grows, search latency degrades at a superlinear rate, causing response speed to become a bottleneck. Designing with restraint in what you store is the key to maintaining long-term performance.

The main measures for preventing bloat are as follows:

• Setting TTLs (time-to-live): Assign expiration dates to episodic memory entries, and automatically archive or delete entries that have not been referenced for a certain period. For data where freshness is critical—such as operational logs—a TTL of roughly 30 to 90 days is a reasonable design target.
• Priority management via importance scores: Assign an importance score at write time using MemGPT function calls, and compress or summarize low-scoring entries before storing them. Replacing detailed original text with summarized entries reduces both storage and retrieval costs simultaneously.
• Optimizing chunk size on the GraphRAG side: According to the documentation, example text chunk size settings range from 50 to 100 tokens. Chunks that are too fine-grained cause an explosion in node count, so adjustment to match the granularity of your business documents is necessary.
• Periodic graph pruning: Regularly delete isolated nodes and low-frequency edges in batch jobs to keep graph density within a defined range.

Combining these measures allows you to maintain a balance between memory freshness and search speed.

Contradictions in memory entries occur when information about the same entity is written at different points in time as separate entries. For example, if the record "the person in charge is Tanaka" and the record "the person in charge is Suzuki" coexist, the agent becomes unable to determine which one to reference.

Two main approaches are effective for detecting contradictions:

• Timestamp comparison: When multiple entries exist for the same entity and attribute, treat the entry with the most recent timestamp as authoritative and assign an "obsolete" flag to older entries.
• Embedding-based similarity check: When writing a new entry, extract existing entries whose cosine similarity exceeds a threshold (e.g., 0.90 or above) as candidates, and delegate contradiction judgment to an LLM.

Resolution policies need to be applied differently depending on the situation. When an entry update is based on an explicit user utterance (e.g., "the person in charge has changed"), overwrite the old entry. When the change stems from an implicit contextual shift, the safer approach is to place both entries on hold, assign confidence scores, and manage them by priority.

When using GraphRAG, a design that records the change history of node attributes as edges is useful for tracking contradictions. On the MemGPT side, it is recommended to embed duplicate-checking logic into the function that writes to archival storage, and to implement a step that prompts the agent itself to confirm whenever a contradiction candidate is detected.