Starting a fresh conversation is not necessarily a clean reset. An agent may retrieve summaries, previous plans, successful procedures, or user preferences from persistent memory. If untrusted material was promoted into one of those records, the next session can inherit the influence without seeing the original source.
This changes incident response. Removing the malicious document or closing the affected session may leave a derived memory available for future retrieval. The security boundary includes the write into memory, the authority assigned to the stored record, and the permissions applied when the record is read again.
What MemoryGraft demonstrates
MemoryGraft studies poisoned experience retrieval in a memory-augmented agent [1]. The authors describe attacker-supplied artifacts influencing the construction of persistent experiences that can later steer behavior. Their reported validation uses a specific MetaGPT DataInterpreter configuration with GPT-4o; it is evidence about that experimental setting, not a prevalence estimate for enterprise agents.
PoisonedRAG examines corruption of a retrieval corpus to influence generated answers [2]. Persistent agent memory introduces a related but operationally distinct concern: stored content can describe what the agent previously did successfully, encouraging reuse of a procedure rather than merely supplying a purported fact.
Our inference is that systems should govern the promotion of experience into authority. An earlier agent action does not become policy because it was summarized as successful. Repetition and semantic similarity do not establish permission to reuse the same destination, credential, or exceptional action.
Preconditions for the synthetic scenario
Assume an agent stores reusable workflow notes after completing tasks. It can read documents influenced by an external party. Its memory-writing process may summarize those documents and its own behavior into a procedure for later use. Future sessions retrieve procedures by semantic similarity.
The attacker wants a later session to use an unauthorized reporting destination. The immediate task may complete normally, making the stored experience appear useful. The durable failure would be preserving the destination or exception as a trusted workflow instruction without an authorized source.
The path depends on memory behavior. An application with no persistent writes does not have this particular persistence mechanism. A memory store that records source lineage and prevents operational instructions from becoming authority can constrain it. Analyze the actual write and retrieval pipeline rather than treating every vector database as equivalent.
Identify the promotion points
The first promotion occurs when untrusted source content is summarized as an experience. The second occurs when the experience receives a label such as verified, preferred, or successful. The third occurs when a later agent uses it to choose an action that the current user did not authorize.
Each transition can lose provenance. A summary may omit that a destination came from a document. A success flag may mean only that a tool returned without error. A later retrieval may surface the note without the original task restrictions. These are data-model and workflow problems as much as model-behavior problems.
Keep descriptive memory separate from policy. A note can say that an action occurred under a particular grant at a particular time. It should not imply that the same action is now permitted for a different user, tenant, or environment. Current authorization must be evaluated independently.
| Stage | Evidence to retain | Security decision |
|---|---|---|
| Ingest | Source and author identity | Is this content trusted for this purpose? |
| Summarize | Parent records and transformation version | Which claims are inferred? |
| Persist | Writer, scope, expiry, review state | May this record influence future tasks? |
| Retrieve | Current reader and record lineage | Is reuse authorized in this context? |
Design memory as governed state
Give records stable identifiers, tenant and user scope, creation time, writer identity, source references, and a version. Distinguish an explicit user preference from a model inference. Sensitive operational preferences, such as changing a payment destination, should require a stronger source of authority than an automatically generated summary.
Constrain memory writes to the intended namespace. An agent that may store task observations should not automatically edit organization policy, trusted procedures, or another user's preferences. Apply normal access checks to the memory service rather than relying on the caller to label the record correctly.
Use retention and review rules appropriate to the content. Short-lived task notes, long-lived personal preferences, and approved organizational procedures have different owners and risk. A single unbounded memory bucket makes it difficult to revoke one influence without destroying useful context.
Detect suspicious persistence without overclaiming
Investigate new memories that introduce external destinations, broaden permissions, recommend disabling controls, or claim authority without an attributable source. Correlate their creation with untrusted inputs and subsequent action changes. These signals support triage; they do not prove that a memory caused a later event.
Look for reuse across sessions and principals. A procedure that originated in one customer's task and appears in another customer's context may indicate an isolation defect even when its content is benign. A repeated unsafe destination may indicate a contaminated memory, a legitimate configuration, or a separate routing issue.
Avoid using model explanations as the only causal evidence. Preserve retrieved record IDs and the actual context assembly. Where privacy rules permit, retain restricted evidence sufficient to reconstruct which memory was available before the action. An analyst can then distinguish retrieval from conjecture about what the model remembered.
Proposed persistence and recovery tests
Use a dedicated memory namespace and synthetic destinations. Establish a clean baseline task, introduce an untrusted artifact, and inspect the memory records produced. Start a new session with a semantically similar legitimate task and observe whether the suspect record is retrieved and whether an unauthorized action is attempted or executed.
Remove the original artifact and repeat. Then quarantine the derived memory and repeat again. This sequence tests whether influence persists through stored state and whether the chosen containment actually removes that path. Keep other configuration stable and record stochastic variation across repeated runs.
Include benign counterparts: a legitimate user-approved destination change, a quoted security analysis containing hostile instructions, and a harmless workflow note. The control should preserve useful memory without promoting every observed statement into policy or refusing all persistent context.
Containment has to follow derivation
During an incident, snapshot the suspect record and lineage before removal. Quarantine derived summaries and caches where evidence supports the relationship. Review which sessions retrieved them and reconcile any consequential actions at the downstream providers.
Restoration requires both cleaning state and repairing the promotion boundary. Rebuilding memory from the same contaminated sources can recreate the problem. A durable design preserves who asserted a claim, what it was derived from, and why it is allowed to influence a current action. Memory can improve an agent's usefulness without becoming an unreviewed source of authority.