Cross-Agent Memory: Sharing Context Without Sharing Hallucinations
By Diesel
multi-agentmemoryshared-state
## The Memory Dilemma
Agent A discovers something important. Agent B needs that information to do its job. Without shared memory, Agent B either rediscovers it (wasting time and tokens) or operates without it (producing inferior results).
Simple solution: shared memory. Agent A writes, Agent B reads.
Now the problem: Agent A hallucinated. It wrote confidently incorrect information to shared memory. Agent B reads it, treats it as ground truth, and builds on it. Agent C reads both Agent A's hallucination and Agent B's hallucination-derived work. By Agent D, the entire system is operating in a parallel reality that sounds perfectly coherent but is completely wrong.
Shared memory without validation is shared hallucination. This is the problem nobody warns you about until you've shipped a multi-agent system that confidently produces elaborate fiction.
## Memory Architecture
First, the storage layer. You need more than a key-value store. You need typed memory with provenance, confidence, and expiry. The related post on [memory architectures](/blog/agent-memory-patterns) goes further on this point.
```python
class MemoryEntry:
key: str
value: any
entry_type: str # "fact", "inference", "plan",
# "observation"
source_agent: str # Who wrote this
confidence: float # 0-1, how sure was the writer
based_on: list[str] # Keys of entries this derived from
created_at: float
expires_at: float | None
access_count: int = 0 # How many agents have read this
validations: list[Validation] = []
class Validation:
validator_agent: str
validated_at: float
result: str # "confirmed", "disputed", "uncertain"
evidence: str | None
```
Every memory entry knows who wrote it, what it's based on, how confident the writer was, and whether anyone else has validated it. This metadata is as important as the value itself.
## Write Policies
Not everything an agent produces should go into shared memory. Most of it shouldn't.
```python
class WritePolicy:
def should_store(self, entry, agent_history):
# Rule 1: Minimum confidence threshold
if entry.confidence < 0.7:
return False
# Rule 2: Facts require higher confidence than plans
if entry.entry_type == "fact" and entry.confidence < 0.85:
return False
# Rule 3: Agent's historical accuracy matters
agent_accuracy = agent_history.accuracy_rate(
entry.entry_type
)
if agent_accuracy < 0.6:
# This agent is wrong 40%+ of the time
# on this type. Don't trust it.
entry.requires_validation = True
# Rule 4: Deduplication
similar = self.memory.search_similar(
entry.value, threshold=0.9
)
if similar:
return False # Already know this
return True
```
Rule 3 is the critical one. Some agents are reliable for certain entry types and unreliable for others. Track accuracy per agent per entry type. An agent that's 95% accurate on code analysis but 40% accurate on architecture decisions should have its architecture opinions flagged for mandatory validation.
## Read Policies
Reading is where hallucinations spread. An agent that reads unvalidated memory and treats it as fact is patient zero in an outbreak of bullshit.
```python
class ReadPolicy:
def prepare_for_agent(self, entries, reader_role):
prepared = []
for entry in entries:
# Tag confidence level visibly
if entry.confidence < 0.8:
entry.value = (
f"[UNVERIFIED, confidence: "
f"{entry.confidence:.0%}] {entry.value}"
)
# Tag validation status
if not entry.validations:
entry.value = (
f"[UNVALIDATED by peers] {entry.value}"
)
elif any(v.result == "disputed"
for v in entry.validations):
entry.value = (
f"[DISPUTED] {entry.value}"
)
# Filter by relevance to reader
if self._is_relevant(entry, reader_role):
prepared.append(entry)
return prepared
```
The tagging is essential. When an agent reads "[UNVERIFIED, confidence: 65%] The API rate limit is 1000 requests per minute," it knows to treat that differently than a verified fact. Without tags, the agent treats everything in memory as equally authoritative.
## Namespace Isolation
Not all memory should be visible to all agents. Namespaces prevent information leakage and reduce noise.
```python
class NamespacedMemory:
def __init__(self):
self.namespaces = {}
def configure_access(self):
return {
"code-patterns": {
"write": ["coder", "reviewer"],
"read": ["coder", "reviewer", "tester"]
},
"security-findings": {
"write": ["security"],
"read": ["security", "coder", "supervisor"]
},
"architecture-decisions": {
"write": ["architect", "supervisor"],
"read": ["all"]
},
"task-state": {
"write": ["supervisor"],
"read": ["all"]
},
"working-memory": {
# Per-agent scratch space, no sharing
"write": ["self"],
"read": ["self"]
}
}
```
The `working-memory` namespace is important. Agents need scratch space for intermediate thoughts that shouldn't pollute the shared pool. An agent's chain-of-thought reasoning, dead-end explorations, and draft outputs stay private until the agent explicitly promotes something to a shared namespace. It is worth reading about [stateful agent design](/blog/stateful-vs-stateless-agents) alongside this.
## The Validation Pipeline
Memory validation is what separates a functioning shared memory from a shared hallucination engine.
```python
class ValidationPipeline:
def __init__(self, validators):
self.validators = validators
async def validate(self, entry):
"""Run entry through validation checks"""
results = []
# 1. Self-consistency: does it contradict itself?
consistency = await self._check_consistency(entry)
results.append(consistency)
# 2. Cross-reference: does it contradict validated
# entries?
xref = await self._cross_reference(entry)
results.append(xref)
# 3. Source verification: can we verify the claim?
if entry.entry_type == "fact":
verification = await self._verify_fact(entry)
results.append(verification)
# 4. Peer review: does another agent agree?
if entry.confidence < 0.9:
peer = await self._peer_validate(entry)
results.append(peer)
# Aggregate
passed = sum(1 for r in results if r.passed)
entry.validation_score = passed / len(results)
entry.validations.extend(results)
return entry
async def _cross_reference(self, entry):
"""Check against existing validated memories"""
related = self.memory.search_similar(
entry.value, namespace=entry.namespace
)
for existing in related:
if existing.validation_score > 0.8:
contradiction = await self._detect_contradiction(
entry.value, existing.value
)
if contradiction:
return ValidationResult(
passed=False,
reason=f"Contradicts validated "
f"entry: {existing.key}"
)
return ValidationResult(passed=True)
```
Cross-referencing against validated entries is the most effective check. If the shared memory already contains validated information that contradicts the new entry, something is wrong. Either the new entry is hallucinated or the old entry needs re-validation. Either way, the contradiction gets flagged rather than silently entering the system.
## Memory Decay and Garbage Collection
Memory that never expires becomes memory that's never relevant. Old entries crowd out new ones, stale information gets treated as current, and the memory store becomes a archaeological dig where agents find information from ten iterations ago.
```python
class MemoryDecay:
async def decay_cycle(self):
all_entries = self.memory.list_all()
for entry in all_entries:
age = time.time() - entry.created_at
# Access-based decay: unused memories fade
if entry.access_count == 0 and age > 3600:
entry.confidence *= 0.9
# Time-based decay: all memories fade eventually
if entry.entry_type == "observation":
# Observations go stale fast
entry.confidence *= 0.95 ** (age / 3600)
elif entry.entry_type == "fact":
# Facts decay slower
entry.confidence *= 0.99 ** (age / 3600)
# Below threshold: archive or delete
if entry.confidence < 0.1:
await self.memory.archive(entry)
else:
await self.memory.update(entry)
```
Different entry types decay at different rates. An observation ("the API returned a 500 error") becomes stale within minutes. A validated architectural decision is relevant for weeks. A coding pattern might be relevant forever.
## Semantic Search Over Memory
Key-value lookup is necessary but insufficient. Agents need to ask "what do we know about authentication?" and get relevant results regardless of how the information was keyed.
```python
class SemanticMemory:
def __init__(self, embedding_model):
self.embeddings = embedding_model
self.index = HNSWIndex(dim=384)
async def store(self, entry):
embedding = await self.embeddings.encode(entry.value)
entry.embedding = embedding
self.index.add(entry.key, embedding)
self.store.put(entry)
async def search(self, query, namespace=None,
min_confidence=0.5, limit=10):
query_embedding = await self.embeddings.encode(query)
candidates = self.index.search(
query_embedding, k=limit * 3
)
# Filter and rank
results = []
for key, similarity in candidates:
entry = self.store.get(key)
if namespace and entry.namespace != namespace:
continue
if entry.confidence < min_confidence:
continue
# Combine semantic similarity with confidence
# and validation score
rank = (
similarity * 0.5
+ entry.confidence * 0.3
+ entry.validation_score * 0.2
)
results.append((entry, rank))
results.sort(key=lambda x: x[1], reverse=True)
return [r[0] for r in results[:limit]]
```
The ranking formula is where the hallucination defense lives. A highly similar but low-confidence, unvalidated entry ranks below a moderately similar, high-confidence, validated one. Relevance without reliability is dangerous.
## The Contamination Problem
Even with all these safeguards, hallucinations can spread through dependency chains. Entry B is "based on" Entry A. If A is later invalidated, B is contaminated.
```python
class ContaminationTracker:
async def invalidate_cascade(self, entry_key):
"""When an entry is invalidated, find everything
that depends on it"""
contaminated = set()
queue = [entry_key]
while queue:
key = queue.pop(0)
dependents = self.memory.find_dependents(key)
for dep in dependents:
if dep.key not in contaminated:
contaminated.add(dep.key)
# Mark as potentially contaminated
dep.confidence *= 0.5
dep.validations.append(
Validation(
validator_agent="system",
result="contaminated",
evidence=f"Depends on "
f"invalidated: {key}"
)
)
await self.memory.update(dep)
queue.append(dep.key)
return contaminated
```
This is why the `based_on` field matters. Without provenance tracking, you can't trace contamination. With it, one invalidation triggers a cascade that halves the confidence of every downstream entry. Those entries don't get deleted. They get flagged for re-validation. This connects directly to [vector database backends](/blog/building-agent-memory-vector-databases).
## Practical Deployment
The full memory stack for a production system:
1. **Namespaced storage** with typed entries and provenance
2. **Write policy** that gates on confidence and agent history
3. **Read policy** that tags unverified/disputed entries visibly
4. **Validation pipeline** for cross-referencing and peer review
5. **Semantic search** with composite ranking (similarity + confidence + validation)
6. **Decay mechanism** with type-specific rates
7. **Contamination tracking** with cascade invalidation
Is this overkill for a three-agent system? Probably. Is it necessary for fifteen agents running in production? Absolutely. The contamination problem scales quadratically with agent count. What's a minor issue with three agents becomes an existential threat with fifteen.
Shared memory is the nervous system of a multi-agent architecture. Protect it like one. A nervous system that transmits false signals doesn't just produce bad output. It produces coordinated, confident, internally-consistent bad output. That's worse than no coordination at all.