How to design agent systems that do not just remember more, but bring in the right evidence and discard the wrong context at the right time.
Full content
## The problem
Most people treat context like a sacred pile: once information is retrieved or remembered, they keep feeding more and more of it back into the model. That creates noise, redundancy, distraction, and context rot.
The better question is not: *How do I store more?*
It is: *How do I bring in only what still deserves to be active right now?*
## The key shift
A strong agent system does not just remember. It also knows what **not** to bring into the active context window.
## Why retrieval should be a separate role
In many systems, one agent tries to:
- search
- retrieve
- filter
- reason
- answer
That often creates overload. A healthier architecture is:
**retrieval/search subagent → evidence set → reasoning/generation agent**
This separates:
- finding evidence
- evaluating relevance
- generating the response
## What a retrieval subagent should do
A retrieval subagent should:
- decompose the search problem
- gather candidate evidence
- remove duplicates
- discard stale or weakly relevant items
- preserve only the most useful evidence for the next stage
It should usually **not** be the final answering agent.
## Context pruning rules
Good pruning heuristics include:
- drop duplicates
- drop stale evidence unless historically important
- keep items that satisfy the current goal, not just the broad topic
- separate background context from active evidence
- prefer source-backed material over inferred summaries
- keep a record of what was pruned, even if it is not passed forward
## Three memory tiers
### Tier A — Active context
What the model needs right now.
### Tier B — Retrievable working memory
Material that can be brought in if needed.
### Tier C — Archival memory
Stored history that should not enter active context without a strong reason.
The mistake is to treat all three as if they belong in the prompt at once.
## What not to confuse
### Pruning is not forgetting
Removing something from active context is not the same as deleting it forever.
### Summarization is not always enough
Summaries can help, but they can also hide important evidence. Sometimes the system needs selective retention, not compression.
## Implications for personal agent systems
If you are building agents for yourself, friends, or family:
- do not keep pushing entire histories into the prompt
- retrieve based on current relevance
- keep active context small and high-signal
- separate personal memory from current evidence
## Implications for CoachOS
CoachOS especially should avoid raw memory dumps.
It needs a retrieval layer that can tell the difference between:
- what matters to this coaching moment
- what is useful background
- what is just archived history
Otherwise, continuity becomes noise instead of insight.
## Practical recommendation
When designing an agent system, add a retrieval layer or subagent when:
- the task depends on lots of history
- the question is multi-hop
- there are many possible sources
- wrong context can distract the model
A good system is not the one with the biggest memory pile.
It is the one that can surface the right evidence cleanly.