Memories with Agentic AI are very difficult.
If you’ve spent any significant time building complex systems with AI assistants, you know the frustration of the “Context Reset.” You start a fresh session, and suddenly your highly capable agent has forgotten the architectural pivot you made yesterday, the naming convention you spent an hour refining, and the specific AWS constraints that almost blew up your staging environment.
This isn’t just a minor annoyance; it’s a structural failure I call the Session-to-Session Wall.
The Session-to-Session Wall#
Right now, the industry standard for “memory” in coding assistants like Codex or tools like Claude Code is largely reactive and dynamic. They rely on “context windows” and RAG (Retrieval-Augmented Generation) to pull in snippets of code they think might be relevant. While impressive, these tools lack a deliberate, structured semantic ledger. They have dynamic memory, but they don’t have wisdom.
The “Session-to-Session Wall” is the invisible barrier where your agent’s productivity dies. You spend two hours yesterday explaining the intricate VPC peering requirements. You wake up today, start a new session, and your agent is back to suggesting standard library defaults that will break your build. You aren’t just losing time; you’re paying a Semantic Repetition Tax—the cognitive and literal token cost of re-litigating decisions that should have been settled.
For a new project I’m currently calling Project Nidus, I’ve realized that semantic memory logging is the only way forward. We need a way to capture the evolution of decisions in a format that AI can audit before it writes a single line of code.
The Evolution of ADRs: Technical Rationale#
This starts with Architecture Decision Records (ADRs). In Project Nidus, these aren’t just for humans. They are structured records designed for agents to read first. When a new agent joins the project, it audits the ADR directory to understand the “immutable” laws of the system.
One of our most critical ADRs involves Zero-Knowledge Envelope Encryption. Why is this superior to standard database encryption? Standard “Encryption at Rest” is often just a compliance checkbox; it protects against someone stealing a physical hard drive. It does nothing if your database credentials are leaked or if an agent decides to “helpfully” log a raw user object.
Envelope encryption wraps data in user-specific keys that are themselves encrypted by a master key. The application—and by extension, the agent—never handles the raw key without explicit, logged authorization. It moves the trust boundary from the storage layer to the identity layer.
# ADR: 20260327-Zero-Knowledge-Envelope-Encryption
## Status
Accepted
## Context
Project Nidus requires that even developers must
not have access to raw user data. Standard RDS
encryption-at-rest is insufficient as it is
transparent to anyone with DB access.
## Decision
We will implement **Envelope Encryption** for
all PII and sensitive user data.
- **Master Key (MK)**: A single AWS KMS CMK
managed by infrastructure-as-code.
- **Data Key (DK)**: A unique, cryptographically
strong key generated per-record.
- **Process**: Data is encrypted locally in
the application layer using the DK. The
DK is then encrypted by the MK and stored
alongside the data blob.
## Consequences
- **Security**: Data is unreadable even if
the database is dumped or credentials
leaked. Agents cannot "browse" PII.
- **Complexity**: Requires careful KMS IAM
policy management to prevent agents from
requesting Decrypt permissions.Solving the Semantic Repetition Tax#
The breakthrough for us was moving toward a Core and Topical memory structure.
Core Memories are the foundational mandates that load every time I start a session (via our Steering files). These are the “always true” rules.
Topical Memories are semantically linked learning repositories. Instead of bloating the AI’s context window with every lesson ever learned, we use a discoverable index. The agent only pulls in the specific domain knowledge it needs for the task at hand. This architecture eliminates the Semantic Repetition Tax by ensuring the agent “remembers” the expensive lessons without being told twice.
graph TD
subgraph Session_Start [Session Initialization]
Core[Core Memories / Steering Files]
end
subgraph Discovery_Layer [Semantic Discovery]
Index{Memory Index}
end
subgraph Topical_Knowledge [Topical Learning Repositories]
AWS[AWS & Security]
SDK[SDK & Tooling]
Workflow[Workflow & Patterns]
end
Core -->|Always Loaded| Index
Index -->|Discover & Pull| AWS
Index -->|Discover & Pull| SDK
Index -->|Discover & Pull| Workflow
AWS -.->|Feedback Loop| Core
SDK -.->|Feedback Loop| Core# Project Memory (Core)
## Core Mandates
- Always run 4-agent code review (never skip).
- Use implement-and-review-loop as default.
- Zero-Knowledge Mandate: No plaintext PII
in storage. Use Encryption Service.
## Topical Memories (Discoverable Index)
- **[AWS & Security](topics/aws-security.md)**:
Fargate, IAM, KMS constraints.
- **[SDK & Tooling](topics/sdk-tooling.md)**:
React Native, Jest, CDK.Shaping the Milestone: The Principal Wrestling Match#
Finally, we’ve introduced the milestone-shaping skill. I call
it the Principal Wrestling Match. This is how we
proactively refine a specific product milestone by forcing a
conversation between technical integrity and user value before
any code is written.
The skill dispatches a Principal PSE (Software Engineer) and a Principal PM (Product Manager) to audit the plan. The PSE’s mandate is technical perfection and security-first integrity. The PM’s mandate is user friction and speed-to-market.
In Project Nidus, we force these two agentic personas to fight it out. The PM wants a “One-Click” login. The PSE wants a “Three-Factor” air-gapped authentication. The result of this friction isn’t a weak compromise; it’s an innovation—like a Passkey-first implementation that satisfies both mandates. We don’t avoid the conflict; we use it to harden the architecture.
# Milestone Shaping
Proactively "shape" a product milestone by reconciling
technical integrity (PSE) with user value (PM).
## Workflow
1. **Milestone Definition**: Define the specific goal
(e.g., "10-user Beta Preview").
2. **Specialist Investigation**:
- **PSE Security Check**: Use `principal-pse` to
audit for Zero-Knowledge integrity and "Transient
Windows" where plaintext might leak.
- **PM Utility Check**: Use `principal-pm` to
audit for friction, onboarding complexity, and
value-to-effort ratios.
3. **Conflict Resolution**: Identify where security
mandates (e.g., "Mnemonic required") clash with
utility goals (e.g., "One-touch login"). The
agents must propose a "Shaped Solution."
4. **Artifact Generation**:
- Create/Update ADRs in `docs/adr/`.
- Update requirements and `backlog.md` structure.
5. **Backlog Promotion**: Prune the backlog and
promote high-signal blockers to P0.
## Specialist Prompts
When shaping, we dispatch these specific instructions:
- **Principal PSE**: "Audit the architecture for [Milestone].
Ensure Zero-Knowledge integrity is maintained. Identify
any 'Transient Windows' where plaintext exists."
- **Principal PM**: "Audit the backlog for [Milestone].
Identify the biggest friction points in onboarding.
Verify that 'Agentic Sensors' provide high-signal
value to users, not just generic noise."Drinking Our Own Champagne#
The most meta part of writing this post? I realized mid-draft that my own blog’s memory.md had become a victim of the very “Session-to-Session Wall” I was describing. It was a 15,000-character monolithic file where Hugo patterns were mixed with AWS CDK deployment notes and stakeholder preferences.
So, we stopped. We refactored the blog’s memory in real-time to match the Project Nidus standard.
We stripped the core memory.md down to just the foundational mandates and moved the technical “wisdom” into specialized topical files. Here is the actual index we created for this blog during the session:
# Project Memory (Core)
## Core Mandates
- Refer to user as Keith Hodo.
- Branch names must include the GitHub issue ID.
- NEVER use em-dashes (—) or double hyphens (--) in content.
## Topical Memories (Discoverable Index)
- **[Hugo & Blowfish](topics/hugo-blowfish.md)**: Theme,
layouts, and page bundles.
- **[CDK & AWS Patterns](topics/aws-cdk.md)**: CloudFront,
S3, ACM, and CI/CD roles.
- **[SEO & Site Polish](topics/seo-polish.md)**: Favicons,
OG, llms.txt, and schema.org.
- **[Content & Workflow](topics/content-workflow.md)**:
Writing style, LinkedIn voice, and Kiro patterns.Now, when I’m working on a new post like this, the agent doesn’t need to load the details of ACM certificate dependencies in us-east-1 unless I specifically ask it to audit the infrastructure. It only loads the Content & Workflow topical memory, keeping the context clean, fast, and high-signal.
We’ve even started backfilling ADRs for the blog’s own infrastructure. Here is a simple but effective ADR that guides how our agents think about the blog’s deployment:
# ADR: 20260305-Static-Hosting-Architecture
## Status
Accepted
## Context
We need a cost-effective, low-maintenance hosting solution
for the migrated blog that supports custom domains and
SSL. The original Azure App Service was costing ~$76/mo.
## Decision
We will use **AWS S3 + CloudFront** for static hosting,
managed entirely via **AWS CDK**.
- **Storage**: S3 bucket with public access blocked,
utilizing Origin Access Control (OAC).
- **CDN**: CloudFront distribution with a managed
cache policy (CachingOptimized).
- **Security**: ACM Certificate in us-east-1 for
Global CloudFront support.
- **Automation**: GitHub Actions via OIDC roles
for zero-secret deployments.
## Consequences
- **Pros**: Reduced monthly cost from $76 to ~$1.
Infinite scalability without server management.
- **Cons**: Requires a CloudFront invalidation for
content updates (automated in CI/CD).By the time I start coding, the path is already shaped. The AI isn’t just following a list of tasks; it’s executing a strategy that it helped refine, grounded in a memory that it won’t lose when I close my terminal.
We aren’t ready to pull the curtain back on Project Nidus just yet, but the way it’s being built—with these precise, agentic workflows—is making the process more enjoyable than anything I’ve worked on in years.
Stay structured,
The Brougham 22