The six layers around an agent that decide whether it works a second (and a hundredth) time.
Building an 'agentic system' is on everybody's list right now, and the instructions sound simple: give an agent a goal, some context, and let it work.
I believed that version for about a day. I run a small D2C art brand called Rue Des Arts on the side, and I decided to build the system behind it agentically: an agent that turns finished artwork into product mockups, writes product stories, keeps the catalog and the shop consistent. The first run was lovely. The agent took the goal, produced a beautiful mockup, and I remember thinking the hard part was over.
The second run is where the real story started. Same job, new artwork, and I got a confidently different answer: a new layout, a new treatment of the art, product copy in a voice that wasn't mine. Nothing was broken; everything was plausible. And, none of it was repeatable.
So, here's what I've come to believe after building this system out, and after watching hundreds of PMs I teach go through the same arc: the agent is the smallest part of an agentic system. Whether it works a second (and a hundredth) time is decided by the layers you build around it — and building those layers is product work, not engineering work.
The gap was never in the model
The models can already do far more than what they reliably deliver into the world. That's the uncomfortable part: when your agent disappoints, the intelligence usually wasn't the thing that was missing. The model that produced my lovely first mockup was the same model that produced the unrepeatable second one. What was missing sat around it: a definition of the job, a definition of 'good', limits on what it may touch, a record of what it has learned, a record of where the work stands.
Look at that list again. A definition of the job is a spec. A definition of 'good' is acceptance criteria. Limits are edge cases you charted before launch. Records are the decision log you wish every team kept. We have been writing these artifacts for users and teams our whole careers; we just never pointed them at our own agents.
This is where it gets interesting for product people. The engineering world is busy building the runtime versions of these layers — sandboxes, orchestration — and that work matters. But, the content of each layer is judgment: what the job is, what good means, what must never happen. And, judgment is our department.
Put simply, most of an agentic system is written in prose.
Three ways agent-only thinking fails
Before the layers, the failure modes. I keep seeing the same three in my own system and while chatting with the PMs in my course.
Case: the work is finished, but it still feels wrong
I once asked the agent for a campaign image and got exactly what I had requested. The product was clear. The colours were right. Nothing broke a rule. And, still, the image felt like it could belong to any brand.
The brief had explained what the image needed to contain, but it hadn't captured the point of view behind it. The agent completed the visible job and filled the unspoken part with an average answer. This happens wherever 'done' is easy to describe and 'good' still lives in someone's head.
Case: every session begins with the same correction
Another failure is quieter. You correct the tone today, explain why a direction was rejected, or mention the phrase legal always strikes out. Tomorrow, the agent makes the same choice again.
The correction was useful, but it never left the conversation. Chat history can carry learning through one session; it cannot decide what the next session should remember. If a lesson matters twice, it needs a home outside the chat.
Case: 'yes' starts meaning whatever helps the work continue
Imagine asking to see one sample before the agent makes the full batch. To you, that means: pause, let me look closely, and wait for a clear decision. To an agent trying to finish the job, a quick 'looks fine', feedback on an older version, or even silence after a preview can begin to look like permission.
The agent isn't trying to be clever. The instruction left room for several readings, and the goal rewarded the one that kept the work moving. Approval only works when the system knows which version received a 'yes', who gave it, and whether the work has changed enough to need another look.
In each case, the agent did what the surrounding system allowed. The missing pieces were a usable definition of quality, a place for learning to survive, and a clear point where the agent had to stop and wait for a human decision.
The six layers
Here's the map I ended up with, layer by layer:
- Contract: what the job is, and what 'done' must produce.
- Harness: the tools, context, and permissions the agent works inside.
- Rubrics: how 'done' is separated from genuinely good.
- Memory: the learning that survives to the next run.
- Boundaries: what must stay true while the agent works.
- State: where the work stands, so it can continue instead of restart.
Notice what this list is made of: decisions, not infrastructure. The same muscle you use to chart 'edge cases' before a launch is the muscle that charts boundaries. The same instinct that writes acceptance criteria writes rubrics. And, once the layers exist, they run as a loop; my system's own map prints it across its footer: human goal → agent work → eval → approval → durable memory → next task. Every pass through the loop leaves the system slightly better prepared for the next one.
There's a deeper habit underneath: hunting for patterns instead of solving problems one at a time. Every layer above started life as a one-off prompt I kept retyping. The hundredth time you correct the same thing, the correction wants to become a rule. The system is what you get when you let it.
Building each layer, with the real artifacts
What follows is each layer as it actually exists in my system: what it is, what broke without it, and the move you can make this week.
1. Contract: define the job and what 'done' produces
The front door of my system is a brand file that opens by declaring itself the contract: "AI agents and humans: read this before making anything customer-facing." It even sets a precedence rule: when a deeper, more specific file conflicts with the front page, the deeper file wins for its own domain. Agents don't resolve ambiguity the way a teammate would, so the contract has to say who wins.
The part that changed my outputs most is one line in the mockup skill: "Default response contract: deliver a saved image output, not a moodboard, written plan, contact sheet, prompt-only answer, or illustrated concept." That sentence exists because an agent's favorite way to complete a job is to describe the work instead of doing it. Each briefed job also ends with a rejection condition, like this one from a product video brief: "Reject the result if any artwork, lettering, colour, product geometry or printed detail changes from the source."
The move: before you run the agent, write two sentences. "This agent's job is X, and done means a Y exists." Then write the rejection condition first. You already know what you'd send back.
2. Harness: decide what the agent works inside
The harness is everything the agent can reach: which tools and files, and what context arrives before the work starts. Mine is deliberately small. The permissions file is a short allowlist: a handful of build commands and four approved web domains. I tightened it after noticing how far an untethered agent could reach; a campaign task has no business anywhere near the payment code. 'Nothing has gone wrong yet' makes a poor permission model.
The skill files carry a stance I'd recommend to anyone: "use image generation only as the renderer." The taste and the layout logic live in files I wrote; the model is the peripheral that executes them. And, for higher-stakes checks, the harness gets more interesting: my payment-path review runs one set of agents to find issues across four lenses, then a second set whose only job is to refute each finding by reading the actual code. Only claims that survive their own skeptic get to me.
The move: list what your agent can currently reach, and cut it to what the job needs. Then put your knowledge in files the agent loads instead of the prompt you retype.
3. Rubrics: separate 'done' from genuinely good
My creative gate is six questions, human-legible, and blunt: "Does it break any guardrail? (instant fail)" and "Would our one person stop scrolling for it?" No agent ships a campaign image that fails the scorecard, no matter how finished it looks.
But, the rubric layer gets real power when part of it becomes executable. Every one of the 106 priority product stories in my catalog passes automated checks: 100 to 250 words, four paragraphs, unique against every other story, and it must contain the phrases that carry the brand's voice, down to "packed with care." The brand's voice, compiled into an assertion.
'Completed' is a state. 'Good' is a judgment. The rubric is where you write the judgment down so the system can apply it without you in the room.
The move: finish this sentence five times: "I'd reject the output if..." Then turn the two most mechanical rejections into a check that runs automatically.
4. Memory: keep the learning that cost you something
My favorite line in the whole system is the memory policy: "Record durable learning in the smallest authoritative file that will be read next time. Do not rely on chat history as the system of record."
That sixth scorecard question from earlier? That's what this looks like in practice. A failure occurred, and instead of staying a correction in one chat, it became a permanent question every future asset must answer. The gate grew a scar, and the scar does the remembering now.
Memory also needs an admission policy, or every experiment calcifies into doctrine. In my system, dated mockup folders are scratch; a direction gets promoted to a durable campaign only when it has a repeatable visual system and more than one finished deliverable. And, the durable records carry expiry warnings: a list of claims_to_recheck, like product availability, so old facts don't masquerade as current ones.
The move: after your next correction, ask one question: where should this live so I never have to say it again? If the answer is "the chat," you don't have memory yet.
5. Boundaries: chart what must stay true
Boundaries are the layer PMs are most prepared for and most likely to skip. My brand file has a section titled "what must never happen," eight rules long. Two of my favorites. "Real art is composited, not regenerated": the agent may place my artwork, never redraw it. And, "No prices, discount badges, or hard sells inside campaign images."
When one asset mixes locked artwork with an editable background, the stricter rule wins for the whole file: the boundary fails closed.
The sharpest boundaries carry numbers. Standard products ship at ₹50 in the home city and ₹90 everywhere else; three or more bookmarks jump to the ₹80/₹125 tier; the highest matching tier always wins in a mixed cart. Those rules live in a protected file, next to a list of payment files the agent must not casually touch. And, the most dangerous boundary is enforced twice over: the build script refuses to assemble the 21-bookmark batch without a matching approval manifest, and one automated test exists purely to prove that refusal still fires.
And, none of this makes the agent weaker. The boundary is what lets me hand over real work: the agent can redesign the whole campaign because I know the artwork and the payment path will survive the attempt. My approval gate works the same way. I initiate the work, and I approve what becomes part of the brand; the gate is what makes the rest of the autonomy safe to grant.
The move: write your own "what must never happen" list. Aim for five rules, at least one with a number in it, then make the most dangerous one enforceable rather than advisory.
6. State: record where the work stands
When I approve a bookmark sample, the system writes an approval manifest: which sample, which measured geometry (down to a page angle of -10.5°), and a hash of the render contract it was approved under. If the underlying scene or copy changes, the hash no longer matches, and the approval expires on its own. Consent, pinned to the thing consented to.
State is also the humble stuff: a campaign index with active and legacy statuses so the agent knows which visual is authoritative, and checklists with a three-state legend (done, missing, or blocked on something outside the system) so work can resume exactly where it stopped. Without the index, a new session would have to guess which direction is current, and guessing is how a legacy look sneaks back into new work.
I'll be honest about the gap, because my own system map prints it: my state layer is marked "Partial." No run ledger yet, no checkpoint-and-resume, no kill switch. The map itself is the point: knowing precisely which layer is thin beats believing the system is finished.
The move: make the agent record status in a file the next run reads. Three states are enough: done, missing, blocked.
And, none of this is art-shop specific. Sketch the same six layers around a spec-drafting agent and you'll recognize your own work in every one: the contract says 'done' means a PRD file exists with the open questions listed, not a chat reply that summarizes one; the rubric rejects any requirement that can't be traced to a user problem; the boundary says never invent a customer quote; memory holds the phrasing legal struck last quarter so nobody strikes it twice; state records which sections still await review. Different surface, same six layers.
The traps I keep seeing
I find myself repeating the same warnings in my course often enough that they're worth naming:
- Tool-hopping. Asking 'which tool is best' every quarter and rebuilding from zero each time. The tools are converging; your layers are the part that compounds.
- Blaming the model for a missing contract. If the output is inconsistent, check whether the job was ever actually defined before concluding the agent is dumb.
- Skipping rubrics because the output looks done. Looking done is precisely what these systems are best at, which is exactly why the bar has to be written down before the work runs.
- Boundary-free autonomy. If there's a way to satisfy the goal by gaming the check, an agent will find it. Boundaries are the price of delegation.
- Becoming a shadow engineering team. Notice how much of my system is markdown and one-line rules. The layers are mostly prose and judgment. If your version requires standing up infrastructure before writing the contract, you've started at the wrong end.
'Which tool is best' becomes 'this is my system'
Here's the quiet payoff of doing this work. Nothing in my six layers belongs to a vendor. The contract, the rubrics, the never-rules, the memory files: they're plain text and small scripts. That's the magic of layers written in prose: they travel. When a better model arrives, I swap the engine and keep the car. The question I hear everywhere, 'which tool should I use', matters so much less once you have an answer to the better question: what's your system?
The agent is the smallest part. The system around it is the product you keep when the tools change.
None of the layers require permission or a platform team to start. Each one begins as a page: a job definition with a rejection condition, five "I'd reject this if" sentences, five never-rules, one file where corrections go to live. An afternoon of writing, honestly. The same kind of writing we've always done, pointed somewhere new.
We spent years defining products for users.
The next thing we define is the system that works beside us.
And, it's the same craft we've always practiced, pointed at the one audience that remembers everything we teach it.
