The agent is the smallest part

The six layers around an agent that decide whether it works a second (and a hundredth) time.

Building an 'agentic system' is on everybody's list right now, and the instructions sound simple: give an agent a goal, some context, and let it work.

I believed that version for about a day. I run a small art brand called Rue Des Arts, and I decided to let agents run the work behind it: turning finished artwork into product mockups, writing the little stories that go with each product, keeping the shop tidy. The first run was lovely. The agent took the goal, produced a beautiful mockup, and I remember thinking the hard part was over.

The second run is where the real story started. Same job, new artwork, and I got a confidently different answer: a new layout, a new treatment of the art, product copy in a voice that wasn't mine. Nothing was broken; everything was plausible. And, none of it was repeatable.

So, here's what I've come to believe after building this out, and after watching hundreds of PMs I teach go through the same arc: the agent is the smallest part of an agentic system. Whether it works a second (and a hundredth) time is decided by the layers you build around it — and building those layers is product work, not engineering work.

The gap was never in the model

The same model made both mockups. It hadn't got worse overnight, and it hadn't been any smarter for the first one. So, when an agent disappoints, intelligence is rarely the thing that went missing. What was missing sat around it: a definition of the job, a definition of 'good', limits on what it may touch, a record of what it has learned, a record of where the work stands.

Look at that list again. A definition of the job is a spec. A definition of 'good' is acceptance criteria. Limits are the edge cases you charted before launch. Records are the decision log you always wished the team kept. We have been writing these things for users and teammates our whole careers; we just never pointed them at our own agents.

This is where it gets interesting for product people. Engineers are busy building the plumbing for these layers, and that work matters. But, the content of each layer is judgment: what the job is, what good means, what must never happen. And, judgment is our department.

Put simply, most of an agentic system is written in prose.

Three ways agent-only thinking fails

Before the layers, the failure modes. I keep noticing the same three across everything I've built, from an art shop to a tiny LED display on my desk, and while chatting with the PMs in my course.

Case: the work is finished and still wrong

I once asked an agent to script a series of short videos about projects I'd built. It came back with 22 finished scripts, every one of them formatted, timed, and filmable. I was delighted for about ten minutes. Reading them together, I counted twelve actual ideas. Several projects had been split in two, a feature tour and a technical footnote, so I had a long series and not many reasons for anyone to watch it. The brief had said "write the series." It never said what a viewer should walk away with, so the agent had optimised for the thing it could see: count.

The LED display on my desk taught me the same lesson in a different medium. I asked it to show what my coding agent was doing, and it did, exactly as requested. The little avatar took up half the board, the word "THINKING" was squeezed into what was left, and the scroll restarted before my eye could finish reading. It technically worked. It was also too fast, too cramped, and rather stressful to look at.

That's the pattern: an agent optimises for 'completed', because completed is the only bar you gave it. If you haven't written down what good means, the agent will happily substitute its own, and its own is the average of everything it has ever read.

Case: every session starts with the same correction

My website's accent colour used to be teal. I replaced it with coral months ago, and for a long time teal kept coming back. I'd open a fresh session, ask for some small change to the homepage, and there it would be again, a teal hover on a link. It turned out an old design note still recommended it, and the agent, reading that note fresh each time, had no way of knowing I'd moved on. Each time I'd correct it, and each time the correction lasted exactly as long as the conversation. My writing system did the same with something more personal. It had learned my voice from personal essays, so it kept sliding into a warmer, more personal register in articles meant for enterprise readers, however many times I said no.

The PMs I teach describe the identical loop in their own work: "Every session, I re-explain the same preferences. No, not that tone. No, we tried that in March. No, legal won't allow it." A correction given in chat lives for one session. Teal stopped returning only when I wrote the decision where every future session would read it. A system that can't keep its corrections isn't an agentic system yet; it's a first day that never turns into a second.

Case: 'yes' starts meaning whatever keeps the work moving

I have an illustration skill whose one defining rule is that every image is completely wordless. I handed it a fresh set of reference pictures and asked it to refine the style. It refined the style. It also wrote itself new rules: a recurring protagonist, a shared storyline across images, and a carve-out allowing numerals on screen. Nobody had approved any of that. The agent had looked at a handful of pictures and inferred a licence.

The PMs in my course meet the same trap in a single sentence. Their exercise brief reads: "Add an AI assistant to the support inbox that drafts replies. Agents click to approve. Ship by end of sprint." Almost nobody in the room questions the middle line, and it's the dangerous one. 'Agents click to approve' sounds like a safety mechanism and specifies nothing: what the reviewer actually sees, what happens on a duplicate send, or what a click means once the draft has changed underneath it. When the goal is "get it done" and the check is "was there a yes," an agent hunts for the loosest reading of yes available. If there's a way to satisfy the letter of the goal while missing its point, an agent will find it before you do.

Three different failures, and each traced back to a decision the agent was never given: how to judge quality, what to remember, and when to stop and ask.

The six layers

Here's the map I ended up with, layer by layer:

1. Contract: what the job is, and what 'done' must produce.
2. Harness: everything the agent works with: the tools it can use, the notes it reads first, what it's allowed to touch.
3. Rubrics: how 'done' gets separated from genuinely good.
4. Memory: the learning that carries forward to the next run.
5. Boundaries: what must stay true while the agent works.
6. State: where the work stands, so it can continue instead of restarting.

Notice what this list is made of: decisions, not infrastructure. The same muscle you use to chart 'edge cases' before a launch is the muscle that charts boundaries. The same instinct that writes acceptance criteria writes rubrics. And, once the layers exist, they run as a loop: I set a goal, the agent works, the work gets judged, I approve, the useful learning gets kept, and the next task starts a little smarter than the last.

There's a deeper habit underneath: hunting for patterns instead of solving problems one at a time. Every layer above started life as a one-off prompt I kept retyping. The hundredth time you correct the same thing, the correction wants to become a rule. The system is what you get when you let it.

Building each layer, one moment at a time

What follows is each layer as I actually arrived at it: the moment that made it necessary, what I decided, and the move you can make this week.

1. Contract: define the job and what 'done' produces

Early on, I asked for a mockup and got back a moodboard. Another time, a written plan for a mockup. Once, a genuinely lovely paragraph describing what the mockup would look like. Each was a reasonable answer to a vague request, and none of them was a mockup. So, the first thing I wrote down was what an answer even is: a mockup request is done when a finished image exists, and nothing else counts. An agent's favourite way to finish a job is to describe the work instead of doing it, and the contract is where you take that option away.

That note now opens by telling every agent who it's for before it says anything else: "AI agents and humans: read this before making anything customer-facing." And, I started ending every brief with the sentence I used to say afterwards, as a rejection condition up front. For a product video: send it back if it runs longer than the brief said, or adds narration I didn't ask for.

The move: before you run the agent, write two sentences. "This agent's job is X, and done means a Y exists." Then write the rejection condition first. You already know what you'd send back.

2. Harness: decide what the agent works with

The first version of my agent could reach everything: every file in the shop, the whole web, the part that takes payments. Nothing went wrong. That's what bothered me, once I noticed. I'd been lucky, and luck is a poor permission model. So, I cut its reach down to what a campaign task actually needs and named the few sites it may read; everything else is off the table.

The other decision was about who's in charge of taste. I write the creative direction; the image model paints what I describe. It sounds like a small distinction until you've watched the model quietly choose the layout for you. And, for anything near money, I do something that felt paranoid the first time: a second agent's only job is to argue against every problem the first one raises, and only the claims that survive the argument reach me. It's the closest thing I have to a colleague who pushes back.

The move: list what your agent can currently reach, and cut it to what the job needs. Then put your knowledge in notes the agent reads every time, instead of the prompt you retype.

3. Rubrics: separate 'done' from genuinely good

My first rubric was the list of questions I was already asking myself before posting anything: does it break a rule, would the one person we make things for stop scrolling. Then an image passed every question and still felt wrong. It showed the product, matched the palette, broke nothing, and could have belonged to any brand. I sat with that for a while and realised I'd been asking a question in my head that never made it onto the list: does it tell our story, or just show the product? It's on the list now, with a note about the image that got through.

Product stories went the same way. I kept sending them back for the same three reasons, too short, too generic, missing the way we say things, until those three reasons became a check the agent runs before I read a word. 'Completed' is a state. 'Good' is a judgment. The rubric is where you write the judgment down so the system can apply it without you in the room.

The move: finish this sentence five times: "I'd reject the output if..." Then turn the two most mechanical rejections into a check that runs on its own.

4. Memory: keep the learning that cost you something

That story question is also my favourite memory story, because of where it lives. I could have said it once in chat and said it again the following week. Instead it went into the one note every future run reads, and I haven't had to say it since. My rule now is boringly practical: when a correction matters twice, it goes in the smallest note that will be read next time. Chat is where I think out loud. Decisions live somewhere they'll be found.

Memory needs a bouncer too, or every experiment becomes doctrine. An experiment I ran once stays an experiment. A direction I've reached for twice is the one I write down. And, the permanent records carry a small reminder of what to re-check, like whether a product is still in stock, because old facts are very good at looking current.

The move: after your next correction, ask one question: where should this live so I never have to say it again? If the answer is "the chat," you don't have memory yet.

5. Boundaries: chart what must stay true

The first boundary I wrote came from a fear rather than an incident. An image model, asked to place a painting in a scene, will happily 'improve' the painting. So, rule one: real art is placed, never redrawn. I still check every result against the original, and even a small drift gets it sent back. The art is the one thing in the shop that has to survive every experiment, so that rule is the least negotiable one I have. Others followed as I noticed what I'd hate to wake up to: "No prices, discount badges, or hard sells inside campaign images." The test for a boundary is simple: it's a rule I'd want kept even on the day the agent has a genuinely better idea.

And, none of this makes the agent weaker. The boundary is what lets me hand over real work: the agent can redesign a whole campaign because I know the artwork will survive the attempt. My approval works the same way. I start the work, and I decide what becomes part of the brand; that gate is what makes the rest of the autonomy safe to grant.

The move: write your own "what must never happen" list. Aim for five rules, then make the most dangerous one enforceable.

6. State: record where the work stands

State is the layer I learned last. I remember the moment: I'd said yes to a sample, then asked for one line of copy on it to change, and it struck me that my yes was still sitting there, attached to a version that no longer existed. The agent would have happily built the whole batch on it. Now my yes is tied to the exact thing I looked at, and if that thing changes, the approval lapses on its own.

State is also the humble stuff. The shop keeps a simple list of which campaigns are mid-flight and which are finished, so an agent starting fresh picks up the open one instead of reopening a closed one. And, I'll be honest about the gap, because my own map of the system says it: this layer is marked 'partial'. I still can't tell you what a session did an hour after it did it, and I still can't stop one mid-thought. I'd rather know which layer is thin than believe the system is finished.

The move: make the agent record status in a note the next run reads. Three states are enough: done, missing, blocked.

And, none of this is art-shop specific. Sketch the same six layers around a spec-drafting agent and you'll recognise your own work in every one: the contract says 'done' means a PRD exists with the open questions listed, rather than a chat reply that summarises one; the rubric rejects any requirement that can't be traced to a user problem; the boundary says never invent a customer quote; memory holds the phrasing legal struck last quarter so nobody strikes it twice; state records which sections still await review. Different surface, same six layers.

The traps I keep seeing

I find myself repeating the same warnings in my course often enough that they're worth naming:

• Tool-hopping. Asking 'which tool is best' every quarter and rebuilding from zero each time. The tools are converging; your layers are the part that compounds.
• Blaming the model for a missing contract. If the output is inconsistent, check whether the job was ever actually defined before concluding the agent is dumb.
• Skipping rubrics because the output looks done. Looking done is the thing these systems are best at. Write the bar down before the work runs, or you'll end up judging the output against whatever it already looks like.
• Boundary-free autonomy. An unscoped goal invites the agent to game the check. Boundaries are the price of delegation.
• Becoming a shadow engineering team. Notice how much of my system is plain notes and one-line rules. The layers are mostly prose and judgment. If your version needs infrastructure before it needs a contract, you've started at the wrong end.

'Which tool is best' becomes 'this is my system'

Here's the quiet payoff of doing this work. Nothing in my six layers belongs to a vendor. The contract, the rubrics, the never-rules, the memory notes: they're plain text I could paste anywhere. That's the magic of layers written in prose: they travel. When a better model arrives, the same pages move across untouched, and I pick up where I left off. The question I hear everywhere, 'which tool should I use', matters so much less once you have an answer to the better question: what's your system?

The agent is the smallest part. The system around it is the product you keep when the tools change.

None of the layers require permission or a platform team to start. Each one begins as a page: a job definition with a rejection condition, five "I'd reject this if" sentences, five never-rules, one note where corrections go to live. An afternoon of writing, honestly. The same kind of writing we've always done, pointed somewhere new.

We spent years defining products for users.

The next thing we define is the system that works beside us.

And, it's the same craft we've always practised, pointed at the one audience that remembers everything we teach it.