The Agent Harness
A model call is four lines of HTTP. Everything that makes it into a product — the loop, the tools, the context budget, the retries, the traces, the evals — is the harness, and the harness is where all the engineering lives. Built up from first principles, then spent on one concrete thing: a design tool that makes posters using models from OpenRouter.
Five words people use wrong
Most confusion about agents comes from six or seven words that get used for four or five different things. Pin them down and the architecture becomes almost boring — which is the goal.
- Model
- A function from a list of messages to a next message. Stateless, pure-ish, non-deterministic. It cannot do anything. It cannot remember anything. It emits text and, sometimes, a structured request that it would like something done.
- Tool
- A JSON Schema you show the model, plus a function you run when the model asks for it. The model never executes code. It writes down a name and some arguments; your code decides whether to honour that.
- Agent
- A loop: call the model, run any tools it asked for, append the results, call the model again, until it stops asking. That is the entire idea. An agent is not a model — it is a control-flow pattern wrapped around one.
- Harness
- Everything around the loop that makes it survivable in production: budgets, retries, timeouts, cancellation, permissioning, persistence, streaming, context assembly, tracing, and the stop conditions. This is your actual software.
- Context
- The exact bytes you send to the model on this call. Not “what the agent knows” — there is no such thing. Every turn you rebuild the whole input from scratch, and choosing what goes in it is the highest-leverage engineering in the system.
- MCP
- A wire protocol for discovering and calling tools that live outside your application. It is not how agents work; it is one way of plugging third-party capability into one. You can build a complete agent without ever touching it.
- Context window
- The hard token limit on that input. A budget you allocate, not a container you fill.
An agent is a while-loop over a stateless function, and the hard part is deciding what to put in the argument. Everything in this document is either a way of running that loop safely or a way of deciding what goes in the argument.
The atom: one model call
Before agents, tools or MCP, get completely clear on the primitive, because every later problem is a consequence of its two properties: it is stateless, and you pay for the whole input every single time.
The shape
A chat model call is a list of messages in, one message out. Four roles do all the work:
| Role | Written by | Purpose | Notes |
|---|---|---|---|
system | You | Standing instructions, persona, rules, output format | Goes first, stays byte-identical across turns — that is what makes prompt caching work |
user | The human, or your harness | The request, and any injected context | Harness-injected content lives here too; the model can’t tell the difference, which is both useful and a security problem |
assistant | The model | Its reply: text, and/or tool_calls[] | You append its own outputs back to keep continuity |
tool | You | The result of a tool the model asked for | Must carry the matching tool_call_id, or the call is malformed |
There is no conversation. The illusion of one is you re-sending the entire history every turn. The model has no memory between calls, no variables, no open files. If something is not in this request, it does not exist.
Cost grows quadratically. A 20-turn agent run does not cost 20 model calls — it costs the sum of 20 progressively larger inputs. A run whose context reaches 100k tokens has re-read most of those tokens dozens of times. This is why §04 exists, and why prompt caching is not a micro-optimisation.
Calling it through OpenRouter
OpenRouter is a single OpenAI-compatible endpoint in front of most commercial and open models. That matters for a design tool specifically: planning, patching, critique and image generation want different models, and here they are one client, one key, one bill, and a string change.
import os, httpx
OR = "https://openrouter.ai/api/v1"
HEADERS = {
"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}",
"HTTP-Referer": "https://poster.studio", # optional: app attribution
"X-Title": "Poster Studio", # shows in OpenRouter rankings
}
r = httpx.post(f"{OR}/chat/completions", headers=HEADERS, timeout=120, json={
"model": "anthropic/claude-sonnet-4.5",
"messages": [
{"role": "system", "content": "You are a poster art director. Be terse."},
{"role": "user", "content": "Three concepts for a jazz night at a Nairobi rooftop bar."},
],
"temperature": 0.8,
"max_tokens": 600,
"usage": {"include": True}, # ask for real cost back in the response
})
d = r.json()
print(d["choices"][0]["message"]["content"])
print(d["usage"]) # prompt_tokens, completion_tokens, cost in USD
vendor/model. Because the surface is OpenAI-compatible you can also point the official openai SDK at base_url="https://openrouter.ai/api/v1" and use it unchanged — convenient, and it means every code sample written for OpenAI works here.Knobs that actually matter
- temperature
- Randomness. Near 0 for anything you will parse or diff; 0.7–1.0 for idea generation. In the poster tool: high for concepting, low for emitting layout patches.
- max_tokens
- A cost ceiling and a runaway guard, not a quality setting. Always set it.
- seed
- Best-effort reproducibility where the provider supports it. Useful for evals; never rely on it for correctness — batching and kernel non-determinism mean identical inputs can still diverge.
- response_format
- Schema-constrained output:
{"type":"json_schema","json_schema":{"name":...,"strict":true,"schema":{...}}}. Enforcement varies by provider — some guarantee conformance, others treat the schema as a strong suggestion. Pair withprovider.require_parameters: trueso you only route to endpoints that really support it. - provider
- OpenRouter’s routing preferences:
order,allow_fallbacks,only/ignore,sortbyprice/throughput/latency, plusdata_collection: "deny"andzdrfor privacy constraints. The shortcuts:floor(cheapest) and:nitro(fastest) appended to a model ID do the common cases in one string.
By default OpenRouter load-balances across providers of the same model by price. That is fine for drafting and wrong for anything you are evaluating, because the endpoint serving you can change between runs and different providers of “the same” model differ in quantisation, context limit and parameter support. For eval runs and for anything user-facing that you have tuned, pin the provider with order and set allow_fallbacks: false, or accept that your measurements have a hidden variable in them.
Tools: how a model acts on the world
A tool is two things that live in different places: a JSON Schema the model reads, and a function you run. The gap between them is where every safety property you have comes from — and it is also the thing people accidentally close by wiring the model straight into their database.
The four-step dance
- You send the message list plus an array of tool schemas.
- The model replies with
tool_calls: a name, an id, and arguments as a JSON string (not an object — it can be malformed, and sometimes is). - You validate the arguments, decide whether to run it, run it, and capture the result.
- You append the assistant message and a
toolmessage carrying the matchingtool_call_id, then call the model again with the longer list.
import json, httpx
TOOLS = [{
"type": "function",
"function": {
"name": "check_contrast",
"description": (
"Compute the WCAG contrast ratio between two hex colours. "
"Use before committing a text colour over a background. "
"Returns the ratio and whether it passes AA for the given size."
),
"parameters": {
"type": "object",
"properties": {
"fg": {"type": "string", "pattern": "^#[0-9a-fA-F]{6}$"},
"bg": {"type": "string", "pattern": "^#[0-9a-fA-F]{6}$"},
"pt_size": {"type": "number", "minimum": 4},
},
"required": ["fg", "bg", "pt_size"],
"additionalProperties": False,
},
},
}]
def check_contrast(fg, bg, pt_size):
def lin(c):
c = c / 255
return c / 12.92 if c <= 0.04045 else ((c + 0.055) / 1.055) ** 2.4
def lum(h):
r, g, b = (int(h[i:i+2], 16) for i in (1, 3, 5))
return 0.2126*lin(r) + 0.7152*lin(g) + 0.0722*lin(b)
a, b = sorted([lum(fg), lum(bg)], reverse=True)
ratio = (a + 0.05) / (b + 0.05)
threshold = 3.0 if pt_size >= 18 else 4.5
return {"ratio": round(ratio, 2), "passes_aa": ratio >= threshold,
"threshold": threshold}
REGISTRY = {"check_contrast": check_contrast}
msgs = [
{"role": "system", "content": "You are a poster art director."},
{"role": "user", "content": "Is #6B5B2A text OK on #2A2418 at 14pt?"},
]
while True:
r = httpx.post(f"{OR}/chat/completions", headers=HEADERS, timeout=120, json={
"model": "anthropic/claude-sonnet-4.5",
"messages": msgs, "tools": TOOLS, "tool_choice": "auto",
}).json()
m = r["choices"][0]["message"]
msgs.append(m) # append the model's own turn, verbatim
calls = m.get("tool_calls") or []
if not calls:
print(m["content"]); break # ← the loop's only natural exit
for c in calls: # models may request several at once
try:
args = json.loads(c["function"]["arguments"]) # can be malformed
out = REGISTRY[c["function"]["name"]](**args)
body = json.dumps(out)
except Exception as e:
body = json.dumps({"error": str(e)}) # feedback, not a crash
msgs.append({"role": "tool", "tool_call_id": c["id"], "content": body})
Designing tools the model can actually use
Tool descriptions are prompt. They are read by the model on every single call, they cost tokens on every single call, and they are the main lever on whether the agent behaves. Treat them like API docs written for a capable but literal-minded new hire who cannot ask you a question.
Verb-noun, in the domain’s language
doc.patch, render.preview, image.generate. Not process, handle, doThing. The model routes largely on the name; a vague name gets called at the wrong moment.
Say when to use it, not just what it does
“Use after any layout change and before export” is worth more than a paragraph on the return type. Include one concrete example call if the arguments are at all subtle.
Constrain hard
Enums over free strings, patterns on ids and colours, additionalProperties: false, sensible minimum/maximum. Every constraint is one fewer retry loop and one fewer malformed call you have to hand-handle.
Few, powerful, composable
Thirty narrow tools blow your token budget and your accuracy — selection error rises with the number of similar options. Prefer one doc.patch that takes an operation list over twenty setters.
Errors are prompts
“Invalid layer id ‘hdln’. Valid ids: headline, subhead, date_block.” recovers on the next turn. A stack trace or a bare 500 just burns two turns and then gives up.
Budget the output
A tool that returns 30k tokens of JSON has just eaten your context and will be re-read on every subsequent turn. Return summaries and handles; let the model ask for detail. “Grep, don’t cat.”
Idempotency and approval
Models retry. Give mutating tools idempotency keys, and route anything expensive, destructive or public through an approval gate in the harness — not through a line in the system prompt asking nicely.
Push work out of the model
Anything computable should be a tool, not a thought. Contrast ratios, text metrics, grid maths, colour conversion, price arithmetic. Models are bad at arithmetic and excellent at deciding which arithmetic to do.
Tools vs structured output
Two different jobs that look similar. Structured output (response_format: json_schema) constrains the model’s final answer into a shape you can parse — use it when you want data and the conversation is over. Tools are for when the model needs something to happen and then wants to keep going. In the poster tool: parsing a client brief into a structured spec is structured output; generating an image mid-design is a tool.
Tool results are attacker-controlled input that you insert directly into the model’s context. A web page you fetched, a filename in a folder you listed, an EXIF comment in an uploaded logo, the description text of an MCP server you did not write — all of it lands in the same stream as your instructions, and the model has no reliable way to tell them apart. Prompt injection is not a solved problem and will not be solved by prompting. The mitigations are structural: least privilege on tools, approval gates on consequential actions, egress allowlists, and never granting one agent both access to secrets and the ability to send data outward.
The harness, from scratch
The loop in §02 works and would be irresponsible to ship. It has no budget, so a confused model can spend $40 in a minute. It has no timeout, so one slow tool hangs the request. It has no approval gate, no trace, no cancellation, and no way to resume. Adding those is the harness, and the harness is your product’s actual engineering.
What the harness owns
| Responsibility | What it means concretely | Skip it and… |
|---|---|---|
| Budgets | Max turns, max total tokens, max USD, wall-clock deadline — checked every turn | a loop burns your month’s credit in ten minutes |
| Stop conditions | No tool calls; budget trip; explicit finish tool; no-progress detection | agents that never end, or end early and silently |
| Retries | Backoff on 429/5xx/timeouts, with a cap and jitter; distinguish retryable from fatal | one blip fails a five-minute run |
| Timeouts | Per model call and per tool call, independently | a hung tool holds a request open forever |
| Approval policy | Per-tool allow/ask/deny, evaluated before dispatch, with the arguments shown | the model publishes something, or spends money, unsupervised |
| Context assembly | Deciding what goes into this request — see §04 | quality degrades as runs get longer and nobody knows why |
| Result shaping | Truncating and summarising tool output before it enters the list | one ls of a big folder ends the run |
| Persistence | Durable run state so a crash or a page reload can resume | every failure is a total loss of an expensive run |
| Cancellation | A cooperative stop that also aborts the in-flight model call | “stop” doesn’t, and users lose trust immediately |
| Streaming | Surfacing partial text and tool intentions as they arrive | a 40-second blank screen, which reads as broken |
| Tracing | A span per model call and per tool call: model, tokens, cost, latency, cache hits | you cannot debug, cost-attribute, or evaluate anything |
import asyncio, json, time, uuid
from dataclasses import dataclass, field
@dataclass
class Budget:
max_turns: int = 24
max_cost_usd: float = 2.00
max_seconds: float = 300.0
spent_usd: float = 0.0
started: float = field(default_factory=time.monotonic)
def check(self, turn):
if turn >= self.max_turns: return "max_turns"
if self.spent_usd >= self.max_cost_usd: return "cost_cap"
if time.monotonic() - self.started > self.max_seconds: return "deadline"
return None
@dataclass
class Tool:
name: str
schema: dict
fn: object
timeout: float = 30.0
approval: str = "allow" # allow | ask | deny
max_result_chars: int = 6000
class Agent:
def __init__(self, client, tools, system, *, on_event=None, on_approve=None):
self.client, self.system = client, system
self.tools = {t.name: t for t in tools}
self.on_event = on_event or (lambda **kw: None)
self.on_approve = on_approve or (lambda name, args: True)
async def run(self, goal, *, model="anthropic/claude-sonnet-4.5",
budget=None, cancel: asyncio.Event = None):
budget = budget or Budget()
cancel = cancel or asyncio.Event()
run_id = uuid.uuid4().hex[:8]
msgs = [{"role": "user", "content": goal}]
for turn in range(budget.max_turns + 1):
if cancel.is_set(): return self._stop(msgs, "cancelled")
if (why := budget.check(turn)): return self._stop(msgs, why)
# 1. assemble — the whole request is rebuilt here, every turn (§4)
request = self.assemble(msgs, model)
# 2. call the model, with bounded retries
t0 = time.monotonic()
reply, usage = await self._call_with_retry(request, cancel)
budget.spent_usd += usage.get("cost", 0.0)
self.on_event(type="model", run=run_id, turn=turn, model=model,
ms=int((time.monotonic()-t0)*1000), usage=usage,
spent=round(budget.spent_usd, 4))
msgs.append(reply)
calls = reply.get("tool_calls") or []
if not calls:
return self._stop(msgs, "complete", text=reply.get("content"))
# 3-6. dispatch every requested call concurrently, each bounded
results = await asyncio.gather(
*(self._dispatch(c, run_id, turn) for c in calls))
msgs.extend(results)
return self._stop(msgs, "max_turns")
async def _dispatch(self, call, run_id, turn):
cid = call["id"]
name = call["function"]["name"]
tool = self.tools.get(name)
def result(body, ok=True):
self.on_event(type="tool", run=run_id, turn=turn, tool=name, ok=ok)
return {"role": "tool", "tool_call_id": cid,
"content": body[:tool.max_result_chars] if tool else body}
if tool is None:
return result(f"No such tool '{name}'. Available: "
f"{', '.join(sorted(self.tools))}", ok=False)
try:
args = json.loads(call["function"]["arguments"] or "{}")
except json.JSONDecodeError as e:
return result(f"Arguments were not valid JSON: {e}. Re-send them "
f"as a JSON object matching the schema.", ok=False)
if tool.approval == "deny":
return result(f"'{name}' is not permitted in this run.", ok=False)
if tool.approval == "ask" and not await _maybe_await(
self.on_approve(name, args)):
return result(f"The user declined '{name}'. Try another approach "
f"or ask them what they would prefer.", ok=False)
try:
out = await asyncio.wait_for(_maybe_await(tool.fn(**args)),
timeout=tool.timeout)
return result(out if isinstance(out, str) else json.dumps(out))
except asyncio.TimeoutError:
return result(f"'{name}' timed out after {tool.timeout}s. "
f"Try a narrower request.", ok=False)
except Exception as e:
# the model can often fix this itself — hand it back, don't raise
return result(f"{type(e).__name__}: {e}", ok=False)
isError: true exists in MCP’s tool results rather than a JSON-RPC error.Failure taxonomy
| Failure | Where it shows up | Handle it by |
|---|---|---|
| Rate limit / 5xx / provider hiccup | Model call | Retry with jittered backoff; consider an OpenRouter fallback provider |
| Malformed tool arguments | Parse step | Return the parse error as a tool message; tighten the schema |
| Tool raises | Dispatch | Return the exception text as a tool result, not an HTTP 500 |
| Tool hangs | Dispatch | Per-tool timeout; tell the model it timed out so it narrows the request |
| Loops — same call, same args, repeatedly | Across turns | Hash the last N calls; on repeat, inject “you have tried this twice, it did not work” and force a different path |
| Context overflow | Assemble | Compact before you hit the wall, not after the API rejects you (§04) |
| Silent quality decay on long runs | Nowhere — that is the problem | Traces plus an eval suite (§07). You will not notice this by using the product. |
Make the run resumable from the beginning: a row per run with the message list, the budget state, and a monotonically increasing turn index, written after every turn. It costs an afternoon and it buys you crash recovery, page-reload survival, background execution, a debug view of exactly what the model saw, and the ability to fork a run from turn 7 to try a different prompt. Retro-fitting it after the loop is entangled with your web request is genuinely painful.
Context engineering
This is the discipline that separates a demo from a product. Everything else in this document is mechanism; this is judgement. The question is always the same and you answer it fresh on every single turn: given a fixed budget, what are the most useful tokens I can put in front of this model right now?
A budget, not a container
Two independent pressures push in opposite directions. Larger context costs more money and more latency on every turn, and it grows monotonically unless you intervene. And separately — the part people underestimate — model quality degrades well before the window is full. Attention is finite and diluted; instructions given at turn 1 compete with 90k tokens of tool output by turn 20; models reliably retrieve a fact from the start and the end of a long input and much less reliably from the middle. A 1M-token window is not permission to use 900k of it.
Four moves, in order of leverage
Offload
Keep the artifact outside the context. The design document, the asset library, the search results, the file — live in a store; context carries a compact projection and a handle. The model asks for detail when it needs detail.
Highest leverage by a distance, and the one that changes your architecture. MCP’s own guidance now says the same thing: return a handle from a creation tool and take it as an argument later.
Select
Retrieve only what this turn plausibly needs. Three relevant reference posters beat forty. A ranked search result beats a directory listing. If you cannot articulate why a token is in the request, take it out.
Compress
Summarise old turns into a durable brief once history crosses a threshold; drop superseded tool results entirely (an old render preview has no value once a newer one exists). Compact before you hit the wall, on your schedule, not the API’s.
Isolate
Give a sub-task its own fresh window and return only its conclusion. “Research five poster references and report three” costs 40k tokens in a sub-agent and adds 400 to the parent. See §06 — and note the cost: whatever the sub-agent saw is gone.
Prompt caching: the discipline that pays for itself
Providers cache the model’s internal state for a repeated prefix of your input, and charge a fraction of the normal input price for the cached portion. On an agent loop, where turn 12 re-sends everything turns 1–11 sent, this is not a micro-optimisation — it is often the majority of your bill and a large share of your latency.
The rule is simple and unforgiving: a cache hit requires a byte-identical prefix. So:
- Order by stability. System prompt first, then tool schemas, then long-lived context, then the volatile conversation. One dynamic token near the front invalidates everything behind it.
- Never put a timestamp, a random id, or a “current time” line in the system prompt. This is the single most common way teams silently disable their own cache. Put it in the last user message instead.
- Serialise tool schemas deterministically — same order, same key order, every time. This matters enough that the 2026-07-28 MCP specification now says servers should return tools from
tools/listin a deterministic order explicitly to improve prompt cache hit rates. - Append, don’t rewrite. Editing an early message to “clean up” history costs you the whole cache. If you must compact, do it rarely and deliberately, and accept the one-off miss.
Tool results are where the budget actually goes
In practice most context bloat is not conversation — it is tool output. A directory listing, a full JSON document, an un-truncated API response, a page of HTML. Rules that hold up:
| Instead of | Do this | Why |
|---|---|---|
| Returning the whole design document after every edit | Return {"ok":true,"ops":2,"doc_version":18} and let a separate doc.summary tool fetch a projection | The doc is the artifact, not the conversation. It can be 100k tokens. |
| Dumping a search result page | Return the top 5 with title, one line, and an id to fetch | Selection is your job, not the model’s |
| Returning a full-resolution render | Return a downscaled preview (long edge ~1024) plus deterministic check results | Images are expensive in tokens and the model does not need print resolution to judge composition |
| Truncating silently at N characters | Truncate and say so: “[showing 40 of 312 results; call again with offset]” | A model that knows it was truncated will paginate; one that doesn’t will conclude there were 40 |
| Keeping every historical tool result | Replace superseded ones with a one-line stub | Only the newest render matters; the previous six are pure cost |
State lives in a store; context holds a projection. The design document — every layer, every coordinate, every colour — is a JSON structure in your database that can run to tens of thousands of tokens. It never goes into context in full. What the agent sees each turn is a compact projection: canvas size, the palette, a one-line summary per layer, the current version number, and the last render as an image. It edits by emitting patches against layer ids, and the store applies them.
This gives you the same four properties the state-handle pattern gives an MCP server: bounded context, atomic edits, a version history you can undo, and the ability for a human to edit the document directly between turns without the agent’s context going stale — because the agent re-reads the projection at the top of every turn anyway.
A worked budget for the poster agent
| Slot | Budget | Contents | Cache |
|---|---|---|---|
| System prompt | ~1.5k | Role, design principles, hard rules, output conventions | stable |
| Tool schemas | ~3k | 9–12 tools, deterministically ordered | stable |
| Brief | ~1k | The parsed client brief, as structured data | stable per run |
| Doc projection | ~2–4k | Canvas, palette, type pair, one line per layer, version | changes each turn |
| Latest render | ~1.5k | One downscaled preview image | changes |
| Check results | ~0.5k | Deterministic lint output: contrast, overflow, margins, DPI | changes |
| Recent turns | ~6k | Last few exchanges verbatim; older ones compacted to a brief | rolling |
| Total | ~16–18k | Flat across the run. It should not grow with turn count — if it does, one of the four moves is missing. | |
MCP, as of the 2026-07-28 revision
The Model Context Protocol solves one specific problem: N agent applications each needing bespoke integrations with M systems is N×M pieces of glue, and every one of them is written twice. MCP makes it N+M by standardising how a capability describes itself and how it is invoked. That is the whole pitch. It is a plug shape, not a theory of agents.
MCP changed substantially in the 2026-07-28 revision, and a great deal of writing about it — including plenty published this year — describes a protocol that no longer exists. The headline: MCP is now stateless. There is no initialize handshake, no Mcp-Session-Id, and the list endpoints no longer vary per connection. Separately, Roots, Sampling and Logging are deprecated, with the suggested migration for Sampling being “integrate directly with LLM provider APIs instead”. If a tutorial opens with an initialize handshake or has your server calling back into the model, it is out of date.
The topology, and the misconception it fixes
The primitives
| Primitive | Controlled by | What it is | Status |
|---|---|---|---|
| Tools | The model | Callable functions with an inputSchema, optional outputSchema, and structured or unstructured results | the main event |
| Resources | The application | Readable data addressed by URI, listed and fetched by the host and injected as context | active |
| Prompts | The user | Named, parameterised templates a server offers — slash commands, essentially | active |
| Elicitation | The server, mid-call | “I need input from the human before I can finish”, now delivered through the MRTR pattern below | active, reshaped |
| Tasks | The server | Long-running work, polled via tasks/get with tasks/update for input | moved to an official extension |
| Sampling | The server | Server asks the host to run an LLM completion on its behalf | deprecated — call a provider API directly |
| Roots | The client | Telling a server which directories it may operate in | deprecated — pass paths as tool arguments |
| Logging | The server | notifications/message log records | deprecated — use stderr or OpenTelemetry |
Deprecated features stay functional for a minimum twelve-month window under the new lifecycle policy, but new implementations should not adopt them.
What changed, and why it matters to you
- Stateless by default
- The
initialize/notifications/initializedhandshake and the session header are gone. Every request carries its own protocol version, client identity and capabilities in_meta. Practically: a server is now an ordinary stateless HTTP service you can put behind a load balancer without sticky sessions. server/discover- Servers must implement it; clients may call it up front to negotiate versions and read capabilities and identity. It replaces the handshake’s discovery role without creating a connection.
- Multi Round-Trip Requests
- The big conceptual change. Instead of a server initiating a request back at the client, it returns a result with
resultType: "input_required"and aninputRequestsmap. The client gathers what was asked for and retries the original request withinputResponsesand the opaquerequestStatethe server handed back. One direction of travel, always. resultTypeon every result"complete"or"input_required". Results from older servers that omit it must be treated as"complete".subscriptions/listen- One long-lived POST-response stream replaces the old HTTP GET endpoint and
resources/subscribe. Clients opt in to specific notification types. Request-scoped notifications likenotifications/progressstill ride the response stream of their own request. - Cacheable lists
tools/list,prompts/list,resources/listandresources/readnow returnttlMsandcacheScope, and servers should return tools in a deterministic order — explicitly to help clients cache and to improve LLM prompt cache hit rates. §04’s discipline, written into the protocol.- No stream resumability
Last-Event-IDand SSE event ids are gone from Streamable HTTP. A broken stream loses the in-flight request and the client re-issues it with a new id. Design your tools to be safely re-runnable.- Transports
- stdio for local subprocesses, Streamable HTTP for everything remote. The old HTTP+SSE transport is formally deprecated.
- Authorization
- OAuth 2.x, with Client ID Metadata Documents now preferred over Dynamic Client Registration, and clients required to validate the
issparameter when present. Credentials are bound to the issuing authorization server and must be keyed by issuer.
Stateful work without protocol state
Because there is no session, a server that needs continuity across calls returns an explicit handle from a creation tool and accepts it as an ordinary argument later. This is exactly the “offload” pattern from §04, and it is what the poster document does:
// → tools/call
{ "name": "doc.create", "arguments": { "preset": "A2_portrait" } }
// ← result
{ "resultType": "complete",
"content": [{ "type": "text", "text": "Created document doc_9fk2 (A2, 300dpi)" }],
"structuredContent": { "doc_id": "doc_9fk2", "version": 1 } }
// → tools/call — the model carries doc_id forward; the server looks it up
{ "name": "doc.patch",
"arguments": { "doc_id": "doc_9fk2", "ops": [
{ "op": "set", "path": "/layers/headline/size_pt", "value": 148 } ] } }
The wire, exactly
// → request. _meta is required and carries what the handshake used to.
{ "jsonrpc": "2.0", "id": 1, "method": "tools/list",
"params": {},
"_meta": {
"io.modelcontextprotocol/protocolVersion": "2026-07-28",
"io.modelcontextprotocol/clientInfo": { "name": "poster-studio", "version": "0.4.1" },
"io.modelcontextprotocol/clientCapabilities": {}
} }
// ← result
{ "jsonrpc": "2.0", "id": 1, "result": {
"resultType": "complete",
"tools": [{
"name": "render.preview",
"title": "Render preview",
"description": "Rasterise a design document to a downscaled PNG. Call after any layout change, before judging composition.",
"inputSchema": {
"type": "object",
"properties": {
"doc_id": { "type": "string" },
"long_edge":{ "type": "integer", "minimum": 256, "maximum": 2048, "default": 1024 }
},
"required": ["doc_id"],
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"properties": { "width": {"type":"integer"}, "height": {"type":"integer"},
"version": {"type":"integer"} },
"required": ["width","height","version"]
}
}],
"ttlMs": 300000,
"cacheScope": "public"
} }
// ← a tool result carrying an image the model can actually look at
{ "jsonrpc": "2.0", "id": 2, "result": {
"resultType": "complete",
"content": [
{ "type": "text", "text": "Rendered doc_9fk2 v18 at 1024x1448." },
{ "type": "image", "data": "iVBORw0KGgoAAA...", "mimeType": "image/png",
"annotations": { "audience": ["assistant"], "priority": 0.9 } }
],
"structuredContent": { "width": 1024, "height": 1448, "version": 18 },
"isError": false
} }
error means the request was structurally wrong — unknown tool, malformed params — and the model probably cannot fix it. A normal result with isError: true means the tool failed in a way the model can act on, and its text is written for the model to read. Use the second one far more often.from mcp.server.fastmcp import FastMCP, Image
mcp = FastMCP("poster-render")
@mcp.tool()
def render_preview(doc_id: str, long_edge: int = 1024) -> list:
"""Rasterise a design document to a downscaled PNG.
Call after any layout change and before judging composition.
Returns the preview image plus its pixel dimensions and doc version.
"""
doc = STORE.get(doc_id) # server-side state, by handle
if doc is None:
raise ValueError(f"Unknown doc_id {doc_id!r}. Call doc.create first.")
png, w, h = rasterise(doc, long_edge=long_edge)
return [
f"Rendered {doc_id} v{doc.version} at {w}x{h}.",
Image(data=png, format="png"),
]
if __name__ == "__main__":
mcp.run() # stdio by default; streamable HTTP for remote
inputSchema from the type hints and the description from the docstring, so the docstring is prompt engineering — write it for the model. Pin your SDK version and read its changelog: the surface tracks a spec that has been moving quickly.When to reach for MCP — and when not to
The capability crosses an application boundary
You want your render service usable from your own app, from Claude Code, from a colleague’s agent, from a CI job. Or you are consuming someone else’s server. Or the tool is a separate process for isolation reasons, and you would otherwise be inventing an RPC protocol.
It’s a function in your own codebase
check_contrast() does not need a JSON-RPC server, a subprocess and a schema round trip. Put it in your tool registry directly. Wrapping in-process functions in MCP for its own sake buys serialisation cost, an extra failure mode, and a deployment artifact, in exchange for nothing.
Third-party servers are third-party code
A server’s tool names, descriptions and annotations are text that lands in your model’s context, and the spec is explicit that clients must treat annotations from untrusted servers as untrusted. A malicious description is a prompt injection with a delivery mechanism. Review what you install, pin versions, and keep a server that reads your files away from one that can post to the internet.
Token cost is real
Every connected server’s tool list is re-sent on every turn. Ten servers with twelve tools each is a permanent tax on every request and a measurable drop in tool-selection accuracy. Expose a curated subset per task rather than everything you happen to have connected.
Multi-agent orchestration
The honest starting position: one well-built loop with good tools beats a multi-agent system almost every time, and most multi-agent architectures are a context problem being solved with an org chart. Reach for a second agent when you can name the specific property you are buying.
| Pattern | Buy this | Pay that | Use it in the poster tool for |
|---|---|---|---|
| Single loop | Simplicity, full history, one place to debug | One context window for everything | Everything, until measurement says otherwise |
| Orchestrator–worker | Parallelism; a clean window per subtask | N× tokens; workers can’t see each other; handoff loss | Concept exploration — three directions researched at once, three one-paragraph reports back |
| Pipeline | A different model and prompt per stage; cheap stages stay cheap | Rigid; errors compound down the chain | brief → concept → layout → imagery → export |
| Maker–critic | A genuine quality lift; the critic isn’t anchored on the maker’s reasoning | 2–3× cost per round; can oscillate if unbounded | Design review against the brief and the deterministic checks |
| Router | Cheap model handles the 80% of easy requests | Misroutes; two behaviours to keep consistent | “make the headline bigger” → small fast model; “redesign this” → the good one |
What crosses an agent boundary is a schema’d artifact, never a conversation. A worker returns {concept, rationale, palette, refs[]} validated against a schema — not a chat transcript, not “here’s what I found” prose. Two reasons: the parent’s context stays bounded and predictable, and the boundary becomes testable, so you can evaluate a worker in isolation instead of only observing that the whole system got worse.
The corollary people miss: a sub-agent that was given a vague brief produces a confident, wrong artifact, and the parent has no way to tell. Sub-agent briefs need to be more specific than what you would say to a human, including what to return and what not to bother with.
Evals and observability
You are shipping a system whose output changes when nothing changed. Every instinct from deterministic software — a test asserts an exact value, a green build means it works — fails here. What replaces them is measurement over distributions, and a large pile of deterministic checks around a small stochastic core.
Trace first. You cannot evaluate what you cannot see.
Before any eval work, emit a span per model call and per tool call. OpenTelemetry is the sane default, and the 2026-07-28 MCP spec even documents trace-context propagation through _meta (traceparent, tracestate, baggage) so a trace can follow a call into a server. Each span carries:
| Field | Why you will want it at 2am |
|---|---|
run_id, turn, parent_span | Reconstruct the exact sequence, including sub-agents |
model, provider | OpenRouter may have routed you somewhere different than yesterday |
prompt_tokens, completion_tokens, cached_tokens | Cache hit rate is the difference between a viable and an unviable margin |
cost_usd | Per run, per user, per feature. Ask for it with usage: {include: true}. |
latency_ms, ttft_ms | Time-to-first-token is what users actually feel |
tool_name, ok, error_class | Tool error rate is the single best early-warning signal you have |
| Full request and response bodies | Sampled, redacted, retained briefly. Reading what the model actually saw resolves most “why did it do that” questions in a minute. |
Three levels of evaluation
Unit tests, no model involved
check_contrast, the grid maths, the patch applier, the renderer. These are ordinary pure functions and should have ordinary fast tests with exact assertions. Most of your code is here and it should be.
Does the model make the right move?
Fixed context in, one turn out. Did it pick the right tool? Were the arguments valid against the schema? Did it call render.preview before claiming the layout was fine? Cheap, fast, run on every PR, assert on a pass rate over ~30 cases rather than on a single run.
Is the poster any good?
A whole run against a real brief. Slow and expensive, so it runs nightly and before releases, over a fixed golden set. This is where judgement is needed, and where the trick below matters.
Judging a poster
“Is this design good?” sounds like it needs a model to answer. Most of it does not, and the parts that do not are the parts you can trust. Split the question:
| Check | How | Verdict |
|---|---|---|
| Text contrast against its backdrop | WCAG ratio computed from the rendered pixels under each text layer | deterministic |
| Text overflow, clipping, collision | Measure laid-out text boxes; compare against layer bounds and each other | deterministic |
| Safe margins and bleed | Geometry against the print spec (3mm bleed, 5mm safe area for A2) | deterministic |
| Minimum legible type size | Point size at final trim dimensions, per layer role | deterministic |
| Effective image resolution | Source pixels ÷ placed physical size ≥ 300 dpi | deterministic |
| Brief compliance — date, venue, price, logo present and correct | String and asset presence in the document, not in the image | deterministic |
| Palette conformance to brand | Colour distance from the allowed set | deterministic |
| Visual hierarchy — does the eye land on the headline first? | Vision model, rubric, pairwise | judged |
| Does it feel like “late-night jazz” rather than “corporate seminar”? | Vision model, rubric, pairwise | judged |
| Is it good? | A human, sampled | unavoidable |
Roughly nine out of ten genuinely bad outputs from a design agent fail one of the mechanical checks: unreadable text, a headline running off the canvas, a logo at 72dpi, the wrong date. These are cheap to detect, unambiguous, and — the important part — you can hand the failures straight back to the agent as a tool result and let it fix them in the loop. Deterministic checks are not just an eval; they are the feedback signal that makes the agent good in the first place.
Making a model judge less unreliable
- Compare, don’t score. “Which of these two better fits the brief?” is far more stable than “rate this 1–10”, which drifts between runs and clusters around 7.
- Randomise position in pairwise comparisons and run both orders — judges have a real preference for whichever came first.
- Give the judge the rubric and the brief, ask for a verdict per criterion with a one-line reason, and use a different model from the maker. Self-preference is measurable.
- Calibrate against humans once. Have a designer rank 40 outputs, then check your judge’s agreement. If it is near chance, the judge is noise dressed as a number and you should say so out loud.
- Track disagreement as a metric. Where the judge and the humans diverge is exactly where the rubric is wrong.
The golden set, and what to assert
Thirty to fifty real briefs, versioned in the repo, covering the boring middle and the edges: very long headlines, Swahili and English mixed, a client logo that is a wide horizontal lockup, a brief with no imagery budget, a brief that contradicts itself. For each, assert properties, never outputs:
import pytest
from studio import run_brief, checks
GOLDEN = load_briefs("evals/briefs/*.yaml")
@pytest.mark.parametrize("brief", GOLDEN, ids=lambda b: b.id)
def test_brief_produces_a_shippable_poster(brief):
result = run_brief(brief, seed=7, budget_usd=1.50)
# hard gates — a failure here is a release blocker
assert result.stop_reason == "complete"
assert result.cost_usd <= 1.50
report = checks.run_all(result.doc)
assert report.contrast_failures == []
assert report.overflow_failures == []
assert report.min_effective_dpi >= 300
for fact in brief.must_appear: # date, venue, price, age rating
assert fact in result.doc.text_content()
# soft signals — recorded per run, alerted on trend, never asserted
metrics.record(brief.id, turns=result.turns, cost=result.cost_usd,
seconds=result.seconds, judge=judge_vs_baseline(result))
The metric that beats all the others
Once real users are on it: the human edit rate. What fraction of agent-produced layers does a designer change before exporting, and which ones? If they always fix the type size, your system prompt or your type tool is wrong. If they always regenerate the image, your imagery prompt is wrong. It is free to collect, impossible to game, and it points at the specific broken thing rather than at a number going down.
The poster tool: architecture
Now spend all of it on one thing. The goal: a client types a brief, an agent designs an A2 poster, a human nudges it, and a print-ready PDF comes out. The single decision that determines whether this works is what the agent actually produces.
The fork: what does the model emit?
| Approach | Model emits | Text quality | Editable after? | Cost per iteration | Print-ready? |
|---|---|---|---|---|---|
| Pure image generation | A prompt; you get pixels | mangled — typography is where image models still fall down, and a poster is typography | No — a flat raster | $0.03–0.15 × full regenerate | No — fixed resolution, RGB, no bleed |
| Pure structured layout | A document: layers, type, shapes | perfect — real fonts, real kerning | Yes, fully | fractions of a cent | Yes |
| Hybrid chosen | A document whose image layers reference generated assets | perfect — text never goes through the image model | Yes, every layer independently | cheap to nudge, pay only to regenerate art | Yes |
The hybrid is how design software works and it is not a compromise — it is strictly better on every axis that matters for a poster. It also gives you the property the whole system hangs on: the design is data, so it can be diffed, versioned, undone, linted, replayed, handed to a human mid-flight, and rendered at any resolution.
The design document
This schema is the product. It is what your renderer consumes, what your React canvas edits, what the agent patches, what your checks lint, and what you version. Design it first and change it rarely.
from pydantic import BaseModel, Field
from typing import Literal, Union, Annotated
Mm = float # one unit everywhere: millimetres
class Canvas(BaseModel):
width_mm: Mm = 420.0 # A2 portrait
height_mm: Mm = 594.0
bleed_mm: Mm = 3.0
safe_mm: Mm = 10.0
dpi: int = 300
colour_space: Literal["sRGB", "CMYK"] = "CMYK"
class Grid(BaseModel):
columns: int = 6
gutter_mm: Mm = 6.0
margin_mm: Mm = 18.0
baseline_mm: Mm = 4.0 # everything snaps to this
class TypeScale(BaseModel):
display: str # font family, must exist in the font set
text: str
ratio: float = 1.333 # sizes are derived, not chosen ad hoc
base_pt: float = 11.0
class Box(BaseModel):
x_mm: Mm; y_mm: Mm; w_mm: Mm; h_mm: Mm
rotation_deg: float = 0.0
class TextLayer(BaseModel):
kind: Literal["text"] = "text"
id: str # stable, human-readable: "headline"
role: Literal["headline","subhead","body","credit","caption"]
box: Box
content: str
step: int = 0 # position on the type scale, not a pt size
weight: int = 400
tracking: float = 0.0
leading: float = 1.15
colour: str = Field(pattern=r"^#[0-9a-fA-F]{6}$")
align: Literal["left","center","right"] = "left"
class ImageLayer(BaseModel):
kind: Literal["image"] = "image"
id: str
box: Box
asset_id: str # a handle. bytes live in the asset store.
fit: Literal["cover","contain"] = "cover"
opacity: float = 1.0
blend: Literal["normal","multiply","screen"] = "normal"
class ShapeLayer(BaseModel):
kind: Literal["shape"] = "shape"
id: str
box: Box
shape: Literal["rect","ellipse","line"]
fill: str | None = None
stroke: str | None = None
stroke_mm: Mm = 0.0
Layer = Annotated[Union[TextLayer, ImageLayer, ShapeLayer],
Field(discriminator="kind")]
class Document(BaseModel):
doc_id: str
version: int = 1
canvas: Canvas = Canvas()
grid: Grid = Grid()
type_scale: TypeScale
palette: list[str] # ordered: ground, ink, accent, …
layers: list[Layer] # painted back to front
step, not size_pt — the model picks a rung on a scale rather than inventing 47.3pt, which is most of what makes generated typography look amateur. Layers reference asset_id — so a 4MB image never touches the document, the context, or a diff.The tool catalogue
Twelve tools. Fewer than you expect, deliberately: every extra tool costs tokens on every turn and dilutes selection accuracy. Each row here carries the same six commitments a plugin extension point does — signature, timing, failure mode, budget, cost, permission.
| Tool | Does | Returns | Cost / latency | Approval |
|---|---|---|---|---|
doc.create | New document from a preset (A2, A1, 1080×1350…) | doc_id, version | ~0 · 20ms | allow |
doc.summary | The projection the agent reads each turn | ~2–4k tokens of structure, no coordinates dumped | ~0 · 10ms | allow |
doc.patch | Apply an op list: set / add_layer / move / remove | {ok, ops_applied, version} — not the document | ~0 · 15ms | allow |
doc.undo | Revert to a previous version | New version number | ~0 · 10ms | allow |
type.pair | Suggest display + text families from the licensed set, with a rationale | 3 candidate pairs | ~0 · 5ms | allow |
palette.build | Derive a palette from a mood, a brand colour, or a source image | Ordered hex list + contrast matrix | ~0 · 30ms | allow |
layout.grid | Snap layers to the column grid and baseline | Which layers moved, by how much | ~0 · 20ms | allow |
render.preview | Rasterise at 1024px long edge | PNG image content block + dimensions | ~0 · 300ms | allow |
checks.run | The deterministic lint suite from §07 | Findings list, each with a layer id and a suggested fix | ~0 · 150ms | allow |
image.generate | OpenRouter /api/v1/images; stores the result | asset_id, dimensions, cost — not the bytes | $0.02–0.15 · 5–25s | ask above a per-run count |
image.edit | Same endpoint with input_references — variations, extend, restyle | asset_id | $0.02–0.15 · 5–25s | ask |
render.export | Print-ready PDF: 300dpi, CMYK, bleed, crop marks, fonts outlined | File handle + a preflight report | ~0 · 3–8s | ask — this is the “done” button |
doc.patch and not doc.write
Letting the model re-emit the whole document every time it wants to change a font size is the most expensive mistake available here. It costs thousands of output tokens per edit, it is slow, it silently loses layers the model forgot to include, it produces useless diffs, and it makes concurrent human editing impossible. A patch is small, atomic, attributable, reversible, and cheap to validate — and it lets a designer drag a layer in the canvas at the same moment the agent is adjusting a colour.
Model routing on OpenRouter
Four jobs with genuinely different requirements, one client, four model ids. This is the concrete reason to sit behind an aggregator rather than a single vendor.
| Job | Wants | Settings | Runs |
|---|---|---|---|
| Brief parsing | Reliable schema conformance, cheap | Small model, temperature: 0, response_format: json_schema with strict, provider.require_parameters: true | once |
| Concepting | Range and taste | Strong reasoning model, temperature: 0.9, three samples in parallel | once |
| Layout & patching | Instruction-following, tool use, speed — this is the hot loop | Mid-tier model, temperature: 0.2, prompt caching on, :nitro if latency bites | 10–30× |
| Critique | Vision, and a different lineage from the maker | A vision model from another vendor, rubric in the prompt, pairwise where possible | 2–3× |
| Imagery | Art direction, style control, reference following | /api/v1/images, aspect_ratio matched to the layer box, seed pinned for reproducibility, input_references for variations | 1–5× |
Pin providers for anything you measure. The default is price-based load balancing across providers of the same model, and providers differ in quantisation, context limit and which parameters they support. Set provider.order and allow_fallbacks: false in evals, or your regression suite has a hidden variable.
Images come back as base64, not URLs. The response is data[].b64_json with a media_type; storage is yours. Decode, hash, write to object storage, and return an asset_id to the model — never let a megabyte of base64 anywhere near the message list.
The human in the loop
The designer is not an approver at the end; they are a participant. Three gates, and the third is the one that makes the product feel like a tool rather than a slot machine:
- Concept selection. Three directions with rationales; the human picks one. Cheap, fast, and it collapses the largest source of wasted work.
- Spend approval. Image generation and export go through the harness’s
askpolicy, with the prompt and estimated cost shown. - Direct editing, mid-run. Because the document is the shared artifact and the agent re-reads its projection each turn, a designer can drag a layer, retype a headline, or lock a layer while the agent is working. Add a
locked: trueflag thatdoc.patchrefuses to override, and the collaboration works without any coordination protocol at all.
The poster tool: the code
Enough of it to see how the pieces fit. Everything here plugs into the Agent class from §03 and the schema from §08.
The patch applier — the tool that does most of the work
from pydantic import ValidationError
ALLOWED_FIELDS = {"box", "content", "step", "weight", "tracking", "leading",
"colour", "align", "opacity", "blend", "fit", "asset_id",
"fill", "stroke", "stroke_mm", "palette", "type_scale"}
def doc_patch(doc_id: str, ops: list[dict]) -> dict:
"""Apply a list of edit operations to a design document.
Each op is one of:
{"op":"set", "layer":"headline", "field":"step", "value":4}
{"op":"add_layer", "layer":{...full layer object...}, "after":"headline"}
{"op":"move", "layer":"credit", "dx_mm":0, "dy_mm":-12}
{"op":"remove", "layer":"stray_rect"}
Operations apply in order and all-or-nothing: if any op is invalid the
document is unchanged and the error tells you which op and why.
"""
doc = STORE.load(doc_id)
if doc is None:
raise ValueError(f"Unknown doc_id {doc_id!r}. Call doc.create first.")
draft = doc.model_copy(deep=True)
by_id = {l.id: l for l in draft.layers}
applied = 0
for i, op in enumerate(ops):
kind = op.get("op")
if kind in ("set", "move", "remove"):
layer = by_id.get(op.get("layer"))
if layer is None:
raise ValueError(
f"op[{i}]: no layer {op.get('layer')!r}. "
f"Layers are: {', '.join(sorted(by_id))}")
if getattr(layer, "locked", False):
raise ValueError(
f"op[{i}]: layer {layer.id!r} is locked by the designer. "
f"Leave it alone and adjust something else.")
if kind == "set":
if op["field"] not in ALLOWED_FIELDS:
raise ValueError(f"op[{i}]: field {op['field']!r} is not "
f"editable. Editable: {sorted(ALLOWED_FIELDS)}")
setattr(by_id[op["layer"]], op["field"], op["value"])
elif kind == "move":
b = by_id[op["layer"]].box
b.x_mm += op.get("dx_mm", 0.0); b.y_mm += op.get("dy_mm", 0.0)
elif kind == "remove":
draft.layers = [l for l in draft.layers if l.id != op["layer"]]
elif kind == "add_layer":
new = parse_layer(op["layer"]) # validates the union
if new.id in by_id:
raise ValueError(f"op[{i}]: layer id {new.id!r} already exists")
idx = next((n for n, l in enumerate(draft.layers)
if l.id == op.get("after")), len(draft.layers) - 1)
draft.layers.insert(idx + 1, new)
else:
raise ValueError(f"op[{i}]: unknown op {kind!r}")
applied += 1
try:
Document.model_validate(draft.model_dump()) # whole-doc invariants
except ValidationError as e:
raise ValueError(f"Result would be invalid: {e.errors()[0]['msg']}")
draft.version = doc.version + 1
STORE.save(draft) # append-only: every version kept
return {"ok": True, "ops_applied": applied, "version": draft.version}
Image generation through OpenRouter
import base64, hashlib, httpx
ASPECTS = {(420, 594): "3:4", (594, 420): "4:3", (1080, 1350): "4:5"}
async def image_generate(doc_id: str, layer_id: str, prompt: str,
style: str = "", seed: int | None = None) -> dict:
"""Generate artwork for one image layer and store it as an asset.
The prompt should describe imagery only — never text, headlines or logos.
Type is set by the layout engine; asking an image model for words is how
posters end up with beautiful gibberish on them.
"""
doc = STORE.load(doc_id)
layer = doc.layer(layer_id)
ratio = nearest_aspect(layer.box.w_mm, layer.box.h_mm)
r = await httpx.AsyncClient().post(
f"{OR}/images", headers=HEADERS, timeout=120,
json={
"model": "google/gemini-2.5-flash-image",
"prompt": f"{prompt}. {style}. No text, no lettering, no logos.",
"aspect_ratio": ratio,
"resolution": "2K", # enough for 300dpi at this box size
"output_format": "png",
"n": 1,
**({"seed": seed} if seed is not None else {}),
})
r.raise_for_status()
body = r.json()
img = body["data"][0]
raw = base64.b64decode(img["b64_json"]) # base64, never a URL
aid = "img_" + hashlib.sha256(raw).hexdigest()[:16]
w, h = ASSETS.put(aid, raw, img.get("media_type", "image/png"))
# check print viability NOW, not at export time
dpi = min(w / (layer.box.w_mm / 25.4), h / (layer.box.h_mm / 25.4))
return { # a handle and facts. never the bytes.
"asset_id": aid, "width": w, "height": h,
"effective_dpi": round(dpi),
"print_ok": dpi >= 300,
"cost_usd": body.get("usage", {}).get("cost"),
"next": f"call doc.patch to set layer {layer_id!r} asset_id to {aid}",
}
Render and check
MM_PER_IN = 25.4
def to_svg(doc: Document, *, embed_assets=True) -> str:
"""The single source of truth for geometry. Preview and export both
go through here, so what the agent judged is what gets printed."""
W, H, bl = doc.canvas.width_mm, doc.canvas.height_mm, doc.canvas.bleed_mm
out = [f'<svg xmlns="http://www.w3.org/2000/svg" '
f'width="{W+2*bl}mm" height="{H+2*bl}mm" '
f'viewBox="{-bl} {-bl} {W+2*bl} {H+2*bl}">']
out.append(f'<rect x="{-bl}" y="{-bl}" width="{W+2*bl}" '
f'height="{H+2*bl}" fill="{doc.palette[0]}"/>')
for layer in doc.layers: # back to front
out.append(render_layer(doc, layer, embed_assets))
out.append("</svg>")
return "".join(out)
def render_preview(doc_id: str, long_edge: int = 1024) -> list:
"""Rasterise to a downscaled PNG. Call after any layout change and
before judging composition or claiming something looks right."""
doc = STORE.load(doc_id)
png = rasterise(to_svg(doc), long_edge=long_edge) # resvg / cairosvg
return [f"Rendered {doc_id} v{doc.version}.",
ImageContent(data=png, mime="image/png")] # ← the model sees this
def checks_run(doc_id: str) -> dict:
"""Deterministic pre-flight. Every finding names a layer and a fix."""
doc, findings = STORE.load(doc_id), []
pixels = rasterise(to_svg(doc), long_edge=1024) # for real contrast
for layer in doc.layers:
if layer.kind == "text":
behind = dominant_colour_under(pixels, doc, layer.box)
ratio = contrast(layer.colour, behind)
pt = doc.type_scale.base_pt * doc.type_scale.ratio ** layer.step
need = 3.0 if pt >= 18 else 4.5
if ratio < need:
findings.append({
"layer": layer.id, "check": "contrast", "severity": "error",
"detail": f"{ratio:.1f}:1 against {behind}, needs {need}:1",
"fix": "lighten the text, or add a scrim behind it"})
w, h = measure_text(layer, doc.type_scale)
if w > layer.box.w_mm or h > layer.box.h_mm:
findings.append({
"layer": layer.id, "check": "overflow", "severity": "error",
"detail": f"text needs {w:.0f}x{h:.0f}mm, box is "
f"{layer.box.w_mm:.0f}x{layer.box.h_mm:.0f}mm",
"fix": "drop one type step, or widen the box"})
if outside_safe_area(layer.box, doc.canvas):
findings.append({"layer": layer.id, "check": "safe_area",
"severity": "error", "detail": "inside the trim margin",
"fix": f"keep text {doc.canvas.safe_mm}mm from every edge"})
if layer.kind == "image":
a = ASSETS.meta(layer.asset_id)
dpi = min(a.w / (layer.box.w_mm / MM_PER_IN),
a.h / (layer.box.h_mm / MM_PER_IN))
if dpi < 300:
findings.append({"layer": layer.id, "check": "dpi",
"severity": "error", "detail": f"{dpi:.0f} dpi at this size",
"fix": "regenerate at higher resolution or shrink the box"})
return {"version": doc.version, "errors":
sum(f["severity"] == "error" for f in findings), "findings": findings}
Wiring it together
SYSTEM = """You are a poster art director working in a structured design tool.
You never draw pixels. You edit a design document made of typed layers, and a
deterministic renderer turns it into a poster. Type is always set by the layout
engine — never ask an image model for words.
Method, every time:
1. doc.summary to see the current state.
2. Make one coherent change with doc.patch. Prefer few, decisive ops.
3. render.preview, then LOOK at it.
4. checks.run. Fix every error before doing anything else.
5. Repeat until checks are clean and the brief is satisfied. Then stop.
Rules:
- Type sizes are steps on the scale. Never invent a point size.
- Everything snaps to the grid and the baseline. Use layout.grid.
- Contrast failures and overflow are never acceptable, even if it looks fine.
- A locked layer belongs to the designer. Work around it.
- Say what you changed and why in one sentence. No commentary on your process.
"""
TOOLS = [ # deterministic order: cache-friendly (§4)
Tool("doc.summary", S.doc_summary, doc_summary, timeout=5),
Tool("doc.patch", S.doc_patch, doc_patch, timeout=10),
Tool("doc.undo", S.doc_undo, doc_undo, timeout=5),
Tool("type.pair", S.type_pair, type_pair, timeout=5),
Tool("palette.build", S.palette_build, palette_build, timeout=5),
Tool("layout.grid", S.layout_grid, layout_grid, timeout=10),
Tool("render.preview", S.render_prev, render_preview,timeout=20),
Tool("checks.run", S.checks_run, checks_run, timeout=20),
Tool("image.generate", S.image_gen, image_generate,timeout=120,
approval="ask"),
Tool("render.export", S.render_export, render_export, timeout=60,
approval="ask"),
]
async def design(brief_text: str, on_event):
brief = await parse_brief(brief_text) # structured output, temp 0
doc = doc_create(preset=brief.format)
agent = Agent(client, TOOLS, SYSTEM, on_event=on_event,
on_approve=ui_approval_prompt)
result = await agent.run(
goal=f"{brief.as_prompt()}\n\nWorking on document {doc['doc_id']}.",
model="anthropic/claude-sonnet-4.5", # the hot loop model
budget=Budget(max_turns=28, max_cost_usd=3.00, max_seconds=420),
)
for _ in range(2): # bounded maker–critic (§6)
notes = await critique(doc["doc_id"], brief) # vision model, other vendor
if not notes.blocking:
break
result = await agent.run(goal=notes.as_instructions(),
budget=Budget(max_turns=10, max_cost_usd=1.00))
return result
The React canvas
The frontend renders the same document the agent edits. Because the document is plain JSON and the geometry lives in one to_svg-shaped function, the canvas is a pure render of state — no separate drawing code to drift out of sync with the print output.
import { useEffect, useReducer } from "react";
// The agent streams { type:"patch", version, ops } events. We apply them
// locally so the canvas updates live, mid-turn, instead of after the run.
function reduce(doc, ev) {
switch (ev.type) {
case "doc": return ev.doc; // full state on load
case "patch": return applyOps(doc, ev.ops, ev.version);
case "locked": return setLayer(doc, ev.layer, { locked: ev.locked });
default: return doc;
}
}
export function Canvas({ docId, onSelect }) {
const [doc, dispatch] = useReducer(reduce, null);
useEffect(() => {
const es = new EventSource(`/api/docs/${docId}/stream`);
es.onmessage = (e) => dispatch(JSON.parse(e.data));
return () => es.close();
}, [docId]);
if (!doc) return <div className="canvas-skeleton" />;
const { width_mm: W, height_mm: H, bleed_mm: B, safe_mm: S } = doc.canvas;
return (
<svg viewBox={`${-B} ${-B} ${W + 2 * B} ${H + 2 * B}`} className="canvas">
<rect x={-B} y={-B} width={W + 2 * B} height={H + 2 * B}
fill={doc.palette[0]} />
{doc.layers.map((l) => (
<Layer key={l.id} layer={l} doc={doc}
onClick={() => onSelect(l.id)}
style={{ cursor: l.locked ? "not-allowed" : "pointer" }} />
))}
{/* print furniture — drawn last, never exported */}
<rect x={0} y={0} width={W} height={H}
fill="none" stroke="#A62834" strokeWidth={0.3} strokeDasharray="2 2" />
<rect x={S} y={S} width={W - 2 * S} height={H - 2 * S}
fill="none" stroke="#12695B" strokeWidth={0.2} strokeDasharray="1 3" />
</svg>
);
}
What frameworks add to the loop you just wrote
Having built §03 by hand, the frameworks become legible: each one is that loop plus a specific opinion. Knowing which opinion you are buying is the whole decision.
| Category | Examples | The opinion you’re buying | What it costs |
|---|---|---|---|
| Provider agent SDKs | OpenAI Agents SDK, Claude Agent SDK | A well-tested loop, tool decorators, handoffs, sessions, built-in tracing. Fastest path from zero to working. | Provider gravity. Several accept an OpenAI-compatible base_url, so they can point at OpenRouter — but the vendor’s own features are the happy path. |
| Graph / state machine | LangGraph | Your agent is an explicit graph with typed state, checkpointing at every node, human-in-the-loop interrupts, and time-travel replay. Deterministic control flow around stochastic nodes. | Real conceptual overhead. Worth it when the flow genuinely branches; heavy when it is one loop. |
| Typed / schema-first | Pydantic AI, Instructor | Validated structured output, typed dependencies injected into tools, model-agnostic by design. Closest in spirit to §03 with the boilerplate removed. | Little. This is often the right default for a Python service. |
| Role / crew abstractions | CrewAI, AutoGen | Multi-agent as a first-class concept: roles, delegation, conversation between agents. | Encourages multi-agent before you have proven you need it, and hides the token cost of doing so. See §06. |
| Durable execution underrated | Temporal, Restate, DBOS | Your run is a workflow: it survives process death, retries individual steps with their own policies, and resumes exactly where it stopped. Nothing agent-specific — which is the point. | Infrastructure. But a 6-minute poster run that dies at minute 5 is exactly the problem these were built for. |
Write the loop yourself once — it is a day, and afterwards no framework is mysterious. Then adopt something for the parts that are genuinely hard infrastructure: durability, checkpointing and tracing. Keep in your own code the parts that are your product: context assembly, the tool definitions, the approval policy, and the evals. Those encode your domain, they are where your quality comes from, and no framework has an opinion about posters.
Build order, and the pitfalls
The order matters more than usual here, because the agent is the last thing you should build, not the first. Every week below produces something a person can use.
- The document, the renderer, the checks — with no model anywhere. A schema, an SVG renderer, a PDF exporter, and the deterministic lint suite. Hand-write three posters as JSON. If the renderer cannot produce something you would print, no agent will fix that. This is also, quietly, a shippable product on its own.
- One model call. Brief text in, a validated
BriefSpecout, via structured output. Fill a hand-made template with it. Still no loop, still no tools. You now have a working generator and a feel for how the model reads briefs. - The loop, with four tools:
doc.summary,doc.patch,render.preview,checks.run. This is the moment it becomes an agent, and the moment tracing has to exist. Do not add a fifth tool until this loop reliably fixes its own check failures. - Imagery.
image.generate, the asset store, DPI validation, and the approval gate. This is the first tool that costs real money and takes real time, which is why it comes after the budget machinery works. - The golden set and the eval harness. Thirty briefs, hard gates on the deterministic checks, cost and latency recorded per run. Wire it into CI before you start tuning prompts, or you will tune against vibes.
- Then, only if measurement says so: a bounded critic pass, parallel concepting, a router for cheap edits, and an MCP server if other applications want your renderer.
| Pitfall | How it presents | The fix |
|---|---|---|
| The model emits the artifact directly | It rewrites the whole document, or the whole SVG, every turn | Patches against a stored document. §08. |
| Context grows with turn count | Turn 20 costs 8× turn 3 and the output is worse | Offload, project, compact. If your context is not roughly flat across a run, one of §04’s four moves is missing. |
| Cache silently disabled | Costs 5–10× higher than expected, no obvious cause | Check for a timestamp or a random id in the system prompt, and for non-deterministic tool ordering |
| Tool sprawl | Twenty-eight tools; the model picks the wrong one; every turn costs 6k tokens before you say anything | Consolidate to a dozen. Expose a task-scoped subset. |
| Exceptions instead of feedback | Runs die on the first bad argument | Every tool failure returns a message that names the fix |
| No budget | A confused loop spends $40 in ten minutes | Turns, tokens, cost and wall-clock, checked every turn |
| Asking the image model for text | Beautiful posters with gibberish headlines | Imagery from the image model, type from the layout engine, always. Put it in the system prompt and in the tool description. |
| Trusting tool output | A fetched page or an uploaded asset’s metadata steers the agent | Least privilege, approval gates on consequential tools, egress allowlists. Never one agent with both secrets and an outbound channel. |
| Multi-agent too early | Five agents, 5× the cost, no measurable improvement, five times the debugging | One loop until an eval says otherwise. §06. |
| Evaluating by using it | “Feels better” after a prompt change; a regression ships | A golden set with hard gates on the deterministic checks. §07. |
Every good decision in this document is the same decision: push work out of the model and into deterministic code, and let the model do only the part that genuinely requires judgement. The renderer is deterministic. The checks are deterministic. The patch applier is deterministic. The grid maths, the contrast maths, the DPI maths, the type scale — all deterministic. What is left for the model is choosing a direction, choosing what to change, and looking at the result — which is exactly what it is good at, and exactly what your code cannot do.
Glossary
- Agent
- A loop that calls a model, executes any tools it requested, appends the results, and repeats until it stops asking or a budget trips.
- Context assembly
- Building the exact message list for this turn. The main tuning surface of an agent system.
- Context rot
- The observed decline in quality as an input grows, well before the hard window limit. Attention is finite and diluted.
- Elicitation
- An MCP server asking for human input mid-call. Since 2026-07-28 it is delivered through MRTR rather than a server-initiated request.
- Golden set
- A fixed, versioned collection of real inputs used to detect regressions. Assert properties, never exact outputs.
- Handle
- An opaque server-minted id returned by one tool and passed to the next. How stateful work happens now that MCP is stateless.
- Harness
- Everything around the loop: budgets, retries, timeouts, approvals, persistence, streaming, tracing, stop conditions.
- MRTR
- Multi Round-Trip Requests. A server returns
resultType: "input_required"; the client gathers the inputs and retries the original request withinputResponses. - Projection
- A compact, context-sized view of a large piece of state. The agent reads the projection; the state stays in the store.
- Prompt caching
- Provider-side reuse of a repeated input prefix, charged at a fraction of the normal rate. Requires a byte-identical prefix.
- Prompt injection
- Instructions smuggled into a model’s context through tool results or fetched content. Not solvable by prompting; mitigated structurally.
- Streamable HTTP
- MCP’s remote transport. The older HTTP+SSE transport is deprecated.
- Structured output
- Schema-constrained generation via
response_format: json_schema. Distinct from MCP’sstructuredContent, which is server-produced result data. - Tool
- A JSON Schema shown to the model plus a function your code runs when the model asks. The model never executes anything itself.