Chain agents in sequence, then fan a job out in parallel

One spawn call handles one article. The re-extraction backlog is a few hundred of them, saved before you fixed the readability pipeline, all sitting in stash’s database with mangled text and stale search entries. You’re not going to write that spawn call by hand a few hundred times - but you also shouldn’t reach for one shape of multi-agent work without asking which shape the job actually has. There are two, and the difference between them is the same independence test you’d apply to any parallel work: if two pieces finished in a different order, would the answer change?

When the answer is yes: chain them

Some jobs are only expressible as a sequence. Before you touch the backlog at all, you actually want to know which articles need re-extraction - and that’s naturally three steps where each one needs what the last one found: audit which saved URLs still read wrong, re-extract exactly that list, verify the search index picked up the changes. Step two can’t run until step one hands it a list; step three can’t run until step two hands it a result.

Pi has no built-in “pipeline” primitive either - you get there with the same spawn call as before, just looped sequentially instead of fired once, with each step’s prompt built from the previous step’s raw output. The YAML below is a design sketch for a driver you write; Pi does not load this file automatically:

steps:
  - prompt: "Audit saved URLs that still need re-extraction"
  - prompt: "Re-extract the URLs from $INPUT"
  - prompt: "Verify the result from $INPUT"

$INPUT is a plain string substitution - whatever the previous step’s child agent wrote as its final answer, dropped into the next step’s prompt verbatim. $ORIGINAL carries the very first task through the whole chain, so a step three levels deep still knows what it’s ultimately in service of. A driver script walks the steps with a plain for...await loop - spawn, wait for agent_end, capture the output, build the next prompt, spawn again. Nothing about this is more sophisticated than the single spawn call from the last lesson; it’s that same call, three times, each one seeded by the last.

Field	Type	Description
`steps`	array	Ordered list of prompts to run sequentially
`steps[].prompt`	string	The prompt for this step. `$INPUT` = previous step’s output, `$ORIGINAL` = the initial input
`model`	string	Optional model override for the chain
`max_turns`	number	Optional cap enforced by your driver; Pi itself has no `--max-turns` flag

When the answer is no: fan it out

The re-extraction itself doesn’t have that shape at all. Whether post-032 gets fixed before or after post-140 changes nothing - each article’s fix depends only on that article, never on another one’s outcome. Running a few hundred of those sequentially, one after another in a chain, would be correct and slow for no reason. This is the other shape: independent slices, run at once.

Split the backlog into batches - by domain, by save date, by whatever keeps each batch a reasonable size - and spawn one child per batch, all at once. Name the pattern before you build it: fan-out is running the same child-agent prompt over N inputs concurrently, batched to respect your rate limit, and collected with Promise.allSettled rather than Promise.all - deliberately, so one batch crashing or timing out doesn’t cost you the other nineteen batches that finished fine. Partial failure in a fan-out is data you triage, not a crash that discards everything.

This is exactly the kind of glue worth having Pi write instead of typing it by hand. You already have the one primitive from the last lesson - the spawn("pi", [...]) child runner, with its JSONL parsing loop wired to agent_end. Hand Pi a prompt roughly like this: “Take the child-runner function from subagents/spawn.mdx and wrap it in a fan-out helper: split an array of URLs into batches of a given size, spawn one child per batch concurrently, and return each batch’s result - success or failure - without letting one crashed batch cancel the others. Use Promise.allSettled, not Promise.all. Cap concurrency so we’re not spawning more than a handful of pi processes at once.” Read the generated .ts before /reloading it - check that it’s actually built on the child runner you already have rather than reinventing spawn logic, that it batches instead of firing every URL as its own process, and that it really does use allSettled. See Spawn a child Pi and read its JSONL for the child-runner primitive this builds on, and the extension anatomy lesson for the shape any generated .ts file should take.

Toggle the same job between running inline in your own window and fanning it out across children, and watch what actually changes:

Notice what the toggle doesn’t do: the total tokens burned don’t go down. Delegating doesn’t make the work cheaper - it makes it happen off to the side, in windows you never have to sit inside, while your own context stays free for the feature you actually came here to think about. That’s the trade fan-out is for. It’s not a speed trick first; it’s a context-hygiene trick that happens to run faster too, because independent batches genuinely can run at once.

The one control knob that matters here

Concurrency isn’t free. Every child is its own model call, its own token spend, its own slice of your rate limit. Fan out four or five batches and you’ll feel the parallelism; fan out fifty at once against a small backlog and you’re paying spin-up cost on children barely bigger than the request that dispatched them, with no bulk win left to earn back. Start narrow - a handful of batches - and only widen if the batch size itself, not the concurrency ceiling, is what’s slowing you down.

The same instinct applies to which shape you reach for at all. A chain when a later step’s prompt has to be built from an earlier step’s real output; a fan-out when the work is genuinely embarrassingly parallel. Reach for fan-out on a chain-shaped job and you’re guessing at what a later step needs before an earlier step has told you. Reach for a chain on fan-out-shaped work and you’re waiting in line for no reason. Both patterns are the exact same spawn call from the last lesson - the only thing that changed is the control flow wrapped around it.

The backlog’s cleared, fast, and off your plate. But look at what you actually did: you accepted every child’s own “done, re-extracted, index updated” at face value. Nothing checked whether that was true. Next: build a second agent whose entire job is to not believe the first one.