Run a model on your own machine

The cheap-hosted-model trick from the last lesson still bills per token, even if the bill is small. There’s a cheaper corner than that: a model running on hardware you already own, answering for the cost of electricity. Pi supports this the same way it supports every other provider - through models.json - because local inference servers like Ollama speak (approximately) the same OpenAI-compatible API shape as everything else Pi already talks to.

Pointing Pi at a local model

The mechanism is the same file you were just in, with an entry for the endpoint your local server exposes and a compat block for the parts of the OpenAI API shape it doesn’t fully implement:

~/.pi/agent/models.json

{
  "providers": {
    "local-ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "api": "openai-completions",
      "apiKey": "not-needed",
      "models": [
        { "id": "qwen3.5-coder", "name": "Qwen3.5 (local)", "contextWindow": 32000 }
      ],
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false,
        "supportsUsageInStreaming": false
      }
    }
  }
}

compat exists precisely because self-hosted servers - Ollama, LM Studio, vLLM - only partially implement the OpenAI shape; without it Pi will assume capabilities the local server doesn’t actually have and the session will misbehave in small, confusing ways. Set contextWindow honestly too, because the next section is about why “honestly” matters more than it sounds like it should.

The speed facts worth knowing before you commit to this

If you’re running Apple Silicon, format can matter as much as model choice: MLX and GGUF builds use different runtimes and can perform very differently on the same hardware. There is no universal multiplier, so benchmark the exact model, quantization, runtime, and workload before treating one format as faster. If an MLX build is available, it is worth testing before settling for the first GGUF download.

Three numbers actually matter when you’re judging whether a local setup is usable, not just “working”:

• Prefill - time to ingest the incoming prompt.
• Decode - tokens/second once it’s generating, the number everyone quotes.
• Wall-clock - the number that actually matters: cold start plus prefill plus decode plus everything else, end to end.

As a usable-speed heuristic: in our testing, there’s a real split between a decode speed that’s fully usable for interactive work and one slow enough that the agentic loop stalls - technically running, practically unusable while you sit there waiting on it. Where that line falls depends on your hardware and the model; benchmark your own setup rather than assuming a number from someone else’s rig.

The real ceiling: context, not raw intelligence

This is the fact that should actually change how you use a local model on stash. An advertised context window is a capability ceiling, not a guarantee of useful speed or accuracy. As prompts grow, prefill time, memory pressure, and answer quality depend on the exact model, quantization, runtime, and hardware. Benchmark your own model and hardware rather than planning around a fixed fraction of the spec-sheet number.

Hold that against how fast an agentic session actually fills up: a real coding session commonly crosses 30,000 tokens of context within just two or three prompts once you’ve read a few files and run a few commands. That’s not a local-model-specific problem - it’s the same context-window math this whole course keeps returning to - but it means a local model’s usable ceiling arrives almost immediately in real use, not after an afternoon of work. Point Pi’s local model at the extractor redesign from the last lesson - the one with several files and a genuine design decision to reason through - and you’ll likely blow past the honest ceiling before it’s finished reasoning. Point it at re-extracting one saved URL at a time - read the row, fetch the page, re-run the extractor, write the result, done - and you’re well inside where a local model is currently reliable.

A local 7B model may be perfectly usable for micro tasks and painfully slow for multi-file work; throughput varies too much by hardware, quantization, runtime, and prompt length to promise a single tokens-per-second figure. The cost is zero, but the time cost is real.

That’s the honest scope for local models on stash right now: micro agent tasks - small, bounded, low-stakes work, ideally one file or one row at a time - not a wholesale replacement for a frontier model doing multi-file, multi-hour agentic work. RAM is the other hard constraint worth naming plainly: if the model doesn’t fit in memory, it doesn’t run, full stop - that’s a ceiling before you ever get to a speed or accuracy question.

The trade-off, named honestly

People who’ve actually lived with a fully local Pi setup for a while have landed on a name for what you’re signing up for: the spaceship problem. Pi’s minimalism strips out the vendor bloat, which is exactly what makes a bespoke local setup possible - but every piece of reliability a bigger, more opinionated tool would have handled for you (retry behavior, sane defaults for a flaky local server, a plan mode you might’ve wanted) is now yours to build and maintain. That’s a fair trade for someone treating their harness as a sandbox for developer agency. It’s a bad trade if what you actually wanted was to plug in a model and never think about it again. Go in knowing which one you are.

Local doesn’t need the ToS caution - the temptation it replaces does

None of this touches the Anthropic subscription caution directly - a model on your own machine has no subscription to misuse. But it’s worth naming the connection: local models and cheap hosted models exist for the same reason people reach for “just use my Claude subscription” in the first place - the desire to route grunt work somewhere that doesn’t bill per token. Local is the version of that instinct with no provider subscription policy in the loop.

Where this leaves the module

You’ve now got every lever config gives you: two settings.json files and where each one wins, a model and a thinking level you route per task instead of pinning once, and - for the right kind of small, bounded work - a model that costs nothing per token because it never leaves your machine. That’s real leverage, and it’s still just configuration. Nothing you’ve done in this module taught Pi a new move - a re-extraction command it doesn’t have, a check it doesn’t know how to run. Stash still has chores that no amount of routing solves, because the tool for the job doesn’t exist yet. Time to build one.