In August 2026 I sketched what it would take to run my AI work at home. The plan had two tiers: a fast GPU for the high-volume background jobs, and a big-memory machine for the long conversation. Over 37 days, 96.6% of the tokens my setup processed were old context being read again. New output was about 0.2%. That makes long context a memory problem before it is a speed problem.
It was a plan, not a purchase. And the same day, the arithmetic argued with half of it.
What I wanted from a local box
Room for more background jobs running at the same time, without paying per token for each one. Spare GPU time for generating video in the gaps between those jobs. And my data staying in the house.
The hard part was one long-running conversation: the main thread my household assistant worked in, which had grown toward a million tokens of context. I wanted to know whether that could run entirely on my own hardware, with other machines picking up the other jobs.
The plan: one tier for volume, one for memory
- Fast tier. One 96 GB workstation GPU (an RTX PRO 6000 Blackwell Max-Q, on paper). Its jobs: digests, summaries, classification, short-context agents and video generation, where speed is what counts.
- Memory tier. A Mac Studio with 512 GB or more of unified memory, for the one huge conversation. I picked it for capacity rather than speed.
This card has no NVLink, so a second one would talk over PCIe, which is too slow for splitting one model across cards. Doubling to 192 GB would not have unlocked a better tier of open models either.
What the workload analysis found
Counted by runs, most of my AI work looked local-friendly. Somewhere between 60% and 85% of runs were things a local 35B-class model could plausibly attempt. The weight sat in a small number of long-context, high-stakes sessions.
The workload analysis sent trivial and moderate work to a local model kept loaded, and kept anything where a mistake has real consequences on a hosted frontier model.
The moderate tier came with a known cost. Long prompts with many layered rules are where open models are weakest. They rarely break the format; they quietly drop one rule from a long list. That is survivable in a digest, and unacceptable in anything another person reads.
The same reasoning picked the resident model. A 35B-class model at 4-bit is about 19.5 GB, which leaves 76.5 GB of the card for context. A 122B-class model at 72 GB leaves 24 GB. The small model won because the context needed the room.
Holding a long context is a memory problem
A model keeps a working copy of everything it has read so far, called the KV cache, so it doesn't redo that work on every turn. Its size is simple arithmetic:
bytes per token = 2 × layers × KV heads × head size × bytes per number
For a common 30B-class layout (48 layers, 4 KV heads, 8-bit cache), that comes to 48 KiB per token, or 45.78 GiB for a million tokens. It has to sit in fast memory for the whole conversation.
My next question was whether this meant saving cache every day, forever. It doesn't. The cache is derived from the conversation text and can be rebuilt from it. The text of a session that size is about 4 MB, roughly 1.5 GB for a year of daily sessions. At most you keep a fixed-size rolling cache on disk.
The arithmetic argued with my tier assignment
The model's design moves the number more than the hardware does. That formula measures how a model's attention is built, and nothing about how good it is. Kimi-Linear-48B-A3B has 27 layers, and only 7 of them keep a growing cache. That puts a million tokens at about 7.5 GiB at 16-bit, roughly six times smaller than the 45.78 GiB layout. At that size, a million-token conversation would fit on the fast tier's card with room to spare.
So the research put long context on the GPU, with a hybrid-attention model and a tiered cache that spills to system memory and then to SSD, and treated the Mac as a second worker rather than the home of the big conversation. That was the recommendation on paper.
Where I was the bottleneck
My first question was a shopping question: would this GPU plus that Mac work? The bigger lever turned out to be which model, and how its attention is built.
The research also questioned the million-token target itself. Across 93 requests in the main conversation, the median was 60,588 tokens. Half of them already fit in 64k. The million was the tail, driven by catching up after a long gap.
Two questions were still open. Nobody had published prefill speed for this kind of model on this card, and that one number could flip the whole design. And holding a million tokens says nothing about whether a model with 3B active parameters is smart enough for the job. Public write-ups on local models with agent tools agreed: good for routine work, weak at tool calls and long multi-step tasks. Only a test against real transcripts would settle it.
If you are about to try this
- Measure your token mix before you price hardware. If most of your tokens are context being re-read, you are shopping for memory first.
- Do the KV arithmetic for the exact model you plan to run. Attention design can change the answer by six times or more.
- Route by difficulty. Keep anything with real consequences on the strongest model you have, and send the bulk work to the cheap tier.
- Don't count on pooling. Check the interconnect before you plan on two cards or two machines acting as one.
- Rent before you buy. An hour on a rented card measures prefill, the number nobody had published for my case.
Build it yourself
What it does: runs AI work on your own hardware. A loaded mid-size model handles the frequent, easy jobs. Long conversations get a model and cache sized for memory. Hard, high-stakes work goes to a hosted frontier model.
The parts: a GPU with as much memory as you can justify, or a large unified-memory desktop. A serving layer such as vLLM, llama.cpp or MLX. Optionally a tiered KV cache such as LMCache. A routing config that sends each job to a tier.
How it fits: each job carries a difficulty label that picks its tier. The long conversation keeps a warm cache, and its plain-text transcript means the cache can always be rebuilt.
Cost: I've left prices out. Price your own build on the day.
If you have run long context on your own hardware, I'd like to hear what you measured.