In August 2026 I sketched what it would take to run my AI work at home. The plan had two tiers: a fast GPU for the high-volume background jobs, and a big-memory machine for the long conversation. Over 37 days, 96.6% of the tokens my setup processed were old context being read again. New output was about 0.2%. That makes long context a memory problem before it is a speed problem.
It was a plan, not a purchase. And the same day, the arithmetic argued with half of it.
Room for more background jobs running at the same time, without paying per token for each one. Spare GPU time for generating video in the gaps between those jobs. And my data staying in the house.
The hard part was one long-running conversation: the main thread my household assistant worked in, which had grown toward a million tokens of context. I wanted to know whether that could run entirely on my own hardware, with other machines picking up the other jobs.
This card has no NVLink, so a second one would talk over PCIe, which is too slow for splitting one model across cards. Doubling to 192 GB would not have unlocked a better tier of open models either.
Counted by runs, most of my AI work looked local-friendly. Somewhere between 60% and 85% of runs were things a local 35B-class model could plausibly attempt. The weight sat in a small number of long-context, high-stakes sessions.
The workload analysis sent trivial and moderate work to a local model kept loaded, and kept anything where a mistake has real consequences on a hosted frontier model.
The moderate tier came with a known cost. Long prompts with many layered rules are where open models are weakest. They rarely break the format; they quietly drop one rule from a long list. That is survivable in a digest, and unacceptable in anything another person reads.
The same reasoning picked the resident model. A 35B-class model at 4-bit is about 19.5 GB, which leaves 76.5 GB of the card for context. A 122B-class model at 72 GB leaves 24 GB. The small model won because the context needed the room.
A model keeps a working copy of everything it has read so far, called the KV cache, so it doesn't redo that work on every turn. Its size is simple arithmetic:
bytes per token = 2 × layers × KV heads × head size × bytes per number
For a common 30B-class layout (48 layers, 4 KV heads, 8-bit cache), that comes to 48 KiB per token, or 45.78 GiB for a million tokens. It has to sit in fast memory for the whole conversation.
My next question was whether this meant saving cache every day, forever. It doesn't. The cache is derived from the conversation text and can be rebuilt from it. The text of a session that size is about 4 MB, roughly 1.5 GB for a year of daily sessions. At most you keep a fixed-size rolling cache on disk.
The model's design moves the number more than the hardware does. That formula measures how a model's attention is built, and nothing about how good it is. Kimi-Linear-48B-A3B has 27 layers, and only 7 of them keep a growing cache. That puts a million tokens at about 7.5 GiB at 16-bit, roughly six times smaller than the 45.78 GiB layout. At that size, a million-token conversation would fit on the fast tier's card with room to spare.
So the research put long context on the GPU, with a hybrid-attention model and a tiered cache that spills to system memory and then to SSD, and treated the Mac as a second worker rather than the home of the big conversation. That was the recommendation on paper.
My first question was a shopping question: would this GPU plus that Mac work? The bigger lever turned out to be which model, and how its attention is built.
The research also questioned the million-token target itself. Across 93 requests in the main conversation, the median was 60,588 tokens. Half of them already fit in 64k. The million was the tail, driven by catching up after a long gap.
Two questions were still open. Nobody had published prefill speed for this kind of model on this card, and that one number could flip the whole design. And holding a million tokens says nothing about whether a model with 3B active parameters is smart enough for the job. Public write-ups on local models with agent tools agreed: good for routine work, weak at tool calls and long multi-step tasks. Only a test against real transcripts would settle it.
What it does: runs AI work on your own hardware. A loaded mid-size model handles the frequent, easy jobs. Long conversations get a model and cache sized for memory. Hard, high-stakes work goes to a hosted frontier model.
The parts: a GPU with as much memory as you can justify, or a large unified-memory desktop. A serving layer such as vLLM, llama.cpp or MLX. Optionally a tiered KV cache such as LMCache. A routing config that sends each job to a tier.
How it fits: each job carries a difficulty label that picks its tier. The long conversation keeps a warm cache, and its plain-text transcript means the cache can always be rebuilt.
Cost: I've left prices out. Price your own build on the day.
If you have run long context on your own hardware, I'd like to hear what you measured.