Local AI on a gaming PC
Making a very large open-weight model useful on the hardware I have, and finding out where the time actually goes.
The problem
Large language models usually need a server. I wanted to know how far I could get running one on my own Windows gaming PC, with an AMD graphics card and a lot of system RAM, and whether it could feel reliable enough to use day to day with tools, code and images.
I run Strata, an open-source engine by another developer that streams parts of a very large mixture-of-experts model between the graphics card and system memory. It isn’t my project. My part is running it on my hardware, measuring it, working with AI coding tools to investigate performance, and writing up what I find.
Benchmarks versus what it feels like
Headline speeds tend to be measured on one long prompt, where fixed costs are spread thin. Real use is mostly shorter requests. When I timed a batch of ordinary requests, most of the wall-clock time wasn’t spent writing the answer at all.
One baseline run on my PC: 26 requests over three balanced passes, October 2026.
Why short prompts are slow
The engine reads a prompt in chunks. On my setup, each chunk carries a large fixed cost, because the parts of the model it needs have to be copied into place first. A 77-token prompt and a 1,024-token prompt took almost the same time. Drag the slider to see what that does to the effective speed.
Dots are measured points; the line interpolates between them. Measured on one machine and configuration. Not a general result.
Inside each chunk, the graphics card spent nearly half its time waiting for data to arrive and only around a tenth doing the actual matrix maths. So the next question isn’t “how do I make the maths faster”, it’s “how do I stop waiting”. That points at memory pinning, copy paths and chunk sizes, which is where the experiments are now.
What I’ve learned so far
- Measure the whole request, not one stage. A faster writing speed barely matters if most of the time goes on reading the prompt.
- Results are tied to the setup. Graphics card, RAM, driver, model size and quantisation all change the picture. I don’t present one machine’s numbers as universal.
- Failures are worth writing down. Not every idea makes things faster. A clear note on why something didn’t work saves more time later than a vague memory of what did.
- AI tools are good at investigation, as long as I check them. They read logs and source code quickly, but every conclusion still needs a measurement behind it.