Local LLMs on Strix Halo: new models, same limits

Same hardware, updated software

The machine is still a Framework Desktop with an AMD Ryzen AI Max+ 395, Radeon 8060S integrated graphics, and 128 GB of shared memory.

Inference runs through llama.cpp with Vulkan and KHR cooperative matrices. Before testing, I updated llama.cpp to commit f5222f25c and built it in Release mode. The graphics driver is Mesa RADV 26.2.5-arch1.1.

One correction to my initial memory assumptions: I was planning around a 96 GB GPU budget. The driver actually exposes 111.56 GiB to Vulkan, with about 109.84 GiB free at the start. No strict 96 GB cap was enforced. This is shared system memory, not a separate bank of dedicated VRAM.

The distinction matters when discussing whether a model fits. A GGUF file’s size is not its total runtime memory requirement. Caches, working buffers, and host-side allocations also need room.

Same benchmark protocol

I used llama-bench, keeping the original prompt size, generation length, thread count, and GPU offload setting:

llama-bench -m MODEL.gguf -p 512 -n 128 -t 6 -ngl 999 -r 5 -o json -v

• pp512 measures processing a 512-token prompt.
• tg128 measures generating 128 tokens.
• 6 CPU threads, maximum GPU layer offload.
• Five repetitions per measurement, with warmup enabled.
• Models run sequentially, not in parallel. Other llama.cpp defaults are preserved, including automatic flash attention selection.

These are separate prefill and generation tests, not generation after consuming a 512-token prompt. They measure short-context engine throughput, excluding tokenization, sampling, model loading, and application overhead. No speculative decoding or vision projector is used.

This is not a controlled year-on-year comparison. I did not rerun the old models on the new stack. Both the models and the software changed, so differences from last year’s table cannot be attributed to either one separately.

Results

All nine variants completed. The logs confirm that every offloadable layer was placed on the GPU, although some host-side buffers remain. GPU layer placement also does not mean every operation runs on the GPU.

Sizes below are GGUF file sizes in GiB, summed across shards, not measured peak memory usage. Throughput is the mean ± one standard deviation, in tokens per second. The quantization names are the exact published variants, including Unsloth’s UD labels.

Mistral Small 4 has the best throughput here

Mistral Small 4 has 119 billion total parameters, but it is a mixture-of-experts model with about 6.5 billion active per token.

That helps explain why it generates faster than the smaller dense models. The full weights need to fit in memory, but they are not all used for every token. Parameter count and file size alone are not enough to predict throughput.

At Q4, it leads both prefill and decode in this set. Moving to Q5 increases the file size to 83.04 GiB and costs about 9% of decode throughput. It still produces almost 30 tokens per second.

This makes it a reasonable candidate for a local workflow. These measurements do not tell me whether it produces better answers than the others.

Lower precision helps decode, not necessarily prefill

In the previous article, I described quantization as a straight performance multiplier. That was too broad.

For generation, the pattern is clear. Qwen goes from 7.50 tok/s at Q8 to 12.82 at Q4, about 71% faster. Gemma goes from 6.46 to 11.27, about 74% faster. Lower precision reduces the amount of weight data to move, which is useful on this shared-memory APU.

For prefill, Q8 is faster than both Q4 and Q6 for both models.

Lower precision does not make every stage faster. Prefill and decode exercise the hardware differently, and the implementation of each quantization format matters. I did not profile the kernels to identify the exact cause here.

For short-context generation speed, Q4 is the useful baseline here. Whether Q6 or Q8 is worth the slowdown depends on quality differences this run did not measure.

GLM-5.3-Flash runs at a very low quantization

GLM-5.3-Flash is a 320B-total-parameter MoE model, with 18B active per token. To run it within the intended memory budget, I used the aggressively quantized UD-IQ1_S version, an 86.69 GiB download.

It loaded successfully and generated at 19.02 tok/s. Its logs reported about 83.76 GiB of Vulkan model buffers, plus roughly 0.48 GiB of Vulkan cache, recurrent-state, and compute buffers during prefill. Those are allocation figures, not a measured total memory peak or proof of a strict 96 GB fit.

The runtime also disabled an unsupported fused top-k-mask optimization. The test still completed with all 47 offloadable layers on the GPU.

This is not a quality benchmark. I did not evaluate answers, reasoning, coding success, or tool use. In particular, I cannot assume that this heavily compressed version retains the capabilities reported for the original model.

What did not make the table

My initial list also included DeepSeek V4.1 Flash, GLM-5.3, and Kimi K3. I left them out because the quantized variants checked did not fit the intended GPU-resident budget.

DeepSeek’s Engram lookup tables can remain disk-mapped, so total download size alone is misleading. But even the smallest GSQ-RCO build checked has a 120.72 GB backbone, before its separate Engram tables and runtime allocations.

CPU offload and SSD streaming would require a different experiment. I kept the original GPU-offload protocol.

The best open models are still behind the frontier

For quality, I need to look at independent evaluations rather than my tokens-per-second measurements. Two sources are useful here: Artificial Analysis and Epoch AI.

In its October 27 release of Intelligence Index v5, Artificial Analysis reports that Anthropic and OpenAI hold the top five model positions. The index combines evaluations of coding, knowledge work, science, and general capabilities. Sixty percent of its weight now comes from evaluations with private test sets or held-out answers. Its methodology explains the evaluation conditions and scoring.

Some results from that release, with the reasoning settings reported by Artificial Analysis:

Kimi K3.1 Pro scores 46 on the aggregate index, but its weights had not been released at the time of that report. It also leads Terminal-Bench 4.0, so this is not a claim that proprietary models win every individual benchmark. They do hold the leading positions overall.

Artificial Analysis describes a larger open-versus-closed gap in v5 than in v4.3. The evaluation mix changed, so that is not evidence that open models suddenly became worse. On this revised evaluation, the gap is larger.

Epoch AI’s analysis by Jack Edwards and Luke Emberson, published May 29, found that the best open-weight models lagged the closed frontier by an average of four months on the Epoch Capabilities Index since January 2026. Their earlier analysis had estimated three months over January 2023 to October 2025. That is a historical estimate, not a guarantee that any particular capability will become available locally four months later. Epoch’s benchmark hub, updated October 28, also identifies Claude Opus 5.5 as the current ECI leader.

Figure: Epoch AI, Jack Edwards and Luke Emberson, “Open models lag state-of-the-art closed models by 4 months”, May 29, 2026. Reproduced without changes under CC BY 4.0. This is the historical chart from that publication, not an October leaderboard.

I use “open-weight” here because it describes access to the model weights. It does not necessarily mean the training code, data, and full development process are open source.

These comparisons already concern the strongest open models. My desktop is running smaller members of model families, or heavily compressed versions, rather than the largest and most capable releases. Qwen3.8-27B and Gemma 4 31B are medium-sized dense models. Mistral Small 4 and GLM-5.3-Flash have larger total parameter counts, but they are still the smaller options in their respective families. GLM also needs a particularly low quantization for this experiment.

Quantization is another constraint, not a way to recover the capability difference. Q4, Q6, and Q8 do not all have the same effect, and I have not measured the quality loss for these exact files. But there is no reason to treat their speeds as evidence of frontier-level intelligence.

Same conclusion as last year

I still see a place for local models in constrained workflows: extracting known fields from documents, classifying text, or handling a narrow internal assistant with limited tools and output validation. Running offline or keeping data on the machine can be more important than getting the strongest possible model. Those are reasons to evaluate a local model on a specific task, not claims that these nine variants have passed such an evaluation.

For difficult software engineering, unfamiliar problems, and tasks requiring the best available reasoning, I would use a frontier model. The independent quality results support that choice, and the models that fit this desktop are further constrained by size and quantization.

This is consistent with how I choose models for development work: use the strongest model when the task needs judgment, and consider cheaper or local options when the work is well defined and the result can be checked.

Newer local models run at useful speeds on this hardware. That does not change my answer to the original question: local AI on a consumer desktop is not a replacement for frontier intelligence today. We do not know whether or when it will reach that level relative to the frontier available at the time. I would not plan around that happening.

Downloads and reproducibility

• Measurements, individual repetition speeds, and model revisions (JSON)
• Results table for spreadsheets (CSV)
• Exact GGUF repositories: Qwen3.8-27B, Mistral Small 4, Gemma 4 31B IT, and GLM-5.3-Flash.

For split GGUFs, download every shard and pass the first one to -m. Use the revisions in the JSON rather than assuming a repository’s current files are identical to the ones tested here.