16 GB unified memory / RAM can't fit SOTA[1] AI general purpose model Qwen3.8-27B (or any other model in that size, like the upcoming Qwen4-27B[2]) in the minimum usable Q4_K_M quant (e.g. huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_M.gguf (to run it use e.g. llama.cpp (it has a nice webUI) or e.g. Unsloth Desktop, that also uses llama.cpp as its backend and both are open source)).
And these ~30B parameters models are no joke:
Quote from: reddit.com/r/LocalLLaMA/comments/1x4cm73/once_again_anthropic_doing_free_marketing_for/Once again anthropic doing free marketing for open weight models
Models of 30B parameters and under can already complete many of the individual steps for replication today, though they make mistakes when composing them end to end. In our simplified evaluations, GLM-5.2 and stronger models are capable of end to end replication. If small models improve at their historical rate, they'll match this capability level by the end of this year. Small weight-based replicators may be possible soon after.
And the top rated comment from that reddit post is:
Quotethey talking about our goat 27b
Yes, that 27B goat that all these 16 GB unified memory/RAM devices can't fit/run.
(and reddit.com/r/LocalLLaMA/comments/1x4ii8e/anthropic_is_doing_free_publicity_for_glm_again)
And this is how far 32 GB unified memory / RAM devices go:
- For a 35B-A3B MoE model:
Quote from: reddit.com/r/LocalLLaMA/comments/1sq94qx/is_anyone_getting_real_coding_work_done_with.. I've come to the conclusion that (1) 32768 is the biggest context I can get away with in an adequately smart model, and (2) it just ain't enough.
It's also possible to calculate the available context[4]: 16,384 [KV cache context tokens per GB]*(32 [GB total memory] - 6 to 8 [GB for the OS] - [GB quant filesize]):
~34,400 context tokens = 16,384*(32-7[5]-22.9) (confirms the quote)
~18,600 context tokens = 16,384*(32-8-22.9) (1 GB more for the OS, this is how much every GB matters)
- For the mentioned 27B dense model:
139,000 context tokens = 16,384*(32-7-16.5).
So, in this memory sense, any gaming laptop that has the additional 6-8 GB VRAM GPU is superior (700-800 bucks for a 16 GB RAM config, then just upgrade to 32 GB RAM, as it also will have upgradable RAM). The additional 8 GB of (fast) memory/VRAM of the GPU makes all the difference in being able to fit and run these general purpose 27-35B SOTA AI models at a useful context size.
If Qwen4-27B will also come as a MoE variant (and maybe replace Qwen3.6-35B-A3B MoE) and not only as a dense variant, then it will run multiple times faster on all these 32 GB unified memory devices/laptops, with the calculated almost 140,000 context tokens.
[1] artificialanalysis.ai/models/open-source/small?models=qwen3-8-27b-medium%2Cqwen3-8-27b-low%2Ck2-horizon-mova-36b-a4b%2Cg9v3-39a5b%2Ck2-horizon-7b%2Cqwen3-8-27b-non-reasoning%2Cqwen3-6-35b-a3b%2Cmuse-glimmer%2Cgemma-4-26b-a4b%2Cqwen3-6-35b-a3b-non-reasoning%2Cgemma-4-31b%2Cqwen3-8-27b#artificial-analysis-intelligence-index (K2 Horizon has expensive KV cache and may reason above average[3])
[2] reddit.com/r/LocalLLaMA/comments/1wmxfjs/qwen_4_announced_at_apsara_conference/
[3] reddit.com/r/LocalLLaMA/comments/1wg0vqz/comment/p9qj8c8
[4] reddit.com/r/Qwen_AI/comments/1vo8pjz/qwen3827b_kv_cache_works_out_to_64_kibtoken_so/
[5] techpowerup.com/353395/windows-11-26h2-update-lowers-ram-consumption[/quote]
English
Español
Français
Italiano
Nederlands
Polski
Português
Русский
Türkçe
Svenska
Chinese
Magyar