Quoteup to 32GB RAM
With only 32 GB of total memory you can fit about this much context for this SOTA AI LLM model[1]:
Quote from: reddit.com/r/LocalLLaMA/comments/1sq94qx/is_anyone_getting_real_coding_work_done_with.. I've come to the conclusion that (1) 32768 is the biggest context I can get away with in an adequately smart model, and (2) it just ain't enough.
In this memory sense, any gaming laptop that has a 6-8 GB VRAM (used for 700-800 bucks) (it's also going to have upgradable RAM) is superior and much cheaper (also it's a whole laptop vs this being only a mini-PC), even if new. The additional 8 GB of memory/VRAM of the GPU make all the difference in being able to fit and run huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/blob/main/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf.
More about running AI models locally: notebookchat.com/index.php?topic=315954.0 ("Your own ChatGPT, offline: AI without the cloud on your laptop") and comments.
[1] artificialanalysis.ai/?models=nemotron-3-5-lightning%2Cmuse-glimmer%2Cqwen3-8-27b-medium%2Cqwen3-6-35b-a3b%2Cqwen3-8-27b%2Cg9v3-39a5b%2Cgemma-4-31b&intelligence=artificial-analysis-intelligence-index#intelligence
(Qwen3.8-27B scores much higher, but due to being a dense model, will run many times slower) (you can look up non-reasoning scores or other models in the table, too)
Calculate available context: 16,384 [KV cache context tokens per GB]*(32 [GB total memory] - 6 to 8 [GB for the OS] - [quant filesize] - 1-2 GB for the window manager / overhead)[3]:
For huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF ->UD-Q4_K_XL.gguf:
~34,400 context tokens = 16,384*(32-6-22.4-1.5) (6 GB for the OS) (confirms the quote)
~ 1,600 context tokens = 16,384*(32-6-22.4-1.5) (8 GB for the OS)
For huggingface.co/unsloth/Qwen3.8-27B-GGUF ->UD-Q4_K_XL.gguf:
~113,000 context tokens = 16,384*(32-6-17.6-1.5) (6 GB for the OS)
~ 80,200 context tokens = 16,384*(32-6-22.4-1.5) (8 GB for the OS)
So, Qwen 27B dense allows for many more context tokens (it will also perform better, but run much slower (3B vs 27B active parameters per token)).
[3] reddit.com/r/Qwen_AI/comments/1vo8pjz/qwen3827b_kv_cache_works_out_to_64_kibtoken_so/
English
Español
Français
Italiano
Nederlands
Polski
Português
Русский
Türkçe
Svenska
Chinese
Magyar