If you don't have enough memory for a 4-bit quant, you may want to try 3- or even 2-bit quants, as they may still be usable[1]:Here's another beginners guide, written by people known in the community, for running Qwen3.6-27B and 35B-A3B AI LLM models locally (and therefore privately): unsloth.ai/docs/models/qwen3.6.
That said, the phrasing of this
Quote from: unsloth.ai/docs/models/qwen3.6Qwen3.6-27B runs on 18GB RAM setups and 35B-A3B runs on 22GB.
is unprecise, because:
- If unsloth means that a Qwen3.6-35B 1-bit UD-IQ1_M quant (10 GB) would fit in 22 GB of total memory, including minus 8 GB for the OS itself (22 GB - 8 GB = 14 GB, and a 10 GB file fits within 14 GB of memory), then yes, but a 1-bit quant of an already small-ish 35B-A3B model is not going to be usable. But since it's available and if you are limited to that kind of total memory (RAM+VRAM), you may still want to try it, but expect not much from a 1-bit of a 35B LLM.
- If unsloth means 22GB of freely available memory for a LLM itself and not minus 8 GB for the OS itself, then one can try and see if a 3-bit quant is usable (the 27B dense LLM may quant better than the 35B-A3B), because the mentioned 4-bit UD-Q4_K_XL (22.9 GB) quant will still not even fit, let alone have enough/any memory left for the context.
But in the guide, scroll down to "Unsloth GGUF Benchmarks" and see that even 3- or 2-bit quants may be usable (especially a 3-bit quant of the 27B dense model). The general consensus is that smaller quants of bigger models perform better than bigger quants of smaller models, e.g.: a 3- or even a 2-bit quant of the 27B or 35B models is going to outperform an 8-bit quant of an 8B model [at the time of this writing, we're not talking about major breakthroughs and major architecture differences]). A 2-bit quant of Qwen3.5-35B-A3B MoE model quants worse than a Qwen3.5-27B dense[1] (this is a know thing for MoE models with a low number of active parameters (3B)).
[1] kaitchup.substack.com/p/summary-of-qwen35-gguf-evaluations
Quote from: Running AI locally/privat on July 17, 2026, 11:00:37Current SOTA AI LLM models, for their size, are:
Let me give you a source reference for this claim: reddit.com/r/LocalLLaMA/comments/1v9xsi8/comment/p0hgky2/ (-> artificialanalysis.ai/models/open-source#intelligence-index-vs-total-parameters).