There's also the world of running open-weight AI LLM models locally and privately.
News related to that: reddit.com/r/LocalLLaMA/top/
Current SOTA AI LLM models, for their size, are:
huggingface.co/Qwen/Qwen3.6-27B (dense architecture) (Downloads last month: 5.0 million)
and
huggingface.co/Qwen/Qwen3.6-35B-A3B (MoE architecture) (Downloads last month: 5.6 million).
Dense vs MoE
In a dense 27B model, all 27B parameters per token are activated vs in the 35B-A3B, where only 3B active are used. Hence, for the same size, a dense model performs much better[2], but the 35B-A3B will running faster.
Popular, bang for the buck, quants are:
huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL (22.9 GB) (good for 80k-120k of context)
and
huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL (17.9 GB).
Required memory to run these quants is: Quant size + 2 GB for 32,000 of context and 8 GB for 128,000 of context (linear)[1].
If you have 32 GB RAM and no 8 GB VRAM GPU: 32 GB RAM - 8 GB for the OS itself = 24 GB free for fitting an AI LLM model. This does not leave you with a lot of memory left for the context. The dense 27B model quant will run slower, but since the quant is smaller than the 35B one, it will allow for greater context.
Quote from: reddit.com/r/LocalLLaMA/comments/1sq94qx/is_anyone_getting_real_coding_work_done_with.. I've come to the conclusion that (1) 32768 is the biggest context I can get away with in an adequately smart model, and (2) it just ain't enough.
But if you have 32 GB RAM and an 8 GB VRAM GPU, then you are much better off, as not only you then have more memory, you can also upload part of the model to the GPU's much faster VRAM.
To run these models you need a so called inference engine. The most popular is llama.cpp. Many wrappers, like TextGen, are using it in the background (I started off with TextGen, before it had a different name). But llama.cpp does have its own nice WebUI.
More open-weight models evaluated: artificialanalysis.ai/?models=qwen3-5-122b-a10b-non-reasoning%2Cqwen3-5-122b-a10b%2Cmimo-v2-5-0424%2Cqwen3-6-27b-non-reasoning%2Cminimax-m2-7%2Cqwen3-6-35b-a3b-non-reasoning%2Cstep-3-7-flash%2Cqwen3-6-35b-a3b%2Cdeepseek-v4-flash-high%2Cqwen3-6-27b%2Cdeepseek-v4-flash-non-reasoning%2Cgpt-oss-120b-low%2Cgemma-4-31b%2Cgemma-4-31b-non-reasoning%2Cdeepseek-v4-flash%2Cmistral-medium-3-5%2Cgpt-oss-120b%2Chy3&intelligence=artificial-analysis-intelligence-index
[1] reddit.com/r/LocalLLaMA/comments/1tvluaj/how_much_vram_needed_for_qwen_36_27b_q8_with_262k
[2] See 27B dense vs 122B-A10B (MoE) evaluation: huggingface.co/Qwen/Qwen3.5-122B-A10B#benchmark-results. The 122B required 4.5 times more memory, but they score about the same.
Deutsch
English
Español
Français
Italiano
Nederlands
Polski
Português
Русский
Türkçe
Svenska
Chinese
Magyar