Run Qwen3.8-Flash-Next on 12 GB VRAM + 64 GB RAM

[AI beginners guide]

Qwen3.8-Flash-Next (Qwen4 preview) is a 125B parameters LLM model, with only 6B activated per token, plus 51B n-gram embedding and 4B MTP.

Inspired by reddit.com/r/LocalLLaMA/comments/1wgiefk/running_qwen38flashnext_locally_on_a_12gb_vram, I finally decided to make this post to point out the most important thing: llama.cpp runs the 51B n-gram embedding from SSD by default, VRAM and RAM are not used/consumed to store it and the tokens per second performance is unaffected!

I noticed this when I loaded huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-Q4_K_XL, because my total RAM consumption only increased from 10 GB to 87 GB. If I disable the n-gram embedding offloading to SSD (llama.cpp command: --lazy-mode off), then the RAM consumption increases to 116 GB, but despite running from the much faster RAM now vs SSD by default, the tokens per second speed does not increase (maybe from 18 to 19 t/s?, but it would need more testing to be sure, but obviously not worth further testing, because the additional available 30-35 GB RAM can be used for more important things now, like using a bigger quant or more KV cache context).

(llama-bench seems to ignore --lazy_mode (bug?) and interestingly, is also much slower vs llama-server / llama-cli (8 vs 16-19 t/s). So I can't test the 18 to 19 t/s speed claims more precisely for now (and, as said, it's not important).)

So, if you see e.g. huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-Q3_K_XL and its 90 GB file size, remember, it's not 90 GB for RAM+VRAM, but 90 GB minus 30-35 GB of the n-gram embedding that runs from SSD.
Btw. the SSD streaming utilization speed, when running UD-Q4_K_XL, is only 5-30 MB/s (using "nvme top" command, which I discovered was introduced in nvme-cli v3.0).

llama.cpp built config (add -j <number_of_threads> to speed up the compilation): cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release
llama.cpp command used:
llama-server -m 'Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf' -lm none -np 1 --threads 4 --temp 0.7 --top-k 20 --top-p 0.8 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 -c 32768 --reasoning-preserve --reasoning off --reasoning-effort low --lazy-mode on --no-warmup
(Increasing the --threads to 8 doesn't increase the speed, but with MTP, it probably would (just like it does with Qwen3.8-27B) (waiting until llama.cpp/pull/28243 is merged, building didn't work). --lazy-mode on is the default, so not needed to be specified. When --reasoning is set to off, then --reasoning-effort is ignored, but I let it be there, because sometimes I use it. PS: Change llama-server to llama-cli, if you want to use the command line.)

Reading /r/localllama, Qwen3.8-Flash-Next replaced Qwen3.8-27B for some/many. But, when it comes to both, the prompt processing (input) and token generation (output) speeds, nothing beats being able to fully offload a model (quant) into a GPU's much faster VRAM (a Qwen3.8-27B quant in this case).

See how well Qwen3.8-Flash-Next, Qwen3.8-27B and others score on AA:
artificialanalysis.ai/?models=qwen3-5-122b-a10b-non-reasoning%2Cqwen3-5-122b-a10b%2Cmimo-v2-5-0424%2Cnemotron-3-5-lightning%2Cqwen3-6-27b-non-reasoning%2Cminimax-m2-7%2Cmuse-glimmer%2Cnvidia-nemotron-3-super-120b-a12b%2Cqwen3-6-35b-a3b-non-reasoning%2Cstep-3-7-flash%2Cinkling-small%2Csolar-open2-250b%2Cqwen3-6-35b-a3b%2Cqwen3-6-27b%2Cqwen3-8-27b%2Cmotif-3%2Chy3%2Cg9v3-39a5b%2Cgemma-4-31b%2Cgemma-4-31b-non-reasoning%2Cmistral-medium-3-5%2Cling-3-0-flash%2Cgpt-oss-120b%2Cdeepseek-v4-flash%2Cqwen3-8-flash-next%2Cglm-5-3-flash%2Cqwen3-8-27b-medium&intelligence=artificial-analysis-intelligence-index