If this did not have the 12 GB VRAM GPU, one could not run SOTA model huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/blob/main/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf with a good amount of context tokens:
Quote from: reddit.com/r/LocalLLaMA/comments/1sq94qx/is_anyone_getting_real_coding_work_done_with.. I've come to the conclusion that (1) 32768 is the biggest context I can get away with in an adequately smart model, and (2) it just ain't enough.
But since it does have the additional 12 GB of (fast) VRAM memory, this is how much context tokens one could fit on it[1]:
~231,000 = 16,384*(32GB RAM + 12GB VRAM - 6GB for the OS - 22.4GB for 35B-A3B-UD-Q4_K_XL - 1.5GB for the window manager)
(or ~198,000 context tokens when giving the OS 8 GB, instead of 6 GB)
Words = context tokens * 0.75, so about 173,000 words, which is pretty good.
There is also Qwen3.8-27B, which scores higher on AA[2], but it will run multiple times slower (use it if you care about quality and can wait much longer). Since it's also smaller than the 35B model, you will have more context tokens (in the formula use 17.9 GB (since the file Qwen3.6-27B-UD-Q4_K_XL.gguf is 17.9 GB), instead of 22.4 GB).
[1] reddit.com/r/Qwen_AI/comments/1vo8pjz/qwen3827b_kv_cache_works_out_to_64_kibtoken_so/
[2] artificialanalysis.ai/?models=qwen3-8-27b%2Cqwen3-8-27b-medium%2Cqwen3-8-27b-low%2Cqwen3-8-27b-non-reasoning%2Cqwen3-6-35b-a3b%2Cqwen3-6-35b-a3b-non-reasoning&intelligence=artificial-analysis-intelligence-index