Quote from: Running AI locally/privat on July 17, 2026, 11:30:29Since this may be a big deal
Well, -- and this is why I wrote "may" -- there is this user experience test of Bonsai-Ternary-27B (there's also Bonsai-27B), which is compared to Qwen3.6-27B Q2_K_XL quant (12 GB, so quite a bit bigger than Bonsai-Ternary-27B) and it
Quote from: reddit.com/r/LocalLLaMA/comments/1uz0z0t/user_experience_of_bonsaiternary27b_on_4060ti/.. wins hands down.
So, if you can fit it into your RAM+VRAM, try it.
Trying IQ2_XXS (9.6 GB) and IQ3_XXS (12.2 GB) quants was also suggested in the comments (IQ quants perform better vs same size, but run slower, but at least you'd may be able to fit them or the context you need).
I compiled prism's llama.cpp branch (since I have a NV GPU, with CUDA set to ON), but the webUI didn't work. I may try Bonsai-Ternary-27B later when its support is merged into official llama.cpp branch. In the meantime I suggest you try the mentioned quants (IQ2_XXS (9.6 GB) is the smallest one) (here they all are: huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF).