Posted by AI per buck
- Today at 16:11:05
Since this review has an "AI" tag and an "AI Performance" category: If you really care about running AI locally (and privately, btw), then know that for the 5400 bucks one can get a much more capable AI desktop (gaming) PC:
1. You want speed: A LLM must fit into the GPU's VRAM, if it's a 27B dense, or similar, one. With each LLM layer offloaded to the much slower system RAM, the speed decreases exponentially.
2. This laptop's 16 GB VRAM will barely fit a usable Qwen3.8-27B dense model quant + context (Windows requires 1-2 GB VRAM for itself). You can partially offload it to system RAM, but it will run much slower.
3. Qwen3.6-35B-A3B MoE model is slightly bigger than the 27B dense one, but due to its only 3B active parameters per token nature, runs much faster than the 27B dense model, even when partially offloaded to system RAM. This is where you'd be still good getting this laptop (the weight or size per FPS makes this laptop very expensive, tho (you may want to read the comments of previous gen)) and run this or other MoE models in a similar class size. But if you want to run the more capable[4] Qwen3.8-27B or Qwen3.8-Flash-Next at a good speed, luckily these build suggestions on consumer hardware exist:
Single GPU solution:
Build a desktop PC around a RTX 5090 (32 GB VRAM at 1.8 TB/s).
Since it's a 575W TDP GPU, you can power limit the TDP to 400W and retain most of its performance (the power efficiency rises when you set the TDP to 400W). I saw a 5090 300W BIOS news (I think I saw it on /r/localllama some time ago), but I can't find it on a quick. This is something you obviously don't want to do straight away and see if you can get by with the 400W setting first. Heavy undervolting should get you around the 350-300W. Here someone undervolted the 5090 to around 200W[2] ("Forbidden 5090 Max-Q") (note, not every card may be able to go this low).
A RTX 3090 and 4090 (1 TB/s ~ 1008 GB/s) also exist, but even their 24 GB VRAM is quite limited. But, depending on your workflow / how much context you need / quant quality, it sure can be an option. Approx. calculation: (24 GB VRAM - 16.5 GB UD-Q4_K_M quant) * 16,000 tokens per GB = 120,000 context tokens, and e.g.: [3].
Dual GPU solution:
Get a motherboard that supports splitting the main PCIe 5.0 x16 into x8/x8[1] (= built-in PCIe 5 bifurcation) and get 2 RTX 5060 Ti 16 GB cards (180W TDP per card), and use tensor parallelism (but note that the GPU architectures may/must be the same) and the bandwidth will effectively double to 448 GB/s * 2 = 896 GB/s. VRAM will double too, of course, to 32 GB VRAM. This is why the prices of 5060 Ti GPUs have been going up recently.
You can combine any 2 cards as long as they fit, because some x8/x8 motherboards won't fit a 4-slot GPU into their first x8 slot.
If you go with the dual 5060 Ti solution or 1 RTX 3090, you will have enough money left to get 2 * 64 GB DDR5 RAM for around 2000 bucks [as of this writing]. Then you could easily fit huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF -> UD-Q4_K_XL (111 GB) (128 GB RAM + 32 GB VRAM or + 24 GB VRAM - 8 GB for the OS itself = 144 to 152 GB of memory).
(Workstation GPUs (NV's RTX PRO cards, unlike AMD's ones, use GDDR7 and offer a higher memory bandwidth) provide an even higher GB of VRAM per Watt TDP, but they are expensive and kinda outside of this scope and may be even unnecessary since power limiting and undervolting exists. For a company build, you probably should use ECC RAM + RTX PRO's ECC VRAM and then prices are much less of an issue.)
Software:
English
Español
Français
Italiano
Nederlands
Polski
Português
Русский
Türkçe
Svenska
Chinese
Magyar