Post reply

Name
Email
Subject
Message icon

[quote author=A post about local AI link=msg=764998 date=1782979007]
[b]About hosting AI LLM models locally and privately:[/b]
[quote author=bustaa link=msg=764901 date=1782923086]
Ok so it's basically crap ... but "good" ?!

Lenovo really gets a special treatment here 😉.
[/quote]
Let's quantify this a bit: If one considers this for AI, know:
32 GB RAM - 8 GB for the OS itself = 24 GB. The SOTA MoE for its size AI LLM model, huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL quant (good for 80-120k unquantized context), is 22.9 GB. Context requires additional 2 GB for 32k and 8 GB for 128k context (linear). This means that the quant barely fits and there's pretty much almost or no space left for any context.
[quote author=reddit.com/r/LocalLLaMA/comments/1sq94qx/is_anyone_getting_real_coding_work_done_with].. I've come to the conclusion that (1) 32768 is the biggest context I can get away with in an adequately smart model, and (2) it just ain't enough.[/quote]

Had this laptop/ThinkPad additional 8 GB VRAM dGPU, then it would be able to fit additional 128k[1] of context and have faster prompt processing and token generation speeds as well.

The good thing about this laptop is that its RAM is upgradable, so I would not exclude buying it for AI. But remember, for the same money a gaming laptop has additional 8 GB VRAM, which is kinda very necessary for running the mentioned 27B dense and 35B MoE SOTA models, especially if you need more than 32k of context. And the gaming laptop would probably have a better screen as well (full sRGB coverage, if not full DCI-P3 coverage).

[1] reddit.com/r/LocalLLaMA/comments/1tvluaj/how_much_vram_needed_for_qwen_36_27b_q8_with_262k
[/quote]

Other options

Return to this topic
Don't use smileys

Verification:

Please leave this box empty:

Shortcuts: ALT+S post or ALT+P preview

Topic summary

Posted by A post about local AI

- Today at 09:56:47

About hosting AI LLM models locally and privately:

Quote from: bustaa on Yesterday at 18:24:46Ok so it's basically crap ... but "good" ?!

Lenovo really gets a special treatment here 😉.

Let's quantify this a bit: If one considers this for AI, know:
32 GB RAM - 8 GB for the OS itself = 24 GB. The SOTA MoE for its size AI LLM model, huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL quant (good for 80-120k unquantized context), is 22.9 GB. Context requires additional 2 GB for 32k and 8 GB for 128k context (linear). This means that the quant barely fits and there's pretty much almost or no space left for any context.

Quote from: reddit.com/r/LocalLLaMA/comments/1sq94qx/is_anyone_getting_real_coding_work_done_with.. I've come to the conclusion that (1) 32768 is the biggest context I can get away with in an adequately smart model, and (2) it just ain't enough.

The dense huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL quant (17.9 GB) would run much slower (because it's dense 27B = 27B active parameters per token vs 3B active for the 35B model), but at least it does somewhat fit with some context.

Had this laptop/ThinkPad additional 8 GB VRAM dGPU, then it would be able to fit additional 128k[1] of context and have faster prompt processing and token generation speeds as well.

The good thing about this laptop is that its RAM is upgradable, so I would not exclude buying it for AI. But remember, for the same money a gaming laptop has additional 8 GB VRAM, which is kinda very necessary for running the mentioned 27B dense and 35B MoE SOTA models, especially if you need more than 32k of context. And the gaming laptop would probably have a better screen as well (full sRGB coverage, if not full DCI-P3 coverage).

[1] reddit.com/r/LocalLLaMA/comments/1tvluaj/how_much_vram_needed_for_qwen_36_27b_q8_with_262k

Posted by bustaa

- Yesterday at 18:24:46

Ok so it's basically crap ... but "good" ?!

Lenovo really gets a special treatment here 😉.

Posted by Redaktion

- Yesterday at 17:44:48

A laptop north of €1,000 is hardly a budget system, but in the year 2026, it may be considered so - especially if it offers 32 GB of memory. The Lenovo ThinkBook 14 G9 IPL is a laptop with a good amount of RAM that many people can still afford, a rare sight in the times of the memory crisis.

https://www.notebookcheck.net/32-GB-DDR5-RAM-and-affordable-Lenovo-ThinkBook-14-G9-IPL-laptop-review.1327762.0.html