News:

Willkommen im Notebookcheck.com Forum! Hier können Sie über alle unsere Artikel und allgemein über notebookrelevante Dinge diskutieren. Viel Spass!

Main Menu

Post reply

Other options
Verification:
Please leave this box empty:
Shortcuts: ALT+S post or ALT+P preview

Topic summary

Posted by Running AI locally
 - Today at 14:06:38
And these ~30B parameters models are no joke:
Quote from: reddit.com/r/LocalLLaMA/comments/1x4cm73/once_again_anthropic_doing_free_marketing_for/Once again anthropic doing free marketing for open weight models

Quote from: anthropic.com/research/self-replicating-aiModels of 30B parameters and under can already complete many of the individual steps for replication today, though they make mistakes when composing them end to end. In our simplified evaluations, GLM-5.2 and stronger models are capable of end to end replication. If small models improve at their historical rate, they'll match this capability level by the end of this year. Small weight-based replicators may be possible soon after.
And the top rated comment from that reddit post is:
Quotethey talking about our goat 27b

Communities e.g.:
reddit.com/r/LocalLLAMA/top
reddit.com/r/LocalLLM/top

More about running AI models locally: notebookchat.com/index.php?topic=315954.0 ("Your own ChatGPT, offline: AI without the cloud on your laptop") and comments.
Posted by Running AI locally
 - Today at 14:05:58
Don't want to be tracked or whatever? Did you know that you can run AI locally (and privately)?

Ideally all you need minimally (it even rhymes, so it must be true), is a gaming laptop with 32 GB RAM and a 6-8 GB VRAM GPU and it can then fit a lot of context, too:
Calculating of available context tokens of existing frontier models (Qwen, as of this writing)[1], using an approx. minimum recommended Q4_K_M quants on 32 GB RAM+8 GB VRAM devices (16,384 [KV cache context tokens per GB[2]]*(32 [GB RAM] + 8 [GB VRAM] - 7 [GB for the OS[3]] - [GB quant filesize])):
  • For Qwen3.6-35B-A3B (MoE) (eg huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF):
    178,500 context tokens = 16,384*(32+8-7-22.1)
  • For Qwen3.8-27B (dense) (eg huggingface.co/unsloth/Qwen3.8-27B-GGUF):
    270,000 context tokens = 16,384*(32+8-7-16.5).

(If Qwen4 gets a reduction in KV cache context tokens memory requirements, like in [4], then the calculated context number will further increase.)

[1] artificialanalysis.ai/models/open-source/small
[2] reddit.com/r/Qwen_AI/comments/1vo8pjz/qwen3827b_kv_cache_works_out_to_64_kibtoken_so/
[3] techpowerup.com/353395/windows-11-26h2-update-lowers-ram-consumption
[4] huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash (see "Global KV Cache Per Token (Bytes)" image)

(due to anti-spam system, 2nd part is in next post)
Posted by Redaktion
 - Today at 10:01:41
ChatGPT, Claude, Gemini, Grok and Perplexity pass chat links, titles and sometimes entire prompts to trackers from Google, Meta or TikTok, according to a study of nine AI assistants by researchers in Madrid. Rejecting cookies only helps so much. Which service shares what, and how to clean up shared chats, cookies and your Android advertising ID.

https://www.notebookcheck.net/ChatGPT-Claude-Grok-Your-chats-end-up-with-trackers.1421879.0.html