No GPU, no problem. Hosting your own LLM is infinately more fun than the censored offerings from the big players and works surprisingly well.

[Redaktion]

Whether you are looking to host a D&D campaign, troubleshoot code, generate ideas, create NSFW content, or bypass the guardrails set out by the big players, hosting your own LLM is easy with KoboldCPP, and works surprisingly well, even without a GPU.

https://www.notebookcheck.net/No-GPU-no-problem-Hosting-your-own-LLM-is-infinately-more-fun-than-the-censored-offerings-from-the-big-players-and-works-surprisingly-well.1260784.0.html

[KoboldHenk]

Thanks for covering us!

I do want to add that the speed mentioned in the article is because of the ability and choice to run it fully on the CPU.

I agree that slightly below reading speed is a fair conclusion for the CPU or most integrated graphics users. But if you have more powerful hardware you can get far faster speeds that are on par with the other AI's you are used to. So if you have a dedicated graphics chip or card with enough vram the software will automatically detect it and make use of it. Speeds then vary based on your hardwares ability to fit and run the particular model of your choice.

If there are any questions ill do my best to answer them and help you get running.