SocialistVibes01@lemmy.ml to Linux@lemmy.mlEnglish · 2 days agoWhich specs are as low as reasonable possible for local LLM models? Do you recommend some distro in particular?message-squaremessage-square14linkfedilinkarrow-up131arrow-down112file-text
arrow-up119arrow-down1message-squareWhich specs are as low as reasonable possible for local LLM models? Do you recommend some distro in particular?SocialistVibes01@lemmy.ml to Linux@lemmy.mlEnglish · 2 days agomessage-square14linkfedilinkfile-text
minus-squareinfinitevalence@discuss.onlinelinkfedilinkEnglisharrow-up1arrow-down1·2 days agoI disagree Gemma 4 easily runs inside of a 16g GPU and is really pretty fast.
minus-squaremeowmeow@quokk.aulinkfedilinkEnglisharrow-up2·2 days agoFast is relative. I’m also commenting on the cost of the entire system, not just the gpu, fyi
minus-squareinfinitevalence@discuss.onlinelinkfedilinkEnglisharrow-up4arrow-down1·2 days agoThat’s fair, but nearly any modern CPU at least 32gb of RAM and a current GPU with 16gb is plenty. No need for a 4k system when a 1k-1.5k will do it. If you’re willing to Frankenstein things some of the used AI/ML/mining cards can be a decent value.
minus-squaremeowmeow@quokk.aulinkfedilinkEnglisharrow-up2arrow-down1·2 days agoYes, but when you compare it to codex and Claude though, it’s significantly slower. Especially over time. Better crank that AC. I think in a few years we will have current cloud levels running pretty efficiently on current computers.
I disagree Gemma 4 easily runs inside of a 16g GPU and is really pretty fast.
Fast is relative. I’m also commenting on the cost of the entire system, not just the gpu, fyi
That’s fair, but nearly any modern CPU at least 32gb of RAM and a current GPU with 16gb is plenty. No need for a 4k system when a 1k-1.5k will do it.
If you’re willing to Frankenstein things some of the used AI/ML/mining cards can be a decent value.
Yes, but when you compare it to codex and Claude though, it’s significantly slower. Especially over time. Better crank that AC.
I think in a few years we will have current cloud levels running pretty efficiently on current computers.