suoko@feddit.it to LocalLLaMA@sh.itjust.worksEnglish · edit-28 days agoI dropped Qwen and Gemma for a 27B model that runs smoothly on my 8GB graphics card - Bonsai 27Bwww.xda-developers.comexternal-linkmessage-square20linkfedilinkarrow-up148arrow-down17
arrow-up141arrow-down1external-linkI dropped Qwen and Gemma for a 27B model that runs smoothly on my 8GB graphics card - Bonsai 27Bwww.xda-developers.comsuoko@feddit.it to LocalLLaMA@sh.itjust.worksEnglish · edit-28 days agomessage-square20linkfedilink
minus-squareOstfriesentee@feddit.orglinkfedilinkEnglisharrow-up3arrow-down1·9 days agoI got a 35B model that runs at 20 tok/s with a 130K context window on my 8GB card. Can be bumped up to 200K if I want, with marginally slower output speeds.
I got a 35B model that runs at 20 tok/s with a 130K context window on my 8GB card. Can be bumped up to 200K if I want, with marginally slower output speeds.