A 40B MoE at 120k context on a laptop with 6 GB of VRAM
How I got a 40B MoE model (2.5B active) running at 120k context on a laptop with a 6 GB RTX 3050, which settings mattered, and what didn't work.
22 tok/s
generation at 32k on 6 GB
How I got a 40B MoE model (2.5B active) running at 120k context on a laptop with a 6 GB RTX 3050, which settings mattered, and what didn't work.