I changed two settings in my local AI setup, and stopped needing more VRAM

Published Oct 31, 2026, 2:30 PM EDT Abhinav pivoted from a career in banking to pursue his first love in writing. Even while working full-time, he continued contributing as an editor-at-large, a role he has held for more than 7 years. A lifelong tech enthusiast who has built three gaming and productivity powerhouse PCs since 2018, his passion for technology keeps him closely following the semiconductor industry, from NVIDIA and AMD to ARM. His MSc dissertation explored how artificial intelligence will reshape the future of work, reflecting his curiosity about the wider social impact of emerging technologies. There are more people running local large language models now than perhaps ever before, especially now that they're so accessible for a wide range of hardware configurations, and easy to set up thanks to runtimes like Ollama and LM Studio. Cloud-based AI subscriptions also have their merits, but not everyone wants to pay monthly fees, contend with weekly limits, or trust a third party's servers with their sensitive information. Naturally, local AI looks appealing to those users. The performance of local models, however, is something that keeps sending newcomers back. Most users want a bigger and smarter model to match the level of service they receive from cloud providers, and when the bigger models crawl on their machine next to a cloud model that's one browser tab away, they quickly arrive at the verdict that their hardware isn't good enough for it, or that local models are too inefficient for everyday use. Both of those assumptions are wrong, and when you're running models locally, a lot depends on the settings most people don't bother to configure. The problem starts with how your models are set up Easy setups come at a cost of control In my experience, the hardware is rarely a problem, but the settings that the beginner-friendly runtimes run those models at almost always are. Getting your models set up correctly can let you run models larger than your VRAM suggests, faster than the online benchmarks tell you, and also potentially save hundreds of dollars on cloud subscriptions every year. Ollama and LM Studio are one of the most popular runtimes for beginners, and the appeal is that there's absolutely no friction when it comes to the setup process. Both have a clean and shiny GUI that anyone can navigate without hassle, which makes it simple to load a model and start chatting with it without ever needing to look at a PowerShell window. The problem is that, in return, these runtimes get to decide a lot of how your models run. They decide how a given model is split between the GPU and system RAM, how much memory goes to context, and which optimizations are switched off. That's exactly why I stick with llama.cpp. Funnily enough, while both Ollama and LM Studio use llama.cpp as their inference engine, but a lot of the tweaks that I use to optimize my local LLM experience on my PC relies on the use of flags that these runtimes either bury in their settings or don't offer altogether, which is why I think llama.cpp is the gold standard for running LLMs locally. Are your mixture-of-experts models too big for your VRAM? You can offload their experts to system RAM for faster speed I run several Gemma 4 models on my Lenovo Legion 5 Pro with only 8GB of VRAM on its mobile RTX 3070 GPU, and as you can probably tell, I have been no stranger to the feeling of having to pass up on larger models for the lack of memory. However, perhaps one of the most valuable tricks that I have learned, especially when it comes to mixture-of-experts models (like Qwen3.6-35B-A3B) is moving experts off the VRAM and parking them into the system memory. The way MoE models are designed is such that the model stores most of its weights as a pool of experts, but during active inference, only a few of them work on a given token, which makes them the safest part of the model to move off the GPU. Llama.cpp's --n-cpu-moe flag does this for a chosen number of layers, while the rest of the model stays resident in the VRAM. In a recent experiment on my Legion 5 Pro, which has 32GB of DDR4-3200, Qwen3.6-35B-A3B at UD-IQ4_XS crawled at an agonizing 4.02 tokens/second with zero offload, but setting --n-cpu-moe to 28 took it to 32.82, which is eight times the speed. If you're using LM Studio, there's a dedicated slider to manage offloading, but Ollama doesn't let you set it at all currently. Is your dense model too slow? Pair it with a draft model to win back some speed While you can comfortably reclaim some speed on an MoE model like Qwen3.6-35B-A3B or Gemma 4 26B-A4B by offloading expert layers, things aren't quite the same with a dense model like Gemma 4 12B, because every pass through a dense model touches all of its weights, and so, there's no idle portion to park in system RAM. So, is there anything you can do besides being stuck with a model that crawls or go back to paying $20 for a cloud subscription of your choice? That was a question I asked myself for a long time before I discovered speculative decoding. Now, there are several names for it depending on what model you're using, but the idea is largely the same. A much smaller model, known as a draft model, writes the next few tokens before the target model gets to them. The target model then reviews the whole batch at once, keeping whatever it would have written anyway and taking over from the first token that's discarded. Gemma 4 calls its version multi-token prediction (MTP) and Google makes a matching drafter for it. For my own use, I pair Unsloth's Gemma 4 12B QAT build with its 242MB MTP drafter at 32K context, which I find reasonable for everyday work. When pairing the model with its drafter, the same model, with the same sampler settings, manages an average of 179 tokens per second, and without it, it only manages an average of 75 tokens per second. The drafter does need its own share of the VRAM, though, so its always best to account for a few hundred megabytes. This is only the tip of the iceberg when it comes to optimizing your LLMs When it comes to both dense and MoE models, there's a way to optimize them for the hardware you're running the inference on, and doing so can help you run models your VRAM was never supposed to fit, at speeds that make going back to paid cloud subscriptions a tougher sell. Expert offloading and draft models are the places where I would start, but KV cache quantization, context size tuning and picking the right quantization all leave further room for optimization. When you're finally done with all of that configuration, you'll have a fast and capable model that's all your own, doesn't phone home, and doesn't depend on an internet connection to be useful.