Running AI Locally: Matching Model Size to Hardware
A quality engineer I know spent a weekend trying to run a 70-billion-parameter model on a laptop with 16 GB of RAM. The model loaded, technically. Then it swapped to disk, and a single response took long enough that he went to make coffee, drank it, and came back to find it still generating. The model wasn't broken. The hardware wasn't broken. The mismatch was the problem.
This is the first real lesson in local AI: model size and hardware size have to agree, and the agreement is mostly about memory. If you're a quality engineer learning AI, you already have the right instincts for this. You've spent your career watching systems fail at their boundaries — under load, at capacity, at the edge of a spec sheet. Local AI is the same discipline applied to a different kind of system.
Why this matters before the mechanics
Cloud AI hides the hardware from you. You send a request, you get an answer, and someone else worries about GPUs. Local AI removes that abstraction. The moment you run a model on your own machine, you become responsible for the resource envelope: how much memory the model needs, how much you have, and what happens when the first number exceeds the second.
For a quality engineer, that's actually an advantage. You get to see the failure modes directly — slow generation, out-of-memory crashes, thermal throttling — instead of reading about them in someone else's incident report. Learning where the boundaries are is the whole exercise.
The one number that governs everything
Model weights are stored as numbers. A model with 7 billion parameters, stored at 4 bits per parameter, needs roughly 3.5 GB just to hold the weights. At 8 bits, roughly 7 GB. At 16 bits, roughly 14 GB. These are approximations — real files carry overhead — but the arithmetic is the point: memory scales with parameter count and precision.
On top of the weights you need working memory for the context window (the conversation so far) and runtime overhead. That pushes real requirements above the raw weight size, sometimes substantially at long context lengths.
So the practical rule is: know your available memory, subtract a healthy margin, and only then pick a model. Not the other way around.
Hardware size, from small to large
"Hardware size" here means memory capacity and memory bandwidth, not physical dimensions. A phone, a laptop, a workstation, and a server rack sit at very different points on that scale.
| Hardware class | Typical memory situation | What it realistically runs |
|---|---|---|
| Phone / tablet | Shared system memory, tight thermal budget | Small models (roughly 1–3B parameters) at low precision, short contexts |
| Consumer laptop, integrated graphics | 8–16 GB shared RAM | Small to mid models (roughly 3–8B) at 4-bit quantization; CPU-only is slow but workable |
| Laptop or desktop with a discrete GPU | 8–24 GB dedicated VRAM | Mid models (roughly 7–14B) comfortably; larger models only with aggressive quantization and offloading |
| Workstation / multi-GPU | 48 GB+ combined VRAM | 30–70B class models, depending on precision and context |
| Server-class | Hundreds of GB | The largest open models, often across multiple devices |
Two cautions on that table. First, the ranges are deliberately loose because real performance depends on the specific model, the quantization method, and the runtime — not just the parameter count. Second, "runs" and "runs usefully" are different claims. A model that fits but generates two tokens per second is technically running and practically useless for interactive work.
The trade-off nobody advertises
Quantization — storing weights at 4 or 8 bits instead of 16 — is how most people fit bigger models onto smaller hardware. It works, and it's the reason local AI is viable on consumer machines at all. But it is a trade, not a free lunch. Lower precision can degrade output quality, and the degradation isn't uniform across models or tasks. The cheap option costs you accuracy, and you won't always notice it without testing.
This is where your quality engineering background earns its keep. Treat quantization levels as configuration variants and test them like you'd test any other variant: same inputs, compare outputs, look for regressions. Don't assume 4-bit is fine because it loaded.
Two concrete examples
Example 1: the 7B model on a 16 GB laptop. A 7B model at 4-bit precision needs roughly 3.5–4 GB for weights, plus context and overhead — call it 6–8 GB in practice. On a 16 GB machine with integrated graphics, that fits with room to spare, and CPU inference is slow but usable for non-interactive tasks. The same model at 16-bit needs around 14 GB for weights alone, which does not fit comfortably once the operating system takes its share. Same model, same laptop, two very different outcomes based purely on precision.
Example 2: the 70B model. At 4-bit, a 70B model needs on the order of 35–40 GB for weights before any context. That does not fit on any consumer laptop and does not fit on most single consumer GPUs. It needs a workstation with multiple GPUs or a server. Attempting it on a 16 GB machine doesn't produce a slow assistant; it produces disk swapping and, often, a crash. This is the case where the honest answer is: this does not work on that hardware, and no amount of configuration will change it.
A practical way to choose
Work backwards from what you have:
- Measure your actual available memory, not the marketing number. Operating systems and other applications take their cut.
- Reserve a margin — a few gigabytes is not excessive.
- Pick a model whose quantized weights plus context fit inside what's left.
- Benchmark generation speed before committing. If it's too slow to use, drop a model size rather than fighting the hardware.
- If output quality matters, test the quantization level against a higher-precision baseline.
The source video for this article (linked below) walks through running models across a range of hardware sizes; treat its specific measurements as the presenter's own and verify them against your machine, since results vary with runtime versions and configuration.
FAQ
Can I run a large model on a small machine by offloading to disk? You can, and it will be extremely slow. Offloading moves weights out of memory and back, and the storage bottleneck dominates. It's useful for testing, not for work.
Is CPU-only inference viable? For small models and non-interactive tasks, yes. For interactive chat with mid-size models, it's usually too slow to be pleasant. A discrete GPU changes the picture substantially.
Does more RAM always mean I can run a bigger model? Memory capacity is necessary but not sufficient. Memory bandwidth affects speed, and thermal limits affect sustained performance. A machine can have enough capacity and still be too slow.
How do I know if quantization hurt quality? Run the same evaluation prompts against the quantized model and a higher-precision version, and compare. There's no shortcut; this is ordinary regression testing.
Conclusion
Local AI is a resource-matching problem, and the resource that matters most is memory. Model size, precision, and hardware size form a triangle — change one and the other two constrain you. The failure mode when they disagree isn't subtle: it's a machine that swaps, stalls, or crashes. The fix isn't better hardware or a bigger model in isolation; it's picking a combination that fits and then testing whether the fit is actually usable. Quality engineers already know how to do this. The only new part is the vocabulary.
Sources and verification
- Source material: The article references a YouTube video (https://www.youtube.com/watch?v=rPGJhrunbxo) on running AI models across hardware sizes. No transcript or extract was provided, so no specific claims, quotations, or benchmark figures from that video are asserted here.
- Owner's measurements: None. No benchmarks were run for this article.
- General arithmetic: The weight-size estimates (parameters × bits ÷ 8) are standard arithmetic, presented as approximations, not measured results. Real memory use is higher due to context and runtime overhead.
- Not verified: Specific model performance on specific hardware, quantization quality impacts for any named model, and generation speeds. These vary by runtime, version, and configuration and should be measured on your own system.