Quantisation is how a 70-billion parameter model that normally needs 140GB of RAM gets compressed into something that fits on 40GB. The trade-off is precision: quantisation reduces the numerical accuracy of each model weight from 16 bits down to 8, 6, 4, or even fewer bits. Less precision means a smaller file and faster inference, at the cost of some response quality. Understanding which quantisation level to use, and what you are giving up, is the difference between a local AI setup that actually works and one that frustrates you into stopping.
In short: Q4_K_M is the practical default for most hardware. It fits 7B models in 5GB of RAM with acceptable quality loss. Q6_K is the sweet spot for quality if you have the RAM. Q8_0 is close to FP16 quality but uses substantially more RAM than Q4_K_M; the exact ratio depends on the model and runtime configuration. Start with Q4_K_M unless you have 16GB or more free for AI inference, in which case Q6_K or Q8_0 is worth the upgrade.
What Quantisation Actually Does
A language model consists of billions of numerical values called weights. In their original training form, each weight is stored as a 16-bit or 32-bit floating point number (FP16 or FP32). These high-precision numbers require significant storage and RAM. A 7B parameter model in FP16 format requires approximately 14GB of RAM to load.
Quantisation maps those high-precision weights to lower-precision representations. An 8-bit integer can represent 256 distinct values. A 4-bit integer can only represent 16 values. Compressing from FP16 to 4-bit is roughly a 4x reduction in size, but the model is now approximating what it used to represent exactly. In practice, language models are surprisingly robust to this approximation because most weights cluster around small values and the information content is highly redundant across billions of parameters.
A Q4_K_M artifact for a roughly 7-8B model is often around 4-5GB, but runtime memory also includes the KV cache and compute buffers, and speed depends on the inference backend and hardware. The quality reduction is real but modest for most tasks. A model asked to summarise text or answer factual questions at Q4_K_M will produce responses indistinguishable from Q8_0 for most users. Tasks requiring precise reasoning, complex code generation, or sustained logical chains show more degradation.
Decoding the Naming System: What Q4_K_M Actually Means
Model files distributed in GGUF format (the standard for Ollama and llama.cpp) use a naming convention that combines the quantisation bit depth, the quantisation method, and the size variant. The format is Q[bits]_[method]_[size].
The bit number (Q4, Q5, Q6, Q8) indicates the target bits per weight. Higher numbers mean more precision and larger files. Q4 uses approximately 4 to 4.5 bits per weight when method overhead is included. Q8 uses approximately 8 bits per weight.
The K suffix indicates k-quants, a newer quantisation approach developed by the llama.cpp team. K-quants use different quantisation levels for different parts of the model. K-quant recipes can mix tensor precisions, but the tensors receiving higher precision depend on the quantisation recipe, model architecture, and tensor type. This produces significantly better quality than the older uniform quantisation at the same average bit depth. Check the complete format name: Q4_0 and Q4_1 are legacy formats, while Q4_K and Q4_K_M refer to the K-quant recipe. K-quants are useful options, but do not assume the K suffix is automatically superior; compare size, throughput, and quality on the target model and workload.
S, M, and L denote Small, Medium, and Large where those variants exist; Q3_K has S/M/L variants, Q4_K and Q5_K have S/M variants, and Q6_K has no size suffix. These variants apply different mixes of quantisation levels within the k-quant framework. K_M (Medium) is the recommended default for most use cases, balancing quality and size well. K_S (Small) saves a small amount of RAM by being slightly more aggressive. K_L (Large) retains more quality at a slightly larger size. For most hardware, K_M is the right choice.
| Q4_K_S | ~4.3 bits/weight. Smallest Q4 variant. Tight on quality. Use when RAM is the hard constraint and K_M does not fit |
|---|---|
| Q4_K_M | ~4.5 bits/weight. Recommended default. Best balance of RAM usage, speed, and quality for hardware with limited RAM |
| Q5_K_M | ~5.5 bits/weight. Noticeable quality improvement over Q4 for reasoning tasks. Good choice when you have a few extra GB of RAM |
| Q6_K | ~6.6 bits/weight. High quality, close to FP16 output. Recommended when 16GB or more is available for inference |
| Q8_0 | ~8 bits/weight. Near-lossless compared to FP16. Largest quantised format. For hardware with abundant RAM and where quality is paramount |
| FP16 (unquantised) | 16 bits/weight. Original training precision. Requires 2x the RAM of Q8_0. Rarely practical on consumer hardware |
RAM Requirements by Model Size and Quantisation Level
The RAM required for a model is the model file size plus overhead for the context window and runtime. Context and runtime overhead varies substantially with model architecture, context length, cache precision, batch settings, and backend; calculate or measure it for the specific model and configuration. The figures below are the model file requirements without context overhead.
RAM Required by Model Size and Quantisation Level
| Q4_K_M | Q5_K_M | Q6_K | Q8_0 | |
|---|---|---|---|---|
| 7B model | ~4.5GB | ~5.7GB | ~6.6GB | ~8.0GB |
| 13B model | ~8.0GB | ~9.8GB | ~11.5GB | ~14.0GB |
| 34B model | ~20GB | ~25GB | ~29GB | ~34GB |
| 70B model | ~38GB | ~48GB | ~58GB | ~70GB |
| Hardware minimum (model only) | 5GB free RAM for 7B | 6GB free RAM for 7B | 7GB free RAM for 7B | 9GB free RAM for 7B |
Available RAM is not the same as total RAM. Your OS, running services, and other applications consume RAM before the model loads. System memory use varies with the OS version, installed services, storage workload, and background applications; measure currently available memory on the target system. Subtract this from your total RAM to determine what is actually available for model inference.
Which Quantisation Level for Which Hardware
The right quantisation level depends entirely on how much RAM is available after the system takes its share. The goal is to fit the model entirely in RAM. If inference causes sustained swapping, performance can degrade severely; measure the target configuration and leave memory headroom for context and runtime buffers.
| 4 to 5GB available for AI (entry NAS, shared system) | 7B at Q4_K_S only. Expect 1 to 2 tokens/sec. Models larger than 7B will not fit |
|---|---|
| 6 to 7GB available for AI (NAS with 8GB total, mini-PC entry) | 7B at Q4_K_M comfortably. Best practical option for this hardware tier |
| 8 to 10GB available for AI (an upgraded NAS or capable mini-PC with more than 10GB of usable system memory) | 7B at Q6_K or Q8_0. 13B at Q4_K_M with caution. 13B may be tight depending on context settings |
| 12 to 14GB available for AI (NAS with 16GB, mid-range mini-PC) | 7B at Q8_0. 13B at Q5_K_M comfortably. Best all-round tier for quality without compromise |
| 16 to 24GB available for AI (capable mini-PC, 32GB system) | 13B at Q8_0 may fit; a 34B Q4_K_M generally requires the upper end of this range plus enough headroom for context and runtime buffers. |
| 32GB or more available for AI (high-end mini-PC or desktop) | 70B at Q4_K_M. 34B at Q8_0. Serious hardware for demanding use cases |
What Quality Difference You Actually Notice
The quality difference between Q4_K_M and Q8_0 on the same model is smaller than most people expect. On conversational tasks, summarisation, and simple question answering, the two outputs are largely indistinguishable. The gap becomes meaningful in specific scenarios.
Complex multi-step reasoning is where aggressive quantisation hurts most. Tasks like solving logic puzzles, following long chains of conditional instructions, or generating syntactically complex code show measurable degradation at Q4 compared to Q6 or Q8. Long-context constraint tracking may vary by model and quantisation format and should be tested on the intended context length and workload.
Creative-writing effects are model- and workload-dependent; compare representative outputs before choosing a quantisation level for this use case. The model defaults to more common phrasings and loses some of the stylistic range available at higher precision.
Some knowledge and question-answering benchmarks remain close to the full-precision baseline at Q4, but sensitivity varies by model, task, and quantisation format. A model asked about documented facts performs similarly across quantisation levels
The practical implication: for most home AI use cases, Q4_K_M is genuinely good enough. If you are using local AI for coding assistance or complex reasoning tasks and quality matters, Q5_K_M or Q6_K is worth the extra RAM if your hardware supports it. Q8_0 is rarely necessary unless you are doing systematic evaluation or fine-tuning work.
Speed Differences Between Quantisation Levels
Lower-bit formats often improve memory-bound token generation, but speed is not strictly ordered by bit depth and depends on the format, kernels, backend, hardware, and workload phase. Q4_K_M generates tokens faster than Q8_0 on the same model because the processor handles less data per weight during inference. The difference is not enormous, but it is measurable.
Token-generation speed varies substantially with the processor, memory bandwidth, model, backend, context settings, and build; benchmark the intended hardware and model rather than relying on generic mini-PC ranges. All of these are fast enough for responsive conversation. The gap becomes more noticeable on slower hardware like a NAS, where the processor is a stronger bottleneck and any extra data processing overhead compounds.
On NAS hardware, the performance difference between quantisation formats depends on the model, context, memory configuration, inference build, and competing NAS workloads; benchmark the exact device and configuration. On this hardware tier, Q4_K_M is the correct choice not just for RAM reasons but for speed.
Which Model to Download: A Practical Decision
When downloading models from Hugging Face or through Ollama's model library, use this decision process. First, determine your available RAM after system overhead. Second, identify the largest model size that fits at Q4_K_M with that RAM. Third, if RAM headroom exists after fitting the model, consider upgrading to Q5_K_M or Q6_K for better quality rather than jumping to a larger model at Q4_K_M.
Do not assume a higher-precision 7B model will outperform a lower-precision 13B model; compare the specific models on representative tasks within the available memory budget. Model capability depends on architecture, training, parameter count, quantisation method, and task; benchmark the actual candidate models instead of prioritising quantisation precision over parameter count by rule. The exception is very large parameter count differences: a 70B at Q4_K_M will outperform a 7B at Q8_0 on complex tasks, because the additional knowledge capacity of the larger model outweighs the precision advantage of the smaller one.
For Ollama running on a NAS, the recommended starting point is a 7B model at Q4_K_M. Older options that remain available include Llama 3.1 8B Q4_K_M, Mistral 7B Q4_K_M, and Qwen2.5 7B Q4_K_M, but compare them with current Llama, Mistral, and Qwen generations before choosing. These fit on systems with 6GB or more of available RAM and produce acceptable quality for most tasks. Llama 3.3 70B at Q4_K_M is not viable in 16GB of RAM; its Ollama model artifact is about 43GB before context and runtime overhead.
Related reading: our NAS buyer's guide.
Related reading: our NAS explainer.
What is the difference between Q4 and Q4_K_M?
The legacy llama.cpp formats are Q4_0 and Q4_1; Q4_K is an alias for Q4_K_M, and complete GGUF models can contain a mixture of tensor precisions. Q4_K_M uses k-quants, a newer method that applies different precision levels to different parts of the model. K-quant recipes can mix tensor precisions, but which tensors receive higher precision depends on the specific recipe and model architecture. The result is noticeably better output quality at the same or slightly larger file size. If both Q4 and Q4_K_M are available for a model, Q4_K_M is the better choice in almost every case.
What quantisation level should I use on a NAS?
For most NAS hardware, Q4_K_M is the correct choice. NAS devices have limited RAM available for inference after the OS and services take their share. Available model memory on an 8GB NAS varies with the OS, installed services, containers, caching, and active workloads; measure free memory on the target system. Q4_K_M for a 7B model requires approximately 4.5 to 5GB, which fits comfortably. Q5_K_M or Q6_K require 5.7GB and 6.6GB respectively and may cause the model to run slowly or fail to load if available RAM is tight. See the local AI NAS guide for current NAS hardware recommendations with their available RAM figures.
Is Q8_0 worth it or is Q4_K_M good enough?
For most conversational and summarisation tasks, Q4_K_M is good enough. Controlled benchmarks on some models show small differences between Q4_K_M and higher-precision formats, but perceptibility depends on the model, prompts, tasks, and evaluation method. Q8_0 is worth considering for complex reasoning tasks, code generation, and work where precision matters, provided your hardware has the RAM to support it without impacting inference speed. If you have 16GB or more available for inference, Q6_K is a better practical choice than Q8_0: you get most of the quality benefit at significantly lower RAM usage and faster inference speed.
Does quantisation affect how fast the model responds?
Yes. Lower quantisation (fewer bits) is faster because the processor handles less data per operation. Lower-bit formats are often faster for memory-bound decoding, but throughput is not guaranteed to follow Q4_K_M > Q6_K > Q8_0; benchmark the exact hardware, backend, and model. The speed difference is more significant on slower hardware like a NAS than on a capable mini-PC with a modern processor. The practical speed difference varies with the processor, memory bandwidth, model, backend, context settings, and workload; benchmark the exact NAS or mini-PC configuration.
Can I run a 13B model on 8GB of RAM?
A 13B model at Q4_K_M requires approximately 8GB of RAM for the model itself. On a system with 8GB total RAM, after OS overhead there is not enough space for the model plus context. This causes the model to use disk-based swap, which reduces inference speed to near-unusable levels. For reliable 13B inference, 16GB of system RAM is the practical minimum, giving approximately 10 to 12GB for the model after OS overhead. Whether a 13B model is viable on a 16GB NAS depends on the exact model artifact, measured available memory, context settings, cache precision, backend, and competing services. The guide on what runs on each RAM tier covers this in more detail.
Ready to run a local LLM but not sure which NAS or hardware to use? The local AI hardware guide covers which models support Ollama, RAM ceilings, and current AU pricing.