Local: llama.cpp, Ollama, vLLM and MLX, and the Sizing Arithmetic

Local inference is not a cost saving measure; it is a latency and sovereignty trade-off that often costs more in engineering time than the API bills you are trying to avoid. The abstraction layer you built in the previous lessons treats the model provider as a black box. Swapping the environment variable from an OpenAI-compatible endpoint to a local host feels trivial in code. It is not trivial in infrastructure. When you run models locally, you stop paying per token and start paying in silicon, memory bandwidth, and maintenance. The choice of runtime determines whether your application serves one user at a time or thousands in parallel. It determines whether your server needs a single consumer GPU or a rack of H100s. You must pick the runtime that matches the shape of your traffic, not just the model you want to run.

Runtime Selection by Workload Shape

There is no single best local inference engine. Each runtime solves a different constraint. llama.cpp is the portable standard. It compiles to C and runs on almost any CPU or GPU. It is the correct choice when you need to deploy to edge devices, browsers via WebGPU, or mixed hardware fleets where driver compatibility is fragile. It lacks advanced batching. It is slow for high throughput. It is excellent for single-request latency on constrained hardware. Ollama wraps llama.cpp in a Dockerised daemon. It provides a unified API that mimics OpenAI. It handles model downloading, quantisation selection, and lifecycle management. Use Ollama for development, prototyping, and internal tools where developer velocity matters more than raw throughput. It abstracts away the pain of managing GGUF files. It adds overhead. It is not designed for production serving at scale. It is a convenience layer. If your team needs to spin up a model in five minutes without reading documentation, Ollama is the tool. If you need to squeeze every millisecond out of a GPU, it is not. vLLM is the production standard for high-throughput serving. It implements PagedAttention, a memory management technique that eliminates fragmentation in the KV cache. This allows it to pack requests tightly. It supports continuous batching. It serves many users concurrently with low latency. It requires NVIDIA GPUs. It is complex to deploy. It needs Kubernetes or careful orchestration. Use vLLM when you have sustained traffic and need to maximise GPU utilisation. It is the wrong choice for sporadic, single-user workloads. The overhead of managing the scheduler outweighs the benefits. MLX is Apple’s framework for machine learning on Apple Silicon. It unifies compute and memory. It leverages the unified memory architecture of M-series chips. It provides near-metal performance on Macs. Use MLX when your deployment target is macOS or iOS. It is not portable to Linux servers. It is not relevant for cloud GPUs. It is the only viable option for high-performance local inference on Apple hardware.

The Sizing Arithmetic

You cannot run a model if you do not have the memory. The calculation is simple but often misunderstood. The base weight size is the number of parameters multiplied by the bytes per parameter. A 7-billion parameter model at 4-bit quantisation requires approximately 3.5 gigabytes for weights. This is not the total memory requirement. You must add the KV cache. The KV cache stores the attention states for the context window. It grows linearly with the number of tokens in the prompt and the response. For a 7B model with a 4096 token context, the KV cache can consume another 1 to 2 gigabytes. Add overhead for the runtime, the operating system, and the application. An 8GB machine can run a 7B model at 4-bit quantisation. It will be tight. The KV cache will fill quickly. Long conversations will cause out-of-memory errors. You must limit the context window. A 24GB machine, such as a consumer RTX 3090 or 4090, can run a 13B model at 4-bit quantisation comfortably. It can run a 7B model with a large context window. It can run multiple 7B models if you use CPU offloading, though this kills performance. A 96GB machine, such as an A100 or H100, can run a 70B model at 4-bit quantisation. It can run multiple 13B models. It enables high-throughput serving with vLLM. Quantisation reduces memory usage at the cost of quality. 4-bit quantisation is often indistinguishable from full precision for chat tasks. 8-bit quantisation provides a safety margin. 16-bit quantisation is rarely necessary for inference. It doubles the memory requirement. It provides marginal quality gains. The trade-off is rarely worth it. Measure the quality loss by running a benchmark suite on your specific dataset. Do not rely on general claims. The quality drop varies by model family and task.

Deploying the Local Model

To run the pipeline locally, you must align the runtime with the hardware. If you have an NVIDIA GPU, install vLLM. Pull the model weights in the format vLLM expects. Start the server. Configure the environment variable in the pipeline to point to localhost. The code remains unchanged. The interface is the same. The latency drops. The cost drops. The responsibility rises. If you have a Mac, install MLX. Load the model. Serve it via a simple HTTP wrapper. Configure the pipeline. The latency is low. The throughput is limited by the single machine. It is sufficient for personal use. It is not sufficient for production. If you have a Linux machine with a consumer GPU, install Ollama. Pull the model. Start the daemon. Configure the pipeline. It works. It is easy. It is fragile. Monitor the memory usage. Watch for out-of-memory kills. Limit the concurrency. The artifact you produce is a local model running. It serves requests. It returns JSON. It integrates with the loop. It proves that the abstraction holds. It exposes the infrastructure constraints. You have moved from a black box to a transparent system. You now see the cost of inference. You now see the value of quantisation. You now see the importance of context management. The next step is selecting the model itself. The runtime is the engine. The model is the fuel. You must choose the right fuel for the engine. The next lesson covers Choosing Weights: Families, Size Classes and Licence Terms.

Apply it to your work