What actually happens when you run AI models locally on your laptop?

A mechanical and architectural breakdown of memory bandwidth, quantisation, and context limits when executing open-weight reasoning models locally.

Executing a query against a cloud API like Claude or GPT-4o feels instantaneous and frictionless. You send an HTTP payload over TLS, let an enterprise data centre manage the compute across clusters of high-bandwidth GPUs, and stream back generated tokens.

Running an open-weight model like DeepSeek on your local workstation flips this mental model entirely. The remote infrastructure vanishes. In its place, your machine's physical constraints — memory bus bandwidth, unified RAM allocation, tensor precision, and thermal dissipation — become the hard boundaries of what your system can deliver.

When you run ollama run deepseek-r1:8b or launch a local llama.cpp server, your computer executes a tightly orchestrated sequence of system calls, matrix operations, and dynamic memory allocations. Understanding this pipeline reveals why local inference can feel remarkably responsive for short tasks — yet stall completely when handling long contexts — and why memory bandwidth, rather than GPU processor speed, is the primary engineering bottleneck in local AI.

1. The engineering reality of "running DeepSeek"

Before dissecting the execution path, it helps to clarify what running "DeepSeek" locally actually entails.

The primary model, DeepSeek-R1, is a 671-billion parameter Mixture-of-Experts (MoE) architecture. At 16-bit floating-point precision (FP16), storing its weights requires over 1.3 terabytes of VRAM. Running the uncompressed 671B model requires enterprise multi-GPU nodes or multi-node Mac Studio clusters linked over high-speed interconnects.

When an engineer runs DeepSeek on a standard workstation or high-end laptop, they are almost certainly running one of two things:

• A distilled reasoning variant: a dense architecture (ranging from 1.5B to 70B parameters, built on Qwen or Llama bases) fine-tuned on reasoning traces generated by the full DeepSeek-R1 model.

• A quantised MoE export: a compressed GGUF version of the 671B model (such as IQ1_S or Q2_K) mapped across system RAM, sacrificing numerical precision to fit within a consumer memory envelope.

Regardless of whether you run an 8-billion parameter distilled model or a compressed MoE export, the underlying low-level execution pipeline remains identical.

2. Storage to memory: mmap() and zero-copy loading

When you initiate a local inference runtime, the engine does not perform a standard disk-to-RAM byte copy of a 5 GB file. Doing so would incur unacceptable latency and double memory usage during initialization.

Instead, local runtimes rely on container formats such as GGUF (GPT-Generated Unified Format). GGUF organizes tensor weights into contiguous binary blocks aligned to hardware memory boundaries, alongside metadata and tensor geometry.

To load these weights, runtimes call the POSIX system function mmap() (memory mapping):

• Virtual address mapping: mmap() maps the GGUF file directly into the process's virtual address space, returning memory pointers without reading the entire payload into physical RAM immediately.

• On-demand page faults: as the execution engine accesses specific tensor layers during computation, the OS kernel loads the required disk pages into physical RAM on demand.

• Unified memory execution: on Apple Silicon architectures, Unified Memory Architecture (UMA) allows the CPU and GPU to read directly from the same physical LPDDR5 memory space. The GPU accesses mapped memory pointers without requiring PCIe bus transfers. On discrete GPU setups (e.g. Nvidia PC builds), mapped weights must instead be staged from system RAM across the PCIe bus into dedicated VRAM.

3. Weight compression: the quantisation mechanics

An uncompressed 8-billion parameter model stored in FP16 format requires 2 bytes per parameter:

Memory Footprint = 8 × 10⁹ parameters × 2 bytes = 16 GB

Because dedicating 16 GB of VRAM solely to model weights is impractical on most non-specialised hardware, runtimes use quantisation — mapping high-precision floating-point values to lower-bit representation formats.

In a standard Q4_K_M (4-bit medium) quantisation scheme:

• Weights are divided into blocks (typically 32 or 256 parameters).

• Floating-point weights are scaled and quantized to 4-bit integers (0 to 15).

• A 32-bit floating-point scale factor (FP32) is stored per block to scale values back during calculation.

Quantisation drops the footprint of an 8B model to ~4.5 GB. However, this introduces an important compute trade-off: during every forward pass, the GPU must dynamically dequantise these 4-bit integer values back into floating-point numbers in registers before executing matrix multiplication.

4. The two execution phases: prefill vs. decode

Once weights are staged in memory, inference operates in two distinct phases with opposite performance bottlenecks: prompt processing (prefill) and token generation (decode).

Phase 1 — Prefill (compute-bound)

During prefill, the engine processes the input prompt in parallel. Because all prompt tokens are known ahead of time, the system can saturate GPU compute cores using parallel matrix multiplication kernels (e.g. Metal Performance Shaders or CUDA BLAS). Performance here is measured in prefill tokens per second and depends primarily on raw arithmetic compute throughput (TFLOPS).

Phase 2 — Decode (memory-bound)

Token decoding is autoregressive: generating token N requires the output of token N-1. To output a single token, the engine must sweep the entire set of model weights through memory to calculate the next attention and feed-forward pass.

Token generation speed is determined by memory bandwidth, not compute capacity.

Consider a laptop with 150 GB/s of memory bandwidth running a 4.5 GB (Q4_K_M) 8B model:

Theoretical Max Throughput = Memory Bandwidth / Model Size
                            = 150 GB/s / 4.5 GB
                            ≈ 33.3 tokens/second

Even if you double the GPU compute core count, generation speed cannot exceed the physical rate at which the memory controller streams 4.5 gigabytes of weight data into registers for every single generated token.

5. Working memory and the KV cache

Beyond the static weight footprint, running a local model requires dynamic working memory. As reasoning models like DeepSeek-R1 generate extended chain-of-thought monologues, a second major memory consumer emerges: the Key-Value (KV) cache.

To avoid recomputing key and value attention vectors for historical tokens at every step, the inference engine caches them in memory. The KV cache memory footprint grows linearly with sequence length:

KV Cache Size = 2 × (Sequence Length) × (Layers)
              × (Attention Heads) × (Head Dimension)
              × (Precision Bytes)

For an 8B model running an 8,192-token context window at 16-bit precision, the KV cache adds 1.0 GB to 1.5 GB of dynamic RAM overhead on top of model weights. If dynamic memory allocation exceeds physical RAM limits, the OS will swap memory pages to disk, causing token generation speed to collapse from 30+ tokens per second to under 1 token per second.

6. Security, privacy, and system boundaries

Deploying local inference changes where computation occurs, but it does not automatically guarantee complete system isolation.

Strategic takeaways: evaluating local AI

Local models should not be evaluated as direct 1:1 drop-in replacements for multi-node commercial APIs. Instead, evaluate them based on architectural suitability:

• Focus on specialised, high-frequency tasks: local 8B and 14B models excel at constrained, structured operations — such as log parsing, code inline completion, entity extraction, and local document summarisation — where data privacy and low network latency outweigh global general-knowledge depth.

• Cap context windows explicitly: set hard boundaries on local context windows ( num_ctx 4096) in runtime configurations to prevent reasoning chains from exhausting system RAM and triggering OS page-swapping or out-of-memory (OOM) process terminations.

• Select precision based on task domain: use 4-bit or 5-bit quantisations ( Q4_K_M, Q5_K_M) for tasks requiring precise logic, code syntax, or mathematical reasoning. Reserve aggressive 2-bit or 3-bit quantisations strictly for general prose generation or basic text classification.

Local inference is fundamentally an exercise in systems engineering — converting physical memory bandwidth and hardware constraints into predictable, private execution.

Sources & further reading

• llama.cpp open-source repository — implementation details on the GGUF specification, mmap() zero-copy allocation, quantisation block structures, and Metal/CUDA backends.

• DeepSeek-R1 research paper — "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025) — model architecture specifications, parameter counts, MoE design, and distilled model derivations.

• Apple Developer Documentation — Unified Memory Architecture — unified CPU/GPU memory access mechanics and memory bus bandwidth constraints.

• Tim Dettmers' LLM hardware & quantisation analysis — memory bandwidth calculations, prefill/decode phase trade-offs, and quantisation precision metrics.