On-device AI · 8 min read

How a language model fits on a phone

Local AI apps ask for a surprising amount of storage, and the reason is not bloat. Understanding where those gigabytes go also explains why the model is fast on a new phone and sluggish on an old one — and what the file name Q3_K_S is actually telling you.

A model is a large pile of numbers

A language model is a set of learned parameters — weights. Each is a number, and there are billions of them. "2B" in a model name means roughly two billion parameters.

Stored at full precision, each weight takes 4 bytes. Two billion weights at 4 bytes each is about 8 GB, for one of the smaller models. That is the problem in one line: the file is large because the numbers are numerous and each one is stored precisely.

Quantisation: fewer bits per number

Quantisation stores each weight using fewer bits. Instead of 32 bits per weight you use 8, or 4, or 3 — with scaling factors per block of weights so the approximation stays close to the original.

The size reduction is roughly linear:

• 32-bit — ~8 GB for a 2B model. Full precision, phone-hostile.
• 8-bit — ~2 GB. Very close to original quality.
• 4-bit — ~1.1 GB. The usual sweet spot.
• 3-bit — ~800 MB. Noticeably lossier, still useful.
• 2-bit — smaller again, and quality falls off sharply.

This is lossy compression, like a JPEG. Push it far enough and you see the artefacts — in a language model those appear as vaguer answers, repetition, or a drift away from the question.

Quantisation does not make a model dumber in a uniform way. It makes it less precise, and imprecision shows up first on the hardest inputs.

Reading the file name

A name like Q4_K_M or Q3_K_S is a recipe, not a version number:

• Q4 / Q3 — roughly how many bits per weight.
• _K — a "k-quant" scheme, which spends its bit budget unevenly, keeping more precision where the model is most sensitive.
• _S, _M, _L — small, medium or large within that scheme. Larger keeps more quality and more bytes.

So Q3_K_S is a small 3-bit k-quant: aggressively compressed, chosen when fitting on the device matters more than the last few percent of quality.

GGUF: the container

GGUF is the file format that holds the quantised weights plus the metadata a runtime needs — tokeniser, architecture, prompt template. One file, no accompanying directory of config.

It matters for phones because GGUF is designed to be memory-mapped. Rather than reading 1.7 GB into RAM, the runtime maps the file into its address space and the operating system pages in the parts actually being used. That is why a model larger than a phone's free RAM can still run, and why the first answer after opening an app is slower than the next — the pages are still being fetched.

Why it warms the phone

Generating a word — a token — means running the input through those billions of parameters. Then doing it again for the next token. A hundred-word answer is a hundred passes.

That is sustained arithmetic on the CPU or neural accelerator, which is exactly the workload phones are thermally worst at. Devices throttle when hot, so a long uninterrupted generation gets slower as it runs. Well-built local apps pace their work — spacing tasks apart so the chip can cool — rather than queuing everything back to back.

Why old phones struggle

Three constraints, all of them hard:

• Memory bandwidth. Every token means streaming a large fraction of the weights. Generation speed is often bounded by how fast memory can be read, not by raw compute.
• RAM. Memory-mapping helps, but a device that is already tight will page constantly, and the OS may simply kill the app.
• Accelerators. Newer chips have hardware suited to this arithmetic. Older ones fall back to general-purpose cores and run several times slower.

This is why local AI apps state a minimum device rather than trying to support everything. Below a certain point the experience is not degraded, it is unusable.

The trade you are making

A phone-sized quantised model will not match a large cloud model on hard reasoning. For a bounded task — what does this word mean in this sentence, translate this phrase — it is entirely sufficient, and it comes with three properties a cloud model cannot offer: it works with no connection, it costs nothing per use, and the text never leaves the device.

That is the whole bargain of on-device AI. You give up the largest possible model, and you get privacy, offline capability and no metering.