21. Quantisation and Model Size

The lesson

Quantisation stores model weights with fewer bits. This lowers memory use and can increase speed, but may reduce quality.

A 7B or 13B quantised model can be very useful locally. Larger models may need more VRAM or careful offloading.

The right choice depends on task risk: drafting and summaries tolerate smaller models; deep reasoning and complex coding often need stronger models.

A maintenance planner at a textile mill picks a low-bit model for summarising machine logs because it fits the office computer with memory to spare.

A junior support analyst at an insurance firm records that an eight-bit model gave clearer answers than a four-bit one on policy wording questions.

A team lead at a pharmaceutical warehouse documents which quantised model fits the available graphics memory on the stockroom computer.

A firmware tester at an electronics workshop chooses a smaller quantised model for routine code comments and a larger one for difficult fault analysis.

A veterinary practice manager notes that a four-bit model runs quickly on the clinic laptop but occasionally muddles the wording of dosage instructions.

A reliability engineer at a mining operator compares an eight-bit and a four-bit model on equipment failure notes before deciding which one to keep.

A junior archivist at a museum records that a small quantised model handles catalogue descriptions well enough for a first draft.

A construction site supervisor notes that a heavily compressed model answers faster on a rugged tablet but loses detail on long specifications.

A customer service team leader at an online retailer documents which model size suits routine reply drafts against complex complaint histories.

A tools programmer at a game studio records the memory saved by a four-bit model when running local checks on a developer workstation.

Check yourself