20. Local AI Hardware

The lesson

Local AI performance depends on memory capacity, memory bandwidth, compute throughput, cooling, and model quantisation.

VRAM is usually the first limit for fast local inference. If the model does not fit in VRAM, offloading to system RAM can become much slower.

TPEE itself is hardware-light, but it helps you plan prompts for the hardware and local model you intend to use.

A junior IT support officer at a secondary school checks the memory and graphics capability of a staff laptop before recommending which local model it can run.

A hardware test engineer at an appliance manufacturer documents the memory bandwidth and cooling limits of a bench computer used for local inference trials.

A team lead at a video editing studio notes that a machine with a mid-range graphics card fits a small quantised model but struggles with anything larger.

An operations manager at a regional hospital plans local model use on ward workstations and flags the ones without memory for anything beyond short summaries.

An automation technician at a robotics workshop records that offloading model layers to system memory slows answers noticeably on the shop-floor computer.

A surveyor at a land surveying practice lists the storage and memory available on a field laptop before choosing a model size for drafting site notes.

A librarian at a community library documents that a shared public computer has no dedicated graphics memory, so only the smallest models are practical.

An energy analyst at a solar farm operator notes the cooling and memory limits of the site monitoring computer before planning any local model work.

An owner of a small family bakery measures how many tokens per second a modest laptop produces before committing staff to a local workflow.

A cybersecurity analyst at a consultancy writes a hardware checklist covering memory, storage and graphics capability before approving local models on staff machines.

Check yourself