74. Ollama

The lesson

Ollama is a free, open-source tool that makes it easy to run large language models on your own computer. It handles the complex steps of downloading model weights, configuring the runtime, and exposing a chat interface or API, so you can focus on using the model instead of managing infrastructure.

The main reason people use Ollama is local execution. When a model runs on your machine, your prompts and documents never leave your network. This is important for privacy, intellectual-property protection, and working offline. There are also no per-token API fees, although you do need capable hardware.

Installing Ollama gives you a command-line program. To start chatting, you type a command such as `ollama run llama3` or `ollama run qwen2`. Ollama downloads the model if you do not already have it, then opens an interactive chat session where you can type prompts and receive answers.

Ollama maintains a large model library. You can find general chat models such as Llama 3, Mistral, and Qwen; coding models such as CodeLlama and DeepSeek-Coder; and smaller models such as Phi 3 and Gemma that fit on laptops with limited memory. The library is constantly growing as the open-source community releases new models.

You can customize models with a Modelfile, which is like a recipe. In a Modelfile you set the base model, the system prompt, parameters such as temperature and context length, and any adapter files. This lets you build specialized agents or enforce a consistent personality across conversations.

Ollama exposes a local REST API and an OpenAI-compatible endpoint. That means tools such as LangChain, OpenWebUI, Continue, and many IDE extensions can talk to Ollama as if it were a cloud provider. This makes it easy to integrate local models into your existing workflow.

Hardware matters. Running a 70-billion-parameter model at full precision requires a powerful GPU and a lot of memory. Most users run quantized versions, which use less memory and run faster at the cost of a small quality drop. Ollama handles quantization automatically when you pull a model.

Best practices: keep Ollama and your models updated, choose a model size that fits your hardware, test outputs before using them in production, and remember that local models can still produce incorrect or biased answers.

A ward clerk at a hospital runs a local Ollama model on a ward computer to draft patient-information letters from her own notes, writing the prompt in TPEE first for consistent wording.

A drill site technician at a mine runs Ollama on an office laptop to summarise daily drill sheets, because the site link is unreliable and the data stays on site.

A hotel duty manager runs a local Ollama model to draft replies to guest feedback, keeping guest names on the property network.

A claims team lead at an insurance office records the system prompt and temperature in TPEE before running a local Ollama model over de-identified claim notes.

A mechanic at a repair workshop pulls a small Ollama model onto the shop laptop to explain fault codes, then checks every answer against the service manual.

A lecturer writes a Modelfile recipe note in TPEE describing base model, tone and context length for a marking-assistant model run locally in the classroom.

A community services coordinator runs Ollama locally to summarise case-worker visit notes, keeping family details off any hosted service.

A pharmacy technician runs a local Ollama model to draft plain-language medicine reminders, with a pharmacist reading each draft before it leaves the counter.

A warehouse supervisor uses a local Ollama model on the depot computer to group stock-take comments into themes for the morning briefing.

An architectural technician runs Ollama locally to summarise planning correspondence, keeping client drawings and names on the office machine.

Check yourself