AI Engineer Agentic Track: The Complete Agent & MCP

Day 3 - Install Ollama and Run Local LLMs: Llama, GPT-OSS and Gemma

Beginner 20 minutes
Rupinder Singh
7 views
Installing Ollama, picking models by size (the Llama 3.2 vs 3.3 trap, the '-cloud' naming gotcha), the ollama ls/serve/pull/rm commands, and calling local models through the same OpenAI-compatible pattern as the cloud providers.

Week 1 · Day 3 · Lecture 15. Lab file: 1_foundations/2_lab2.ipynb. Optional but recommended if your machine can handle it: install Ollama, pull a couple of small open-source models, and call them locally through the same OpenAI-compatible pattern from the last lecture.

This step is optional. Running local models needs a reasonably capable computer. If yours isn't, skip this lecture entirely and carry on with the cloud providers — nothing later in the course depends on having done this.

Installing Ollama

  1. Go to ollama.com and click Download for your OS (Mac, Linux or Windows).
  2. Run the installer. It briefly opens a chat window you can close — Ollama keeps running in the background after that.

(Deeper, step-by-step installation instructions are covered in more detail in the companion AI Engineering Core Track, if you need them.)

Choosing a model: size is the whole game

Browse available models on Ollama's models page — but ignore the "applications" section it shows you (that's for using Ollama inside other apps; here, you'll call it directly from your own code). What matters is the size listed next to each model.

Rule of thumb: stick to models around 3GB or smaller, unless you know your machine can handle more. Bigger models eat memory and can make your computer unusable while running. A common trap:

  • Llama 3.2 — the usual starting point, small and manageable. Good default choice.
  • Llama 3.3 — looks like a natural "upgrade" from 3.2, but it's 70 billion parameters and ~43GB on disk. It will overwhelm most computers. Avoid it (and Llama 4, also too large) unless you specifically know your hardware can cope.
  • Gemma (Google) — comes in several sizes; some are reasonable, others still quite large. Check the size before pulling.
  • Qwen (Alibaba Cloud) — a strong model family; Qwen 3.6 doesn't offer small variants, but Qwen 3.5 does have genuinely tiny options worth using.
  • GPT-OSS — the 20B version is about 14GB, right at the edge of what a well-equipped laptop can handle in memory. This is the largest model demonstrated in this lecture.

Watch for "-cloud" in a model's name. A model ending in "cloud" doesn't run on your computer at all — like Groq, it's served remotely. If your goal is genuinely local inference, avoid anything named that way.

Once you've picked a model, copy its exact name from the Ollama site — you'll need it verbatim for the pull command below.

Managing models from the terminal

Open a terminal (Ctrl/Cmd + backtick in Cursor) and use:

  • ollama ls — list models already downloaded (empty if you just installed).
  • ollama serve — start the Ollama service if it isn't already running ("address already in use" means it's already running — that's fine).
  • ollama pull <model-name> — download a model (this can also be run from inside a Jupyter cell by prefixing the line with !, which tells Jupyter to run it as a shell command).
  • ollama rm <model-name> — remove a downloaded model to free up disk space.

If your terminal says ollama isn't recognized, open a fresh terminal (or restart Cursor, or your computer) so it picks up the newly installed program.

Verifying Ollama is running

Ollama listens on port 11434. Two quick sanity checks in your browser:

  • http://localhost:11434 — should respond with "Ollama is running".
  • http://localhost:11434/v1/models — lists every model you've pulled, using the same OpenAI-compatible /v1/models shape you'd see from a cloud provider.

Calling local models — same pattern as before

Exactly like Anthropic, Google, DeepSeek, Groq and OpenRouter in the previous lecture, Ollama gets its own OpenAI-compatible client — just point base_url at localhost:11434. The call itself is identical: ollama.chat.completions.create(model=..., messages=...).

Three local models were added to the running comparison from the last lecture:

Local modelAnswer (paraphrased)
Llama 3.2 (1B version)Humility, over-optimized, becomes excessive self-importance — prioritizing your own needs over others'.
GPT-OSS (20B, run locally)Perseverance can become stubbornness — relentless pursuit of a goal blinding you to changing circumstances.
Gemma 4 (latest, Google's open-source model)Diligence, over-optimized, becomes workaholism — sidelining self-care and personal interests.

Worth noting: the smallest model here (Llama 3.2 1B) produced a noticeably more generic, boilerplate-feeling answer than the larger models — a useful, concrete reminder that model size still matters even when the prompt and task are identical. The 20B GPT-OSS model, by contrast, is demanding enough to visibly load a GPU while it runs (and took roughly 20-25 seconds per answer in this lecture's run, similar to Gemma 4).

Nine competitors, one question, run independently

With these three local models added to the six cloud calls from the previous lecture, the running comparison now has nine competitors answering the same LLM-generated question: GPT nano, Claude Sonnet, Gemini, DeepSeek, GPT-OSS-120B (via Groq), Kimi K2.6 (via OpenRouter), Llama 3.2 (local), GPT-OSS 20B (local), and Gemma 4 (local). Each call ran independently and sequentially in this walkthrough — they could equally have been run in parallel, since none of them depends on another's output.

What you should now understand

  • Running local models is optional and hardware-dependent — nothing later in the course requires it.
  • Model size is the main thing to check before pulling — stick to roughly 3GB or under unless you know your machine can handle more; Llama 3.3/4 and other 70B+ models will overwhelm most computers.
  • A model name ending in "cloud" is NOT local — it's served remotely, similar to Groq.
  • ollama ls, ollama serve, ollama pull, and ollama rm cover the basic model lifecycle from the terminal (or a Jupyter cell prefixed with !).
  • Ollama exposes the same OpenAI-compatible /v1 shape as cloud providers — the only change is pointing base_url at localhost:11434.
  • Smaller local models can produce noticeably more generic/boilerplate answers than larger models — a visible, practical illustration of the capability-vs-size trade-off.

Resource links

Next

With nine answers now collected, it's time to pick a winner — using an LLM itself as the judge.

Frequently Asked Questions (7)

Do I have to install Ollama to continue with the course?

No — running local models is entirely optional. It requires a reasonably capable computer, and nothing later in the course depends on having done it. Feel free to skip straight to the next lecture if your machine isn't suited to it or you'd rather not.

How do I know if a model is safe to run on my computer?

Check the size listed on Ollama's model page. As a rough rule of thumb, stay at or under about 3GB unless you know your machine has the memory to spare. A model listed at 40+GB with 70 billion parameters (like Llama 3.3) will overwhelm most consumer computers.

I typed "ollama" in my terminal and got "command not recognized." What do I do?

Open a brand new terminal window (or restart Cursor, or your whole computer) so it picks up the newly installed Ollama program. Your existing terminal session was opened before Ollama was added to your system PATH.

I got "address already in use" when running ollama serve — is that a problem?

No — that error just means Ollama is already running in the background. You don't need to run "ollama serve" again; you can go straight to using it.

What's the difference between a normal Ollama model and one with "-cloud" in its name?

A "-cloud" model, despite being listed on Ollama's site, does not run on your computer — it's served remotely, similar to how Groq runs models on its own hardware rather than yours. If your goal is genuinely local inference, avoid anything named that way.

Can I use the exact same code to call Ollama that I use for OpenAI?

Yes — Ollama exposes an OpenAI-compatible endpoint, so you use the same OpenAI Python client and the same chat.completions.create call. The only change is the base_url, which you point at http://localhost:11434 instead of a cloud URL.

Why did the smallest local model give a more generic-sounding answer than the others?

Smaller models generally have less capacity to produce nuanced, less-generic responses — this is a real capability trade-off, not just a resource-usage one. It's a useful, visible reminder to weigh model size against the quality a task actually needs, not just what fits on your disk.