buttercup.sh updates unit 1

Unit 1 — a model in the tab: no key, no server, no bill

Before an agent can do anything it has to talk to a model, and every tutorial starts by asking you for a credit card. This one does not. WebLLM runs a real language model inside the browser, on your own GPU, with the network off. No account, no key, no server, nothing installed. The point of this unit is to get you to the one exchange the whole course is built on — messages in, tokens out — with every obstacle removed and nothing at stake.

The one idea

A language model is a function. It takes an array of messages and returns text, one token at a time. That is the entire interface, and it does not change when the model gets cleverer, when you put it behind an API, or when you wrap it in an agent. Everything in the five units after this one is a wrapper around this single call.

So the useful first question is not “which model” or “which framework.” It is: what exactly do I send, and what exactly comes back?

you send — an array 
[
  { role: "system",
    content: "Answer in one sentence." },
  { role: "user",
    content: "Why is the sky blue?" }
]
you get back — text, in pieces 
"Blue"  " light"  " scatters"
" more"  " than"  " red"  "."
There is no session, no memory and no connection. Each call is the whole conversation, sent again. Keeping that array is your job, and by unit 3 it will be the only state your agent has.

Normally seeing that costs you a signup, a key, a billing page and a nagging worry about a loop you left running. Running the model in the tab deletes all four at once, which is why it goes first.

What “in the tab” actually means

Not a proxy, and not a small server you forgot you started. WebLLM compiles model weights to run on WebGPU — the browser's access to your graphics card. The weights are ordinary files, fetched once over HTTP and kept in the browser's cache. After that first fetch you can turn off your wifi and keep talking.

  1. The runtime arrives a couple of MB of JavaScript and WebAssembly, from a CDN
  2. The weights arrive — once a gigabyte or several, in shards, into the browser cache
  3. The GPU is loaded weights to VRAM, plus a paged cache for the conversation
  4. Tokens come out from here on, nothing touches the network at all
Step 2 is the one you feel: minutes, once per model. Step 3 is the one that fails on a small GPU. Steps 1–3 happen on the first send and never again for that model on that origin — so the second question you ask is fast, and the tenth costs nothing.

The code — one file, no build step

Complete and runnable. Save it as index.html and serve it — python3 -m http.server, then open localhost:8000. It must be served rather than opened as a file:// path: module imports and the cache both need a real origin.

<!DOCTYPE html>
<meta charset="utf-8">
<title>a model in this tab</title>
<button id="go">LOAD AND ASK</button>
<pre id="out">press the button.</pre>

<script type="module">
import { CreateMLCEngine } from
  "https://cdn.jsdelivr.net/npm/@mlc-ai/web-llm@0.2.84/lib/index.js";

const out = document.getElementById("out");
const say = (t) => { out.textContent = t; };

document.getElementById("go").onclick = async () => {

  // ── 1. the engine. The first call downloads the weights; every call
  //       after this one reads them straight out of the cache. ──────────
  const engine = await CreateMLCEngine(
    "Qwen3.5-2B-q4f16_1-MLC",
    { initProgressCallback: (r) => say(r.text) },   // the load messages
    { context_window_size: 8192 },
  );

  // ── 2. the request. The same shape as an OpenAI chat completion,
  //       because that is the shape this engine speaks. ────────────────
  const stream = await engine.chat.completions.create({
    messages: [
      { role: "system", content: "Answer in one sentence." },
      { role: "user", content: "Why is the sky blue?" },
    ],
    stream: true,
    max_tokens: 512,
    // Qwen will otherwise deliberate for a page before answering, and on a
    // 2B in a tab you feel every token of it.
    extra_body: { enable_thinking: false },
  });

  // ── 3. the tokens, as they arrive. Nothing here touches the network. ─
  let text = "";
  for await (const chunk of stream) {
    text += chunk.choices[0]?.delta?.content || "";
    say(text);
  }
};
</script>

Three things worth noticing. The messages array is exactly the shape from the diagram. initProgressCallback is not decoration — without it a five-minute download looks like a hung page, and you will assume you broke something. And the request body is the chat-completions shape, so the code you write here is nearly the code you will write against a hosted model in unit 2.

Keep the tab responsive

Run that and the page freezes in bursts, because decoding tokens is real work happening on the main thread. Move the engine to a worker and the freezing stops. It is one extra file, and no change at all to the code that does the asking:

// ── worker.js — the whole file. It hands every message to web-llm's
//    own handler, which does the work off the main thread. ────────────
import { WebWorkerMLCEngineHandler } from
  "https://cdn.jsdelivr.net/npm/@mlc-ai/web-llm@0.2.84/lib/index.js";

const handler = new WebWorkerMLCEngineHandler();
self.onmessage = (msg) => handler.onmessage(msg);


// ── in the page — one line different from before. ────────────────────
import { CreateWebWorkerMLCEngine } from
  "https://cdn.jsdelivr.net/npm/@mlc-ai/web-llm@0.2.84/lib/index.js";

const worker = new Worker(new URL("./worker.js", import.meta.url),
                          { type: "module" });

const engine = await CreateWebWorkerMLCEngine(
  worker,
  "Qwen3.5-2B-q4f16_1-MLC",
  { initProgressCallback: (r) => say(r.text) },
  { context_window_size: 8192 },
);
// …and engine.chat.completions.create is unchanged. The worker engine
//    is a proxy with the identical interface.

Do this early rather than as a polish pass. On the main thread the tab stops painting while it decodes, so every judgement you make about the model's speed — and about whether it has hung — is wrong.

Keep the engine in a variable and reuse it. Creating a second one for the same model reloads the GPU for no reason, and a model you have finished with should be unload()ed before you load another — VRAM is the resource you will run out of first, and only one model at a time really fits.

What it costs you instead of money

Free is not the same as free. The bill is paid in other currencies, and being surprised by them later is worse than reading them now:

That last one is a feature here, and only here. You are learning what the exchange looks like, and a weak model fails visibly, instantly and for free. The same failures behind a hosted key cost money and hide inside plausible prose.

Two ways to give it tools

Ask this model to check the weather and it will make the weather up. It has no tools yet — and giving it some is where the in-tab runtime gets interesting, because it has two ways to do it and the obvious one is the wrong one.

The obvious one is native: web-llm really does accept a tools parameter, in the same chat-completions shape a hosted vendor uses, and hands back a parsed tool_calls array.

// the native path. Runs, and worth knowing — read the three
// conditions under it before you build on it.
const engine = await CreateMLCEngine("Hermes-3-Llama-3.1-8B-q4f16_1-MLC");

const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "do I need an umbrella in Paris?" }],
  tools: [{
    type: "function",
    function: {
      name: "get_weather",
      description: "current weather for a city",
      parameters: {
        type: "object",
        properties: { city: { type: "string" } },
        required: ["city"],
      },
    },
  }],
});

// → [ { function: { name: "get_weather", arguments: '{"city":"Paris"}' } } ]
reply.choices[0].message.tool_calls;

The three conditions. It works on five model builds — the Hermes-2-Pro and Hermes-3 ones, which were fine-tuned for a specific tool prompt; pass tools with any other id and the request throws rather than degrades. It refuses your own system prompt, because it needs that slot for the tool prompt it writes itself — so a request with a system message and tools together is an error. And it constrains the entire reply to a JSON array of calls: a turn is calls or prose, never a sentence and then a call. (tool_choice is accepted and then ignored, which is worse than rejecting it — check what you got back rather than trusting it.)

An agent needs all three of the things that path takes away: any model, its own system prompt, and a turn that can say something and then act. So the second way is the one you actually use — and it needs nothing from the runtime at all, because tool calling is a convention, not a capability. Describe the convention in the system prompt — to run a tool, reply with a <tool_call> block and end your turn — then find the tags in the stream yourself and act on them:

  1. user — do I need an umbrella in Paris?
  2. assistant — text <tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call>
  3. user — text <tool_response>{"output": "18°C, light rain"}</tool_response>
  4. assistant — Yes — light rain in Paris at 18°C. Take one.
A tool call with the structure taken away: just tags in a string, which your code finds and acts on. This is how the harness drives an in-tab model — any build, its own system prompt intact, prose and calls in one turn — and it is worth seeing once, because when unit 2 hands you a real tool_use block, you will recognise it as the same agreement with the parsing done for you.

The cost of doing it yourself is that nothing is enforced any more. The native path could not return malformed JSON if it tried — the grammar forbids it — whereas here the model is merely asked to keep the shape, and small models keep this bargain badly. They invent parameter names, narrate the call instead of making it, or write the <tool_response> themselves and answer their own question. So the parser is yours to make suspicious: match the tags, parse what is between them, and treat a failure as a message back to the model rather than a crash. Watch it go wrong once and you will understand why unit 2 asks for a capable model.

The exercise

Run the file above, with the console open. Then, before you change anything, three questions you can answer by looking:

Then break it on purpose, which is the half people skip:

You can also do all of this in the harness without writing a file: the completion backend panel has WebLLM — in this tab alongside Ollama, vLLM and the hosted vendors, and the console logs each step of the load, the cached shard count and the tokens per second.

Next week

Unit 2 — tool calling gives the model hands. You hand it a list of functions, it asks for one by name, your code runs the real thing and passes the answer back. That loop is the whole of agents, it is about twenty lines, and it arrives next Tuesday.

Where this sits in the whole course, and what comes after: the syllabus.

If someone forwarded you this, the lessons are free and weekly and the archive keeps the ones you missed:

open the harness

One lesson a week