Lesson 1, WebLLM: a language model in the tab, no key, no server, no bill
Every agent needs a model to talk to. Most tutorials start by asking for your credit card. We are going to skip that step. WebLLM runs a real language model inside your browser, on your own GPU, with the network off. No account. No key. No server. Nothing to install. By the end of this lesson you will have the one exchange the whole course is built on: messages in, tokens out.
The one idea
A language model is a function. You hand it an array of messages. It hands you back text, one token at a time. That is the whole interface. It does not change when the model gets smarter, when you put it behind an API, or when you wrap it in an agent. Every one of the five lessons after this one is a wrapper around this single call.
So the first question is not “which model” or “which framework.” The first question is: what do I send, and what comes back?
[ { role: "system",
content: "Answer in one sentence." }, { role: "user",
content: "Why is the sky blue?" }]- "Blue"
- " light"
- " scatters"
- " more"
- " than"
- " red"
- "."
Usually you pay for a look at that with a signup, a key, a billing page, and a little worry about the loop you left running. Run the model in the tab and all four go away. That is why we start here.
Why WebLLM is important
Zero friction is not just a nice-to-have. It changes what you can build and who gets to run it.
You start in seconds. No signup. No key to rotate. No rate limit. No dashboard. No meter running while you read the docs. You can get it wrong a hundred times in one afternoon and it costs you nothing. That is how developers learn. When a tutorial opens with a billing page, most people close the tab, and the ones who stay play it safe. Here you can just experiment.
Your data stays on your machine. This is not a privacy policy you have to trust. There is nowhere for the data to go. The weights load into your GPU, and from then on the tab is talking to your graphics card. That is the difference between a demo you can show a hospital, a bank or a law firm and a demo you cannot. Somebody asks where your prompts go, and the answer is: nowhere.
Your costs stay flat. Run an agent on a hosted model and every new user adds to your bill. Run it in the tab and your inference cost is zero, because every user brings their own GPU. Ten users or a hundred thousand users, you pay the same. The web made this trade a long time ago with rendering, and it is how you ship an agent to a lot of people on a hobby budget.
It works offline. Download the model once. After that you can be on a plane, in a basement, or on conference wifi that dies in the middle of your talk, and the model keeps answering. Nothing else in this course does that.
Now the honest part. This is not a frontier model. A 2B model in a tab is small, a little slow, and confidently wrong. That is fine, and it is exactly why we start here. You get to watch every part of an agent break in the open, for free, before lesson 2 gives you a model good enough to hide its mistakes.
What WebLLM is, and what “in the tab” means
WebLLM is an open-source in-browser inference engine from MLC AI. It compiles quantized model weights to run on WebGPU, the browser's access to your graphics card, and puts them behind the same chat-completions API a hosted vendor gives you. It is not a proxy. It is not a little server you forgot you started. The weights are ordinary files. You fetch them once over HTTP and the browser keeps them in its cache. After that first fetch you can turn off your wifi and keep talking.
Here is the whole requirements list: a browser with WebGPU, so Chrome or Edge on the desktop or a recent Safari or Firefox, plus a card with a couple of gigabytes free. That is it.
- The runtime arrives a couple of MB of JavaScript and WebAssembly, from a CDN
- The weights arrive once a gigabyte or several, in shards, into the browser cache
- The GPU is loaded weights to VRAM, plus a paged cache for the conversation
- Tokens come out from here on, nothing touches the network at all
The WebLLM code: one file, no build step
This is complete and runnable. Save it as index.html and serve
it with python3 -m http.server, then open
localhost:8000. Serve it, don't open it as a
file:// path. Module imports and the cache both need a real
origin.
<!DOCTYPE html>
<meta charset="utf-8">
<title>a model in this tab</title>
<button id="go">LOAD AND ASK</button>
<pre id="out">press the button.</pre>
<script type="module">
import { CreateMLCEngine } from
"https://cdn.jsdelivr.net/npm/@mlc-ai/web-llm@0.2.84/lib/index.js";
const out = document.getElementById("out");
const say = (t) => { out.textContent = t; };
document.getElementById("go").onclick = async () => {
// ── 1. the engine. The first call downloads the weights; every call
// after this one reads them straight out of the cache. ──────────
const engine = await CreateMLCEngine(
"Qwen3.5-2B-q4f16_1-MLC",
{ initProgressCallback: (r) => say(r.text) }, // the load messages
{ context_window_size: 8192 },
);
// ── 2. the request. The same shape as an OpenAI chat completion,
// because that is the shape this engine speaks. ────────────────
const stream = await engine.chat.completions.create({
messages: [
{ role: "system", content: "Answer in one sentence." },
{ role: "user", content: "Why is the sky blue?" },
],
stream: true,
max_tokens: 512,
// Qwen will otherwise deliberate for a page before answering, and on a
// 2B in a tab you feel every token of it.
extra_body: { enable_thinking: false },
});
// ── 3. the tokens, as they arrive. Nothing here touches the network. ─
let text = "";
for await (const chunk of stream) {
text += chunk.choices[0]?.delta?.content || "";
say(text);
}
};
</script>Three things to notice. First, the messages array is exactly
the shape from the diagram. Second, initProgressCallback is not
decoration. Without it, a five-minute download looks like a hung page and
you will assume you broke something. Third, the request body is the
chat-completions shape, so this is very close to the code you will write
against a hosted model in lesson 2.
Take a good look at that last block. A stream does not hand you an answer. It hands you fragments. The answer is a variable you build out of them:
- "Blue"
- " light"
- " scatters"
- " more"
- " than"
- " red"
- "."
Nothing happens for a long time, and then everything happens at once. Open devtools on the Network tab for the first load and watch the shards arrive. Reload and watch them not arrive. That is the cache. Two minutes of watching this now will save you an hour of debugging a download you thought had hung.
Keep the tab responsive: WebLLM in a web worker
Run that file and the page freezes in bursts. Decoding tokens is real work, and right now it is happening on the main thread. Move the engine into a worker and the freezing stops. It takes one extra file, and the code that does the asking does not change at all:
// ── worker.js: the whole file. It hands every message to web-llm's
// own handler, which does the work off the main thread. ────────────
import { WebWorkerMLCEngineHandler } from
"https://cdn.jsdelivr.net/npm/@mlc-ai/web-llm@0.2.84/lib/index.js";
const handler = new WebWorkerMLCEngineHandler();
self.onmessage = (msg) => handler.onmessage(msg);
// ── in the page: one line different from before. ────────────────────
import { CreateWebWorkerMLCEngine } from
"https://cdn.jsdelivr.net/npm/@mlc-ai/web-llm@0.2.84/lib/index.js";
const worker = new Worker(new URL("./worker.js", import.meta.url),
{ type: "module" });
const engine = await CreateWebWorkerMLCEngine(
worker,
"Qwen3.5-2B-q4f16_1-MLC",
{ initProgressCallback: (r) => say(r.text) },
{ context_window_size: 8192 },
);
// …and engine.chat.completions.create is unchanged. The worker engine
// is a proxy with the identical interface.Do this early. It is not a polish pass. On the main thread the tab stops painting while it decodes, so every judgment you make about how fast the model is, or whether it has hung, will be wrong.
One engine, one model. Keep it in a variable and reuse it, and call
unload() before you load another one. A second engine for the
same model reloads the GPU for no reason, and only one model fits at a time
anyway. Which brings us to the next section.
What WebLLM costs you instead of money
Free is not free. You pay in other ways, and it is better to know about them now than to be surprised later. The first one is your graphics card. The rule of thumb is easy: at 4-bit, a model costs about half a byte per parameter. So 2B is about a gigabyte and 9B is about five. Then the conversation needs room of its own on top of that.
- The download. That same gigabyte or five, over the wire. Once per model per origin, and slow the first time.
- Context. 8192 tokens here, versus hundreds of thousands on a hosted model.
- Text only. These builds cannot see images. Attach a screenshot and it simply is not there. That is lesson 5's whole subject, and it is why that lesson wants a different model.
- Smarts. A model small enough to run in a tab is not a smart model. It repeats itself, loses the thread, and mangles formats with total confidence.
The missing smarts are the one cost that works in your favor here, and only here. You are learning what the exchange looks like, and a weak model fails right away, in the open, for free. Behind a hosted key, those same failures cost you money and hide inside prose that sounds fine.
Two ways to give WebLLM tools
Ask this model to check the weather and it will make the weather up, because it has no tools yet. Giving it some is where the in-tab runtime gets interesting. There are two ways to do it, and the obvious one is the wrong one.
The obvious one is native. web-llm really does accept a tools
parameter, in the same chat-completions shape a hosted vendor uses, and it
hands back a parsed tool_calls array.
// the native path. Runs, and worth knowing, read the three
// conditions under it before you build on it.
const engine = await CreateMLCEngine("Hermes-3-Llama-3.1-8B-q4f16_1-MLC");
const reply = await engine.chat.completions.create({
messages: [{ role: "user", content: "do I need an umbrella in Paris?" }],
tools: [{
type: "function",
function: {
name: "get_weather",
description: "current weather for a city",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
}],
});
// → [ { function: { name: "get_weather", arguments: '{"city":"Paris"}' } } ]
reply.choices[0].message.tool_calls;It runs. And here are the three conditions that make it the wrong way to go:
| what it takes away | what that means in practice |
|---|---|
| Your choice of model | Five builds support it: the Hermes-2-Pro and Hermes-3 ones,
fine-tuned for one specific tool prompt. Pass tools
with any other id and the request throws instead of degrading. |
| Your system prompt | The runtime needs that slot for the tool prompt it writes itself. A
system message and tools in the same
request is an error. |
| Prose in the same turn | The whole reply is constrained to a JSON array of calls. A turn is either calls or prose, never a sentence and then a call. |
One more gotcha: tool_choice is accepted and then ignored,
which is worse than rejecting it. Check what came back instead of trusting
that it did what you asked for.
An agent needs all three of the things that path takes away: any model, its
own system prompt, and a turn that can say something and then act. So the
second way is the one you will actually use. It needs nothing from the
runtime at all, because tool calling is a convention, not a feature. You
describe the convention in the system prompt. To run a tool, reply with
a <tool_call> block and end your turn. Then you find
the tags in the stream yourself and act on them:
- user: do I need an umbrella in Paris?
- assistant: text
<tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call> - user: text
<tool_response>{"output": "18°C, light rain"}</tool_response> - assistant: Yes, light rain in Paris at 18°C. Take one.
tool_use block you will recognize it as the
same deal with the parsing done for you.The cost of doing it yourself is that nothing is enforced anymore. The native
path could not return malformed JSON if it tried, because the grammar forbids
it. Here you are only asking the model to keep the shape, and small
models are bad at that. They invent parameter names. They describe the call
instead of making it. They write the <tool_response>
themselves and answer their own question. So write a suspicious parser: match
the tags, parse what is between them, and when that fails, send a message back
to the model instead of crashing. Watch this go wrong once and you will know
exactly why lesson 2 asks for a capable model.
The exercise
You do not need to save a file for this part. Open the
harness, or any page you are serving from localhost, and
open devtools on the Console tab. Paste this in and press
enter:
// ── paste 1: the engine and a helper to ask it things. ────────────────
var { CreateMLCEngine } = await import(
"https://cdn.jsdelivr.net/npm/@mlc-ai/web-llm@0.2.84/lib/index.js");
var MODEL = "Qwen3.5-2B-q4f16_1-MLC";
var t0 = performance.now();
var engine = await CreateMLCEngine(
MODEL,
{ initProgressCallback: (r) => console.log(r.text) },
{ context_window_size: 8192 },
);
console.log("loaded in", ((performance.now() - t0) / 1000).toFixed(1), "s");
// one call, streamed, printed as it arrives. Everything below reuses it.
var ask = async (messages, opts = {}) => {
const stream = await engine.chat.completions.create({
messages, stream: true, max_tokens: 512,
extra_body: { enable_thinking: false },
...opts,
});
let text = "";
for await (const chunk of stream) {
text += chunk.choices[0]?.delta?.content || "";
}
console.log(text);
return text;
};
await ask([
{ role: "system", content: "Answer in one sentence." },
{ role: "user", content: "Why is the sky blue?" },
]);Two console details worth knowing, because both of them will bite you. The
top-level await only works in a devtools console, so this
snippet is not the same code as the file above. And every declaration is
var on purpose. You are going to re-paste these lines a dozen
times, and var lets you redeclare while const
argues with you about it. If the import itself fails with a
content-security-policy error, then the page you are on will not load
modules from a CDN. Move to the harness or your own localhost
and try again.
Now, before you change anything, answer three questions:
- How much of that was the download? Drop the engine and
build it again. The weights are cached now, so the second number is the
honest cost of loading the GPU:
The gap between those two numbers is the cache doing its job.await engine.unload(); var t = performance.now(); engine = await CreateMLCEngine(MODEL, {}, { context_window_size: 8192 }); console.log(((performance.now() - t) / 1000).toFixed(1), "s from cache"); - What is the system message worth? Ask the same question
without it:
await ask([{ role: "user", content: "Why is the sky blue?" }]). The answer gets longer and vaguer. That one line was doing real work, and it is the cheapest control you have over any model. - What does it do when it does not know?
await ask([{ role: "user", content: "What is today's date?" }]). Watch it answer anyway, with total confidence. That is the exact hole tools fill, and it is where lesson 2 starts.
Now break it on purpose. This is the half everybody skips, and it is where the lesson really lands:
// ── a model id that doesn't exist. Read the error, you'll meet it again
// the first time you typo a model against a paid API. ───────────────
await CreateMLCEngine("Qwen3.5-2B-q4f16_1-MLC-turbo")
.catch((e) => console.log("→", e.message));
// ── truncation. The reply stops mid-word and NOTHING throws. ──────────
await ask([{ role: "user", content: "Explain how rainbows form." }],
{ max_tokens: 20 });
// ── how fast was that, really? tokens/sec for the last call. ──────────
console.log(await engine.runtimeStatsText());
// ── move up a size. Either it's meaningfully better, or your GPU tells
// you the honest truth about its memory. Unload first: one at a time.
await engine.unload();
engine = await CreateMLCEngine("Hermes-3-Llama-3.1-8B-q4f16_1-MLC",
{ initProgressCallback: (r) => console.log(r.text) });The truncation one is the result to remember. A truncated answer is not an error. The stream just ends, the promise resolves, and your code is holding half a sentence that looks exactly like a whole one. An agent that treats truncation as success is a bug you will hit for real in lesson 3.
Once the weights are cached you cannot re-time the first load, and the first load is the number you want. Devtools → Application → Storage → Clear site data puts you back to zero. That is also how you get those gigabytes off your disk when you are done with a model.
And if you would rather not drive it from a console at all, the harness has the same thing wired up to a panel. The completion backend list carries WebLLM in this tab alongside Ollama, vLLM and the hosted vendors, and it logs every step of the load, the cached shard count, and the tokens per second as it goes.
Questions people ask about WebLLM
- What is WebLLM?
- An open-source in-browser inference engine from MLC AI. It compiles quantized model weights to run on WebGPU and serves them behind the chat-completions API you already know. No server, no key, nothing installed.
- Is WebLLM free?
- Yes, and there is nothing to sign up for, because your own GPU does the inference. You pay in download, VRAM, context window and smarts instead. Here is the full list of what it costs you instead of money.
- Why is WebLLM important?
- You start in seconds, so learning is free and mistakes are cheap. Prompts stay on your machine, so you can work with private data without a policy to trust. Every user brings their own GPU, so your inference cost stays flat as you add users. And it keeps answering with the network off. The longer version.
- Which browsers does WebLLM work in?
- Any browser with WebGPU: current Chrome and Edge on the desktop, and
recent Safari and Firefox. Serve the page over http:// or https://. From
a
file://path, the module imports and the cache both fail. - How much VRAM does WebLLM need?
- About half a byte per parameter at 4-bit, so a gigabyte for a 2B and five for a 9B, plus room for the conversation cache. Overshoot and you lose the GPU device instead of slowing it down.
- Does WebLLM support tool calling?
- Natively on the five Hermes builds only, and it costs you your system prompt and any prose in the same turn. For an agent, describe the convention in the system prompt and parse the tags yourself.
- Does WebLLM work offline?
- After the first load, completely. Runtime and weights are cached once per model per origin, and nothing after that touches the network.
- WebLLM or Ollama?
- Use WebLLM when it has to run for someone who will not install anything: a demo, a lesson, a page you send to a stranger. Use Ollama when it is your own machine and you want a bigger model, one download shared across projects, and no browser in the way. The harness speaks both.
Next week
Lesson 2: tool calling gives the model hands. You hand it a list of functions. It asks for one by name. Your code runs the real thing and passes the answer back. That loop is all an agent is, it takes about twenty lines of code, and it arrives next Tuesday.
Where this sits in the whole course, and what comes after: the syllabus.
If someone forwarded you this, the lessons are free and weekly and the archive keeps the ones you missed: