AI that runs in your browser: what it can and can't do

Not long ago, anything labelled “artificial intelligence” meant sending your data to a server. It was unavoidable: the models didn’t fit anywhere else.

Today, part of it does fit. Removing the background from a portrait, detecting faces, reading text from an image or upscaling a small photo can happen entirely inside a browser tab, without the file leaving your computer.

But the line is real and fairly sharp. It’s worth knowing which side each task falls on.

What had to happen

Three pieces had to come together.

Small models. A segmentation model — the one that separates a person from the background — might be 5 to 50 MB. One of the fashionable language models is tens of gigabytes. That’s three orders of magnitude, and it explains almost everything else.

Quantisation. A model is trained with 32-bit numbers, but running it usually works fine with 8. That cuts the size to a quarter and speeds up computation, at a precision cost that is nearly imperceptible on visual tasks.

GPU access. The browser can use the graphics card through WebGL or, more recently, WebGPU. Without it none of this would be viable: on CPU alone, an image model takes tens of seconds where the GPU takes one.

What already works well

Person segmentation. Separating someone from the background. It’s what powers blurred video-call backgrounds, and it runs in real time.

Face and landmark detection. Fast and reliable. The basis of filters.

Text recognition (OCR). Extracting text from an image. Good with scanned documents, mediocre with skewed or poorly lit photos.

Image upscaling. Enlarging a small photo by inventing plausible detail. It works, but it’s the heaviest of the group: on large images it can take a while and use a lot of memory.

Classification and tagging. Recognising what’s in a photo.

Noise reduction. Cleaning grain from low-light photos.

All of these share a trait: they’re specialised vision models, trained for one specific task. That’s where the browser performs.

What doesn’t fit

Large language models. The ones that converse and write. The smallest useful ones are several gigabytes; the good ones far more. There are experiments loading them in the browser, and they work, but downloading three gigabytes before the first reply isn’t an experience anyone wants twice.

Image generation. Diffusion models need a lot of GPU memory and many computation steps. Reduced browser versions exist, but the output and the wait don’t compete with a server-side service.

Real-time video beyond the basics. Segmenting one frame is viable. Applying a heavy model to thirty frames a second isn’t.

Anything needing your whole library. Searching ten thousand photos, training on your data, comparing against a large database. The browser processes what you hand it, not what you have stored.

The price you pay

Worth being explicit about, because these tools are sometimes presented as costless:

The first load is heavy. The model has to be downloaded. Between 5 and 100 MB depending on the task. It’s cached afterwards, but the first time is noticeable, especially on mobile data.

It depends on your machine. A laptop without a dedicated GPU can take ten times longer than one with. There’s no uniform experience the way there is with a server-side service, where every user shares the same hardware.

It uses your memory. A large image may simply not fit. These tools commonly cap the input size, and that isn’t an arbitrary restriction: past a certain point the browser runs out of memory and fails.

Quality is somewhat lower. A quantised 20 MB model doesn’t match a full 2 GB one. For most uses the difference isn’t visible, but it exists.

What you gain

In exchange there are three advantages that aren’t small:

Your file never leaves your computer. For a photo of a document, of a child, or of material under NDA, that isn’t a detail: it’s the difference between being able to use the tool and not.

There’s no cost per use. A server-side service pays for GPU time on every operation, which is why they all end up with limits, sign-ups or subscriptions. What runs on your machine costs nobody anything.

It works offline. Once the model is loaded, the tool keeps working even if your connection drops. That’s also the most convincing proof that it isn’t sending anything: if it works with the Wi-Fi off, there’s no server behind it.

How to choose

The useful question isn’t “is local or server better?” but what are you processing:

• Personal documents, family photos, confidential work material → local, without hesitation. Privacy outweighs the quality difference.
• A huge image, or maximum quality for professional work → server. Your browser isn’t going to win that comparison.
• Occasional use of something simple → local. It’s faster in practice, because there’s no upload and no queue.
• Generating text or images from scratch → server. There’s no real debate here yet.

Where it’s heading

WebGPU is arriving across browsers and multiplies performance over WebGL. Small models will keep shrinking with better compression techniques. And operating systems are starting to expose their AI accelerators to web applications.

The line will move, and it will move in favour of local. But it won’t disappear: there will always be models too large for a tab, and those will keep needing a server.

What’s interesting is that the part which already fits covers much of what people do with images day to day. And that part can now be done without handing the file to anyone.