Running AI on the user's computer: what we learned building Focal
Most AI features send data to a provider’s servers and pay for every request. Focal, the Windows screen recorder we built, does the opposite: it makes captions and checks recordings for private details with AI that runs entirely on the user’s computer. Nothing is uploaded, there is no per-use bill, and it works offline. The trade-off is a one-off model download and less powerful models, which is why on-device AI suits some products and not others. Here is how Focal does it, and how to decide whether your own product should.
Why Focal keeps AI on the computer
People record their screens while they work. Recordings catch email inboxes, customer records, internal dashboards and sometimes passwords. Uploading those to a third party for captions would be a hard sell, and Focal’s promise is that recordings, edits and captions stay on the computer they were made on. There is no account and no telemetry.
That rules out the usual approach of calling a speech recognition API. So Focal ships its AI with it, and the work went into making a small model good enough and fast enough on ordinary hardware.
Captions with a small model
Focal uses Whisper, an open-source speech recognition model, through whisper.cpp, a widely used C++ version that runs well on normal computers. The first time someone makes captions, Focal downloads the model, a 190 MB file, and checks it against a known fingerprint (a SHA-256 checksum) before using it. After that, captions take seconds.
Picking the model was a measured decision, not a guess. We tested three sizes on the same technical script on a 16-thread desktop processor:
| Model | Size | Speed | Technical terms right, with hints |
|---|---|---|---|
| base.en | 148 MB | 27.6x real time | 11 of 12 |
| small.en, 5-bit | 190 MB | 11.2x real time | 11 of 12 |
| large-v3-turbo, 5-bit | 574 MB | 3.7x real time | 10 of 12 |
The largest model was three times slower and no more accurate on what mattered. The small model won, and it is the one Focal uses. (“5-bit” means the model’s numbers are stored with less precision, which makes the file smaller and faster with little loss in quality.)
On computers with a dedicated graphics card, the model runs there through Vulkan, a cross-vendor graphics interface. On an NVIDIA RTX 3090, 29 seconds of speech took 0.84 seconds, against 2.6 seconds on the processor, with the same words and timings. Built-in graphics chips are skipped on purpose: in our tests the one in a desktop Ryzen processor ran at half the speed of the processor beside it.
The screen is the best hint
Speech models know ordinary English well. What they get wrong are names and jargon: “tauri” comes out as “tori”, “useEffect” as “use effect”. In a screen recording, those words are usually on screen at the moment someone says them.
So Focal reads frames of the recording every few seconds with Windows’ own text recognition, which is built into the operating system and also runs locally. It keeps the words that look technical or that the model is likely to find unusual, and gives each stretch of speech the lines those words appeared on as a prompt. The model treats the prompt like something said just before, which nudges its spelling. Words the person typed during the recording count too, when key presses were recorded.
In our end-to-end test this turned four hard terms (“npm run tauri dev”, “useEffect”, “NVENC in HEVC” and “WASAPI loopback”) from wrong to right. It is a good example of a general rule: a small model with the right context often beats a big model without it.
Not giving the model room to invent
Speech models have a known habit of inventing words in silence. Focal finds where speech actually is by loudness against the background noise, and only sends those stretches to the model, in pieces cut at quiet moments. Long pauses cost nothing and leave nothing to invent in. A 71-second recording with a 12-second silence was captioned in 4.3 seconds with nothing made up in the gap.
Finding private details
Focal 1.1 added a privacy check, and it is part of every version since. It reads the recording’s frames about once a second and lists email addresses, phone numbers, card numbers, bank accounts, postal and IP addresses, keys and passwords it sees, with where and when they appeared, so the person can blur them in one click.
Two design choices made it trustworthy:
- Ordinary code checks the AI’s reading. A string of digits only counts as a card number if it passes the check digit test real card numbers pass (the Luhn check), and a bank account only if its own built-in check works out. Text recognition finds candidates, and plain rules decide whether they are real. This is the same layered approach we describe in how we stop AI from making things up.
- Findings are suggestions, not verdicts. The editor lists what it found for the person to keep or drop. A password typed into a field is found from the typing and the “Password” label above the field, and hiding it also blanks the on-screen keystroke display for that stretch.
Like the captions, all of it happens on the computer. The tool that finds sensitive information never sends that information anywhere.
When on-device AI makes sense for your product
On-device AI is not the right default. Cloud models are more capable, need no download and behave the same on every machine. But it is worth considering when several of these are true:
- The data is sensitive. Customer records, health or financial information, internal documents, or anything people would hesitate to upload.
- Usage is high and repetitive. A cloud API charges for every request, as we explain in what it costs to run AI features. A local model costs nothing per use once it is shipped.
- The task fits a small model. Transcription, classification, text recognition and extraction often do. Open-ended writing and complex reasoning usually do not.
- It must work offline or instantly. No network round trip, no outage on someone else’s side.
- You control or know the hardware. Your own staff’s laptops are easier to plan for than every customer’s device.
Many products end up with a mix: private or high-volume steps run locally, and the rare hard task goes to a cloud model with only the data it needs.
Questions to ask before choosing
- What data does the feature touch, and would your customers be comfortable with it leaving their device?
- Have you tested a small model on real examples, with the context it would have in your product?
- What happens on the slowest machine you need to support?
- How will you check the model’s output with ordinary code before anyone relies on it?
Where to start
Collect twenty or thirty real examples of the task and try them on a small open model and on a cloud model. If the small one is close, with the right context, on-device is worth a serious look. Our AI integration service covers designing and building features like these, whether they run in the cloud, on the device or both, and the Focal case study shows the result in a shipped product. You are welcome to get in touch to talk it through.
Questions people ask
What is on-device AI?
On-device AI means the model runs on the user's own computer or phone instead of on a provider's servers. The data never leaves the device, each use costs nothing extra, and it works offline, in exchange for a download, slower hardware and less powerful models.
When should a product use on-device AI instead of a cloud API?
When the data is private or sensitive, when usage is high and per-request fees would add up, when the task fits a small model, or when it has to work offline. Cloud APIs are the better choice for tasks that need the largest models or that run on hardware you cannot predict.
How does Focal make captions without uploading anything?
Focal runs Whisper, an open-source speech recognition model, through whisper.cpp on the user's PC, on a dedicated graphics card when there is one. It reads the recording's screen with Windows' built-in text recognition and gives the words it finds to the model as hints, so names and technical terms are spelled correctly.