On-device AI means AI inference—running a model to generate an output—happens locally on your phone, tablet, or laptop rather than sending your inputs to a remote server. That’s the practical definition that matters to users. You type, speak, or show the camera something; the device processes it; you get an answer. No upload required.

Training is a separate story. Many companies still train models in centralized data centers. Others use privacy-preserving methods where learning happens across devices, with only aggregated updates sent back. Confusing inference with training is one of the easiest ways to misunderstand a product’s privacy claims.

The surge in on-device AI from 2024 through 2026 has a tangible hardware driver: modern consumer devices increasingly ship with NPUs (neural processing units) built for machine-learning workloads. Microsoft’s guidance for Copilot+ PCs ties new Windows AI features directly to NPUs capable of 40+ TOPS—tera operations per second—a clear performance bar that signals how central local inference has become to the platform story. Microsoft’s own documentation frames NPU-class hardware as a prerequisite for “new Windows AI features.” (Microsoft Learn)

For readers, the appeal is less technical and more visceral:

- Lower latency (responses feel immediate)
- Offline operation (features still work on a plane or subway)
- Privacy by architecture (inputs need not leave the device)
- Power tradeoffs (local compute can save network costs but draw battery)

Many products label an experience “on-device” even when only part of it runs locally. A feature might handle simple steps locally, then send harder parts to a server. That can be reasonable. It also means the real privacy story hinges on when data leaves the device and what constraints apply when it does.

“AI learns from you” gets tossed around as if it’s a single mechanism. In practice, there are at least three distinct ideas—and only one of them is what most people mean by “on-device.”

On-device inference is the everyday workhorse: the model runs locally, and your text, audio, or images need not be sent to a server to produce an output. Common examples include:

- Live captions
- Voice recognition
- Image enhancement
- Smart replies

This is the easiest to understand, and the easiest to test. Turn on airplane mode. Does the feature still work? That won’t prove everything, but it’s a revealing start.

Federated learning (FL) changes the training pipeline. Instead of uploading raw user inputs, devices compute model updates locally and send only those updates for aggregation. Done correctly, the server never sees raw user text or audio.

Google has described federated learning as a default approach for training some on-device language models in products such as Gboard. In a Google Research post on “private training for production on-device language models,” the company frames FL as a way to learn from user interactions without collecting the underlying content. (Google Research)

Differential privacy (DP) is not a product feature; it’s a mathematical guarantee. DP adds calibrated noise so aggregate statistics—and models trained on them—become less likely to reveal information about any one person.

Google Research has written about deploying DP in production for Gboard-related training, including an example of a 2022 Spanish language model trained with a formal DP guarantee, and it describes DP as “by default” for future launches of certain Gboard neural language models trained on user data. (Google Research)

Apple’s research makes a crucial point: federated learning alone doesn’t guarantee privacy. In a July 2024 paper, Apple researchers argue that FL can still leak information and motivate DP for robust guarantees in federated settings. (Apple Machine Learning Research)

The most important design question in consumer AI right now is deceptively simple: Which requests stay on the device, and which get routed elsewhere? That decision—often called routing—determines whether the user’s prompt, image, or context is processed locally or sent to a cloud model.

Routing is not inherently bad. Local models have limits. Some tasks demand more compute, more memory, or access to up-to-date information. A practical system may need to escalate. The risk is opacity: users may not know when escalation happens, what gets sent, and whether the off-device environment has meaningful privacy protections.

Apple has positioned on-device processing as a cornerstone of Apple Intelligence. When more capacity is needed, Apple says requests can be routed to Private Cloud Compute (PCC) while sending only the data relevant to the task. (Apple Newsroom)

That phrase—“only the data relevant to the task”—is a promise worth interrogating. It implies minimization by design: fewer inputs transmitted, smaller exposure if something goes wrong, and less temptation to retain data for unrelated purposes. It also raises a practical question: can independent experts verify the claim?

Apple says yes. Its newsroom announcement describes PCC as designed so independent experts can verify the protections. (Apple Newsroom)

Many privacy assurances in tech are policy-based: “trust us, we won’t.” Apple’s PCC messaging, at least as described publicly, leans toward architecture plus verifiability. Apple’s security team has also described a Private Cloud Compute Security Guide, a PCC-focused security research program, and a Virtual Research Environment (VRE) intended to help researchers inspect and validate protections. (Apple Security)

That emphasis matters because routing decisions happen at scale. When millions of requests are eligible for cloud escalation, privacy becomes a systems property. It can’t rest solely on good intentions.

Apple’s security blog frames Private Cloud Compute as an attempt to bring iPhone-like security assumptions into the data center. The company describes several elements that are specific enough to evaluate, even if outsiders will still want deeper technical artifacts.

Apple says PCC uses custom Apple silicon servers with concepts associated with device security such as Secure Enclave and Secure Boot. (Apple Security) The idea is familiar: if you can establish a chain of trust from hardware upward, you can reduce the risk of tampering and limit what software is allowed to run.

Apple describes a hardened OS designed with a deliberately narrow attack surface. (Apple Security) In security engineering, narrowing the attack surface is not marketing—it is one of the few strategies that consistently scales. Fewer services, fewer exposed interfaces, fewer opportunities for compromise.

One of the more striking claims: Apple says PCC removes typical data-center admin tooling—like remote shells—and replaces them with more limited, deterministic operational metrics. (Apple Security) That choice cuts against decades of data-center culture, where remote access is convenience and control. It also aligns with a privacy posture: if humans can’t casually log in, humans can’t casually inspect.

Apple’s posture invites comparison not just with rivals, but with itself. Device security has long been Apple’s rhetorical strong suit. PCC attempts to extend that strength to cases where on-device inference isn’t enough—while acknowledging that “cloud” doesn’t have to mean “open season.”

Google approaches privacy for certain on-device experiences with a different emphasis: not just where inference happens, but how models are trained when user interactions matter.

Google has positioned federated learning as a pathway to improve models without collecting raw user data—an approach especially relevant for keyboard suggestions, personalization, and language modeling on devices. In its public research communications, Google describes FL-based production training for on-device language models associated with products like Gboard. (Google Research)

The premise is pragmatic: the company can still learn from what works and what doesn’t, but it doesn’t need to ingest the underlying private text to do so.

Google Research goes further by describing the use of differential privacy in production training for Gboard models, including:

- A 2022 Spanish language model trained with a formal DP guarantee
- A statement that DP is “by default” for future launches of certain Gboard neural language models trained on user data (Google Research)

DP’s strength is that it is legible. You can argue about parameters and implementation, but the concept is not “trust us.” It’s “here’s the guarantee.” Apple’s research echoes why that matters: FL alone can reduce data collection yet still fail to guarantee privacy without additional protections like DP. (Apple Machine Learning Research, July 2024)

Where Apple focuses on hardened compute environments for cloud escalation, Google’s research messaging highlights methods that aim to limit what training can reveal about individuals—even when learning is continuous.

A useful way to evaluate on-device AI is to ignore the label and ask what it changes in your daily life.

Local inference can be fast because it avoids round trips to a server. It can also remain available when connectivity is poor. These are not abstract benefits; they’re the difference between a feature you use and one you forget exists.

Practical test: try a feature with airplane mode enabled. If it still performs core functions, odds improve that inference is local—though some products cache or degrade gracefully.

NPUs exist because AI workloads can be power-hungry. Running a model locally shifts cost from network to compute. Specialized silicon can make that efficient, but “efficient” is not “free.” The best implementations will feel invisible; the worst will drain your battery for novelty.

Microsoft’s NPU threshold—40+ TOPS—is a reminder that vendors expect meaningful local compute demand. (Microsoft Learn) Hardware requirements are not just about features; they’re about making those features tolerable on a battery.

On-device inference can minimize what leaves your device. Federated learning can reduce what gets collected for training. Differential privacy can limit what can be inferred. None of these guarantees that nothing leaves your device.

The deciding factor is often routing: what triggers cloud escalation, and what safeguards exist when it happens? Apple’s answer is PCC with verifiability claims and hardened infrastructure. (Apple Newsroom; Apple Security) Google’s answer for certain domains includes FL and DP to reduce exposure in training. (Google Research)

Different strategies, same underlying pressure: users want personalization and power without surveillance.

You don’t need to be a cryptographer to evaluate AI privacy claims. You need a few disciplined questions—and a willingness to treat ambiguity as meaningful.

When a company says “on-device,” ask:

- What runs locally, exactly—everything or a subset of tasks?
- What triggers routing to the cloud?
- What data is sent when routing happens (the whole prompt, or “only the data relevant to the task”)? (Apple Newsroom)
- Does the company publish technical documentation or research that can be scrutinized?
- Are there independent verification pathways? Apple claims PCC is designed for independent verification and offers a research program and VRE. (Apple Newsroom; Apple Security)

If you’re buying hardware during the 2024–2026 transition, a few implications stand out:

- Local AI performance is now a spec-worthy capability. Microsoft explicitly ties Windows AI features to NPUs at 40+ TOPS. (Microsoft Learn)
- Privacy differences will increasingly come from routing and training design. Look for companies that explain both.
- “Works offline” is the most user-visible proxy for local inference. Not perfect, but informative.
- “Federated learning” and “differential privacy” are meaningful terms when backed by published methods. Google and Apple have both published research describing these approaches. (Google Research; Apple Machine Learning Research)

A sober view helps. Cloud models will remain part of the picture for the foreseeable future. The goal isn’t a world with no cloud, but a world where cloud use is minimized, controlled, and verifiable.

On-device AI is often pitched as convenience: faster responses, fewer loading spinners, features that work without a connection. The more profound shift is governance. Local inference changes who has access to your data by default. Federated learning and differential privacy change how models can improve without turning personal behavior into a dataset.

The industry’s next privacy fight won’t be over whether your phone can run a model. It already can. The fight will be over routing—over the moments a device decides local isn’t enough, and over whether the off-device path is engineered to deserve your trust.

Apple is betting that hardened, verifiable cloud infrastructure—Private Cloud Compute—can make escalation compatible with privacy. Google is betting that privacy-preserving training methods like federated learning and differential privacy can let products improve without collecting raw user data. Both strategies reflect the same reality: the old cloud-first default is no longer a comfortable answer.

If you want a single rule as a reader, make it this one: treat “on-device” as the start of a question, not the end of it.