Models & Evaluation

Edge AI

Edge AI runs machine learning inference directly on the device that captures the data, such as a phone, laptop, camera, car, or microcontroller, instead of sending the data to a remote cloud server. It trades raw model size for low latency, offline operation, and keeping data local.

What Edge AI Means

Edge AI is the practice of running a trained machine learning model on or near the device that generates the input data, rather than streaming that data to a centralized cloud data center for processing. The model file lives on the device, and inference, the forward pass that turns an input into a prediction, happens on local silicon: a CPU, a mobile GPU, a digital signal processor, or a dedicated neural processing unit. The term covers a wide span of hardware, from a microcontroller with kilobytes of RAM running a keyword spotter, to a flagship phone running a multi-billion-parameter language model, to a car fusing camera and radar streams in real time.

The defining property is locality. In a cloud setup, the device is a thin client that captures audio, an image, or text and ships it over the network; the heavy computation happens elsewhere. In an edge setup, the device itself holds the weights and does the math. Training almost always still happens in the cloud or a data center, because training a large model needs far more compute and memory than any edge device has. Edge AI is about deployment and inference, not training.

Edge AI sits on a spectrum rather than being a binary choice. Many production systems are hybrid: a small on-device model handles common cases and low-latency interactions, and harder queries are escalated to a larger cloud model. Apple's design, for example, pairs a roughly 3 billion parameter on-device model with a larger server model reached only when needed, so the simplest tasks never leave the phone.

Inference runs on the device that captures the data, not in a remote data center.
Training still happens in the cloud; edge AI concerns deployment and the forward pass.
Targets range from microcontrollers with kilobytes of RAM to phones, laptops, cameras, and cars.
Hybrid designs route easy requests on-device and escalate hard ones to a larger cloud model.

Why Run Models On Device

Four motivations drive edge deployment. Latency is the first: a round trip to a cloud server adds network delay that on-device inference removes, which matters for real-time uses like autocomplete, live translation, camera effects, and driver assistance where a response is needed in milliseconds. Privacy is the second: when raw audio, photos, health readings, or message text never leave the device, the attack surface shrinks and there is no server-side copy to breach, subpoena, or repurpose for training. This is why an app can offer a local-only mode and credibly promise that sensitive content stays on the user's hardware.

Offline operation is the third motivation. A model that runs locally keeps working with no connectivity, on a plane, in a basement, in a rural area, or during an outage. The fourth is cost and bandwidth: cloud inference bills per request and per token, and streaming high-resolution video or audio upstream consumes bandwidth and battery. Pushing inference to the edge moves that recurring marginal cost off the provider's books and off the network entirely, because the compute is already paid for in the device the user owns.

These benefits are why privacy-preserving features increasingly keep data on the device. A note-taking or memory app that processes a user's documents locally can run classification, summarization, or semantic search without ever transmitting the underlying text, which is both a privacy guarantee and a way to avoid per-request inference fees. The same logic explains on-device dictation, photo search by content, and smart-reply suggestions on modern phones.

Latency: no network round trip, enabling real-time autocomplete, translation, and camera effects.
Privacy: raw data never leaves the device, so there is no server-side copy to breach or reuse.
Offline: inference works with no connectivity, on a plane or during an outage.
Cost and bandwidth: avoids per-request cloud fees and upstream data transfer.

The Constraints That Make It Hard

Edge devices are severely resource-constrained compared to a data center. Memory is usually the binding limit: model weights, the key-value cache, activations, and the runtime all have to fit in a few gigabytes of RAM shared with the operating system and other apps, where a cloud server can dedicate tens of gigabytes of high-bandwidth memory per accelerator. A model that is comfortable in the cloud often simply will not load on a phone without compression. Memory bandwidth, the rate at which weights can be streamed to the compute units, frequently bounds token generation speed more than raw arithmetic throughput does.

Compute and power form the second constraint. An edge accelerator delivers a fraction of a data-center GPU's throughput, and unlike a plugged-in server it runs on a battery and has no fan-driven cooling. Sustained inference heats the chip, and thermal throttling then cuts clock speed to protect the device, so a benchmark measured for a few seconds can overstate steady-state performance. Energy per inference matters directly to battery life, which is why dedicated accelerators that do more work per joule are central to edge design.

These limits force a different engineering posture than cloud serving. Rather than scaling out across many servers, edge developers shrink the model until it fits the device, then optimize the runtime and the hardware mapping to hit a latency and power budget. The art of edge AI is getting acceptable quality out of a model small enough and efficient enough to live inside these envelopes.

size_bytes ≈ N_params × (bits_per_weight / 8)

Llama-class 7B in FP16: 7e9 × 2 = 14 GB
Same model in INT4:    7e9 × 0.5 = 3.5 GB

Raw weight memory scales linearly with bits per weight. Moving 7 billion weights from 16-bit to 4-bit cuts the footprint about 4x, the difference between not fitting and fitting on a phone. Real INT4 formats add a small overhead for per-group scales, so the true size is slightly above this floor.

Memory is usually the binding constraint; weights, KV cache, and activations share a few GB of RAM.
Memory bandwidth often limits token generation more than raw arithmetic does.
Battery power and the lack of active cooling cause thermal throttling under sustained load.
Energy per inference directly affects battery life, motivating efficient accelerators.

Techniques That Shrink Models

Quantization is the workhorse. It stores and computes weights, and sometimes activations, at lower numeric precision than the 32-bit floats used in training. Going from FP32 to INT8 cuts the weight footprint by a factor of four, because each weight uses 8 bits instead of 32, and INT4 cuts it by roughly eight. Lower precision also speeds inference, since integer units are cheaper and less data has to move across the memory bus. Quantization-aware training, which simulates the rounding during training, recovers most of the accuracy lost at very low bit widths; Apple's on-device model, for instance, uses quantization-aware training with a mixed 2-bit and 4-bit scheme averaging about 3.7 bits per weight to fit Apple silicon.

Pruning removes weights or whole channels that contribute little, yielding a smaller and sparser network. Knowledge distillation trains a compact student model to imitate a larger teacher's outputs, transferring much of the teacher's behavior into a model small enough for the edge; Microsoft's Phi family of small models was trained heavily on data curated and generated by larger models in this spirit. The complementary move is to start small: small language models in the 1 to 8 billion parameter range, such as Google's Gemma and Microsoft's Phi, are designed from the outset for constrained hardware rather than trimmed down after the fact.

These techniques compose. A typical on-device LLM is a small base model, distilled or trained on high-quality data, then quantized to 4-bit or lower, and finally mapped onto a hardware accelerator. Each step alone gives a modest gain; together they turn a model that needs a server into one that runs on a handset within its memory, power, and latency budget.

Quantization to INT8 cuts weight size 4x versus FP32; INT4 cuts it roughly 8x and speeds inference.
Quantization-aware training simulates rounding during training to recover low-bit accuracy.
Pruning drops low-value weights; distillation compresses a large teacher into a small student.
Small language models like Gemma and Phi are built for constrained hardware from the start.

Hardware And Software Stacks

Modern edge devices ship dedicated neural accelerators, fixed-function units that run low-precision matrix and tensor operations far more efficiently per watt than a general CPU or GPU. Apple's Neural Engine is built into A-series and M-series chips and is reachable through Core ML. Google's mobile tensor blocks accelerate on-device Gemini Nano features, and Google's Edge TPU targets embedded boards. Qualcomm's Hexagon NPU ships in Snapdragon mobile and laptop chips, with the Snapdragon X Elite NPU rated at 45 TOPS. The shared motivation is energy efficiency: an NPU can deliver several times more operations per watt than running the same workload on the GPU.

On the software side, several runtimes turn a trained model into on-device inference. Google's LiteRT, the successor to TensorFlow Lite and renamed in September 2024, runs compact .tflite models on CPU, GPU, and NPU across Android, embedded, and microcontroller targets. Apple's Core ML maps models onto the Neural Engine, GPU, and CPU on iOS and macOS. Microsoft's ONNX Runtime runs ONNX models on many platforms and, through its Core ML execution provider, can dispatch to the Apple Neural Engine. For language models specifically, llama.cpp is a C and C++ engine that runs quantized models stored in the GGUF format on laptops, phones, and CPUs with minimal setup.

Higher-level kits sit on these runtimes. Google's MediaPipe and the newer LiteRT-LM stack provide an on-device LLM inference path, and Apple's Foundation Models framework, introduced with iOS 26, lets developers call the on-device model directly. The practical landscape moves quickly, and brand names and APIs shift, so production teams pin to a specific runtime version and re-verify which accelerator a model actually dispatches to.

python

from llama_cpp import Llama

# Load a 4-bit quantized model in GGUF format from disk.
# n_gpu_layers offloads some layers to the GPU/Metal when available.
llm = Llama(
    model_path="gemma-2b-it-Q4_K_M.gguf",
    n_ctx=2048,        # context window in tokens
    n_gpu_layers=-1,   # -1 = offload all layers the device can hold
)

out = llm(
    "Summarize edge AI in one sentence:",
    max_tokens=64,
)
print(out["choices"][0]["text"])

Running a quantized small model fully on-device with the Python bindings for llama.cpp. The Q4_K_M tag denotes a 4-bit quantization that typically shrinks the weights about 75 percent versus 16-bit with minimal quality loss. Verify the exact model filename against the GGUF you downloaded.

NPUs such as Apple's Neural Engine, Qualcomm Hexagon, and Google's Edge TPU run low-precision math efficiently.
LiteRT, formerly TensorFlow Lite, renamed in September 2024, runs .tflite models on CPU, GPU, and NPU.
Core ML and ONNX Runtime target Apple silicon; ONNX Runtime's Core ML provider reaches the Neural Engine.
llama.cpp runs quantized GGUF language models on CPUs, laptops, and phones.

On-Device LLMs And The Tradeoff

The hardest edge case is the large language model, because even a small LLM dwarfs a vision keyword model. The current generation makes it workable. Google's Gemma 3n, fully released on June 26, 2025, ships in E2B and E4B variants whose memory footprints behave like 2B and 4B models and run in as little as 2 GB and 3 GB of memory respectively. Google's Gemini Nano runs inside Android's AICore service on supported devices, and Apple's roughly 3 billion parameter model runs on iPhone and Mac through the Foundation Models framework offline and free of per-call cost. Microsoft's Phi small models target the same niche on laptops.

The tradeoff against cloud inference is real and worth stating plainly. A cloud model can be far larger and therefore more capable on hard, open-ended, knowledge-heavy tasks, and it can be updated centrally without a device update. An on-device model is smaller and narrower, so it shines at scoped tasks like summarization, rewriting, entity extraction, classification, and short dialog, and it degrades on tasks that need broad world knowledge or long, complex reasoning. Apple is explicit that its on-device model is tuned for everyday tasks and is not meant to be a general-knowledge chatbot.

The right architecture follows from the task. Latency-sensitive, privacy-sensitive, or offline features belong on-device; open-ended reasoning over fresh world knowledge belongs in the cloud. Many shipping products combine both, keeping common interactions local and escalating only when the local model is out of its depth, which preserves privacy and cost savings for the majority of requests while retaining a fallback for the hard ones.

Gemma 3n E2B and E4B (released June 2025) run in as little as 2 GB and 3 GB of memory.
Gemini Nano runs in Android's AICore; Apple's ~3B model runs via the Foundation Models framework.
On-device models excel at summarization, rewriting, and classification, not broad world knowledge.
Hybrid systems keep common requests local and escalate hard ones to a larger cloud model.

Key takeaways

Edge AI runs inference on the device that captures the data; training still happens in the cloud.
Latency, privacy, offline operation, and lower bandwidth and cost are the main reasons to go on-device.
Memory, memory bandwidth, battery power, and thermal limits are the binding constraints.
Quantization (INT8 is 4x smaller than FP32, INT4 roughly 8x), pruning, and distillation make models fit.
NPUs like Apple's Neural Engine, Qualcomm Hexagon, and Google's Edge TPU run on-device math efficiently per watt.
On-device LLMs such as Gemma 3n, Gemini Nano, and Apple's ~3B model handle scoped tasks; cloud handles broad reasoning.

Frequently asked questions

Edge AI runs the model on the local device, so inference happens on the phone, laptop, or sensor that produced the data. Cloud AI sends that data over the network to a remote server that holds the model and returns the result. Edge wins on latency, privacy, and offline use; cloud wins on raw model size and central updates. Many products combine both in a hybrid design.

Through compression. Quantization stores weights at lower precision, so INT8 uses a quarter of the memory of FP32 and INT4 about an eighth. Pruning removes low-value weights, and distillation trains a small student model to mimic a large teacher. Combined with small base models like Gemma or Phi, these techniques bring a multi-billion-parameter model down to a footprint that loads in 2 to 4 GB of memory.

When inference runs fully on-device, the raw input never leaves the hardware, so there is no server-side copy to breach, subpoena, or reuse for training. That is a genuine privacy improvement over cloud inference. The caveat is that some apps are hybrid and escalate certain requests to the cloud, so privacy depends on the specific feature actually running locally rather than the device being capable of it.

Edge inference can run on the CPU, the mobile GPU, or a dedicated neural processing unit (NPU). NPUs such as Apple's Neural Engine, Qualcomm's Hexagon, and Google's Edge TPU are fixed-function accelerators built for low-precision matrix math and deliver far more operations per watt than a CPU or GPU, which is why they handle most sustained on-device AI work.

Common runtimes include Google's LiteRT (formerly TensorFlow Lite) for .tflite models across Android and microcontrollers, Apple's Core ML for the Neural Engine, and Microsoft's ONNX Runtime for ONNX models on many platforms. For language models specifically, llama.cpp runs quantized GGUF models on CPUs, laptops, and phones with minimal setup.

Not for every task. On-device models are smaller and tuned for scoped jobs like summarization, rewriting, entity extraction, and short dialog, and they handle those well. They lack the breadth of knowledge and deep reasoning of a large cloud model, so the practical pattern is to use the local model for common, latency- or privacy-sensitive work and fall back to the cloud for hard, open-ended queries.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free