Edge AI

Full definition

Edge AI is enabled by efficient model architectures (small language models, distilled models, quantized models), specialized accelerators (Apple Neural Engine, Qualcomm Hexagon, Google Edge TPU, NVIDIA Jetson), and runtimes (Core ML, TensorFlow Lite, ONNX Runtime, MLC). Apple Intelligence, Google Pixel's Tensor models, and Tesla FSD are major consumer-scale edge-AI deployments.

Why it matters

Edge AI changes the privacy and cost equation: data never leaves the device, no per-query cloud cost, no roundtrip latency. For regulated industries (healthcare, defense, automotive) and consumer products with privacy as a feature, edge AI is increasingly the default rather than the exception.

Example

Apple Intelligence runs many AI features (summarization, writing tools, smart replies) on a 3B-parameter on-device model, falling back to Apple's Private Cloud Compute or third-party LLMs only for tasks that exceed the on-device model's capability.

Related terms

• Small Language Model (SLM)
  A Small Language Model (SLM) is a compact language model — typically under 10B parameters — designed to run cheaply, on-device, or in latency-sensitive workflows where a frontier LLM would be overkill or too expensive.
• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0-2% quality loss.
• Distillation
  Knowledge distillation is the practice of training a smaller "student" model to mimic the outputs of a larger, higher-quality "teacher" model — capturing most of the teacher's quality at a fraction of the inference cost.
• Inference
  Inference is the process of running a trained AI model on new, unseen inputs to produce predictions, classifications, or generated content — the part that runs every time a user interacts with the system.
• GPU (in AI context)
  A GPU (Graphics Processing Unit) is a massively parallel processor that, for AI workloads, executes the matrix multiplications at the heart of neural networks 10-100× faster than a CPU — and the dominant hardware for both training and inference of modern AI models.