AI Implementation Glossary: 150 Definitive Terms

Artificial Intelligence (AI) is the field of computer science focused on building systems that perform tasks typically requiring human intelligence — including learning from data, reasoning under uncertainty, recognizing patterns, understanding language, and making decisions.

Machine Learning (ML) is the subfield of AI in which algorithms improve their performance on a task by learning statistical patterns from data, rather than following rules a human wrote by hand.

Deep Learning is a class of machine learning that uses neural networks with many layers ("deep" architectures) to learn hierarchical representations directly from raw data such as images, audio, or text.

A neural network is a machine-learning model composed of layers of interconnected nodes ("neurons") whose numerical weights are adjusted during training so the network maps inputs to desired outputs.

An algorithm is a finite, well-defined sequence of computational steps that takes an input and produces an output, designed to solve a class of problems — for example, sorting a list, ranking search results, or training a neural network.

Training data is the curated dataset used to teach a machine-learning model — every pattern the model can recognize, every bias it inherits, and every limit to its accuracy ultimately traces back to this data.

Inference is the process of running a trained AI model on new, unseen inputs to produce predictions, classifications, or generated content — the part that runs every time a user interacts with the system.

An AI model is the trained artifact — a specific set of numerical weights plus an architecture — produced when a learning algorithm runs against training data, and the thing that actually gets deployed and audited.

Supervised learning is the machine-learning paradigm in which a model learns from training examples paired with correct labels, then predicts labels for new, unseen inputs — the dominant approach in production ML today.

Unsupervised learning is a machine-learning paradigm where a model is given only inputs — no labels — and must discover structure in the data: clusters, density, low-dimensional representations, or anomalies.

A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.

A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.

Generative AI is a class of AI systems that produce new content — text, images, code, audio, or video — by learning the distribution of their training data and sampling from it, rather than classifying or predicting from existing inputs.

The transformer is a neural-network architecture built around the self-attention mechanism that has become the dominant model design for language, vision, audio, and multimodal AI since 2017.

A diffusion model is a generative model that learns to reverse a gradual noising process — starting from random noise and iteratively denoising it into a coherent image, audio waveform, or video — and is the dominant architecture behind modern AI image and video generation.

A multimodal model is an AI model that natively understands or generates more than one type of input — typically text plus images, audio, or video — within a single network rather than via separate task-specific models.

Mixture of Experts (MoE) is a neural-network architecture in which only a small subset of "expert" sub-networks activate for any given input, cutting compute and inference cost dramatically while preserving large total parameter capacity.

A Small Language Model (SLM) is a compact language model — typically under 10B parameters — designed to run cheaply, on-device, or in latency-sensitive workflows where a frontier LLM would be overkill or too expensive.

An attention mechanism is a neural-network operation that lets each output position weigh every input position by learned relevance scores, replacing fixed-window context with content-addressed lookup.

Transformer architecture is the internal design of a transformer model: a stack of identical blocks, each combining multi-head self-attention, a position-wise feed-forward network, residual connections and layer normalization, arranged as an encoder, a decoder or both. For what the transformer is and why it dominates AI, see Transformer.

An encoder-decoder model is a neural architecture with two stacks: an encoder that compresses input into a representation and a decoder that generates output from it, with cross-attention linking the two.

An autoregressive model generates output one token at a time, where each new token is conditioned on every previous token in the sequence, producing text by repeated next-token prediction.

A masked language model (MLM) is a model trained to predict tokens that have been hidden in the input, learning bidirectional context rather than left-to-right next-token prediction.

Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so it specializes in a particular style, domain, or behavior — without retraining from scratch.

Reinforcement Learning from Human Feedback (RLHF) is a training technique that aligns a language model's outputs with human preferences by training a reward model on human comparisons of candidate responses, then optimizing the LLM against that reward.

Pretraining is the first, most compute-intensive stage of training a foundation model — typically self-supervised next-token prediction over trillions of tokens of text, code, and other modalities — that produces the base model later fine-tuned for specific tasks.

Transfer learning is the practice of using a model trained on one task as the starting point for a related task, dramatically reducing the data and compute required to reach high accuracy on the new task.

Few-shot learning is the ability of a model to perform a new task given only a handful of examples at inference time; zero-shot learning is the same idea with no examples — the model relies entirely on its pretraining and the natural-language instruction.

Prompt engineering is the practice of designing the inputs to a language model — instructions, examples, role definitions, output formats, and constraints — to reliably produce a desired output without changing the model's weights.

Chain-of-Thought (CoT) is a prompting and training technique in which a language model is encouraged to reason step-by-step before producing its final answer — improving accuracy on math, logic, and multi-hop reasoning in large models (in Wei et al., 2022, GSM8K math accuracy more than doubled for the largest GPT and PaLM models).

An AI hallucination is when a language or generative model produces content that is plausible-sounding but factually incorrect, fabricated, or unfaithful to its sources — one of the biggest failure modes of LLMs in regulated and high-stakes domains.

Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0–2% quality loss.

Knowledge distillation is the practice of training a smaller "student" model to mimic the outputs of a larger, higher-quality "teacher" model — capturing most of the teacher's quality at a fraction of the inference cost.

Synthetic data is artificially generated training data — produced by AI models or simulators rather than collected from the real world — used to augment, replace, or supplement real datasets while preserving privacy or covering rare cases.

Federated learning is a training paradigm where models are trained across many decentralized devices or organizations — phones, hospitals, banks — by exchanging model updates instead of raw data, so sensitive data never leaves its source.

Model evaluation is the systematic measurement of an AI model's performance, safety, and behavior across representative tasks — using fixed benchmarks, golden datasets, human ratings, and LLM-as-judge methods — both before and after deployment.

A golden dataset is a curated, expert-validated set of inputs and reference outputs that serves as the canonical benchmark for evaluating an AI model on a specific task — the team's "ground truth" for is-this-shippable decisions.

An eval harness is the software framework that runs evaluation tasks against AI models, collects model outputs, applies scoring rubrics, and produces comparable, reproducible metrics across models, prompts, and versions.

ReAct (Reasoning + Acting) is a prompting pattern that interleaves reasoning steps ("Thought") with tool actions ("Action") and tool results ("Observation"), letting an LLM iteratively decompose a task, query the world, and self-correct.

Self-consistency is a prompting technique that samples multiple chain-of-thought reasoning paths from an LLM at non-zero temperature, then selects the most common final answer by majority vote — improving reasoning accuracy at the cost of more tokens.

Tree of Thoughts (ToT) is a reasoning framework where an LLM explores multiple reasoning branches as a tree, generating intermediate "thoughts," evaluating partial states with the model itself, and backtracking — turning generation into deliberate search.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that freezes a pretrained model's weights and injects small trainable low-rank matrices into selected layers — cutting trainable parameters by up to 10,000x and GPU memory by about 3x with negligible quality loss on most tasks.

Model pruning is the technique of removing unnecessary weights, neurons, or structures from a trained neural network to reduce model size and inference cost — often with minimal quality loss when done carefully.

Supervised Fine-Tuning (SFT) is the stage of LLM post-training where a pretrained model is fine-tuned on curated input-output pairs, typically instruction-response demonstrations written by humans.

Data labeling is the process of attaching ground-truth annotations to raw data — text, images, audio — so a supervised model can learn from it, ranging from yes/no classification to structured extraction to multi-turn preference comparisons.

Active learning is a labeling strategy in which the model selects which unlabeled examples a human should label next — typically the cases the current model is least certain about — focusing labeling effort on the most informative examples.

Weak supervision is a paradigm for training models on labels generated programmatically by labeling functions, rules, regex, knowledge bases, or distant heuristics — rather than by hand — and then learning to denoise them.

Data augmentation is the set of techniques that synthetically expand a training dataset by applying label-preserving transformations to existing examples — flips, crops, paraphrases, back-translation, mixup — improving generalization without new labels.

Label noise is the presence of incorrect labels in a training or evaluation dataset, which degrades model accuracy, biases evaluation metrics, and is often mistaken for a model-capability ceiling.

Instruction tuning is fine-tuning a pretrained language model on a dataset of (instruction, response) pairs — often spanning many tasks — to produce a model that follows natural-language instructions zero-shot.

Retrieval-Augmented Generation (RAG) is a technique where a language model retrieves relevant documents from an external knowledge base before generating a response, which reduces hallucination and enables citation of sources.

A vector database is a database optimized for storing and searching high-dimensional vectors (embeddings) by similarity rather than by exact match — the storage layer of every RAG and semantic-search system.

An embedding is a dense numerical vector — typically 384 to 4096 dimensions — that represents the semantic meaning of a piece of text, image, audio, or other content, so that semantically similar items end up near each other in vector space.

A token is the basic unit a language model reads or writes — usually a sub-word fragment (about 4 characters of English text) — and the unit by which API pricing, context-window limits, and inference cost are all measured.

Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.

Model drift is the degradation of an AI model's performance over time as the data it sees in production diverges from the distribution it was trained on — the silent failure mode of every deployed ML system.

MLOps is the discipline of operating machine-learning and AI systems reliably in production — covering data pipelines, model training, deployment, monitoring, drift detection, governance, and incident response — analogous to DevOps for traditional software.

An AI pipeline is the end-to-end sequence of stages that turns raw data into deployed AI predictions or content — ingestion, cleaning, feature engineering, training, evaluation, deployment, monitoring, and retraining — usually orchestrated as code.

An AI agent is an LLM-driven system that, given a goal, plans a sequence of steps, calls tools or APIs, observes the results, and iterates until the goal is reached — going beyond single-turn chat to multi-step autonomous action.

LLM tool use is the pattern in which a language model acts through external functions, APIs, databases or code execution: the model requests a tool, the host application runs it and returns the result, and the model continues, turning a text generator into a controller that can act. Function calling is the structured-output mechanism most tool use is built on.

A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.

A knowledge graph is a structured representation of entities (people, products, customers, drugs) and the relationships between them, stored as a graph and used to ground AI systems in verified, queryable facts rather than free-form text.

A feature store is a centralized data system that stores, versions, serves, and reuses curated machine-learning features — guaranteeing the same feature definitions are used at training time and at online inference time.

A model registry is a versioned catalog that tracks every trained ML model artifact, along with its metadata, training data lineage, evaluation metrics, approval status, and deployment stage (staging, production, archived).

A/B testing for ML is the practice of randomly splitting live traffic between a control model and a candidate model to measure the candidate's causal impact on real business metrics — revenue, conversion, retention, defect rate.

A canary deployment routes a small slice of production traffic (e.g., 1–5%) to a new model version while the majority continues hitting the stable version, surfacing latency, error, and quality regressions before full rollout.

A shadow deployment sends production traffic to a new model in parallel with the live model but never returns the new model's predictions to end users — allowing teams to validate latency, cost, and prediction distribution under real load with zero user risk.

Batch inference runs a model over a large set of inputs offline on a schedule — typically nightly or hourly — optimized for throughput and cost-per-prediction rather than latency.

Online inference returns model predictions synchronously in response to a live user or system request, within an interactive latency budget.

Model monitoring is the continuous measurement of a deployed model's inputs, outputs, and performance signals — including data drift, prediction drift, label drift, latency, and ground-truth accuracy — to detect degradation before it harms business outcomes.

RAG-as-a-Service is a managed offering that handles document ingestion, chunking, embedding generation, vector storage, retrieval, and LLM grounding behind a single API — letting teams ship retrieval-augmented features without building the underlying pipeline.

An agentic workflow is a multi-step AI process where a model plans, takes actions through tools, observes results, and iterates toward a goal — replacing single-shot prompting with a loop that reasons, acts, and self-corrects.

Function calling is an LLM capability where the model — given a schema of available functions — emits a structured invocation (function name and JSON arguments) instead of free-form text, letting downstream code execute the call deterministically.

A multi-agent system orchestrates multiple specialized AI agents — often LLM-powered, with distinct roles, tools, and prompts — that communicate, coordinate, and divide labor to solve a problem larger or more diverse than any single agent could handle reliably.

Dense Passage Retrieval (DPR) is a retrieval method that embeds queries and passages into the same vector space using two BERT encoders (a dual encoder) trained with contrastive loss on (question, positive-passage) pairs.

An embedding model is a neural network that maps text, images, or other inputs into fixed-dimensional vectors where semantic similarity corresponds to geometric closeness (cosine or dot-product distance).

Semantic search is information retrieval that ranks documents by meaning rather than exact-keyword overlap, using vector embeddings of query and documents to measure similarity in a learned semantic space.

Hybrid search combines lexical retrieval (BM25 or sparse vectors like SPLADE) with dense vector retrieval and fuses the results, capturing both exact-match precision and semantic recall.

A re-ranker is a second-stage retrieval model that re-scores a candidate set (typically top 50–100) from a fast first-stage retriever using a more expensive cross-encoder that jointly attends to query and document.

A chunking strategy is the rule set used to split source documents into retrievable units before embedding, which directly determines retrieval recall and answer faithfulness in a RAG system.

Agent memory is the mechanism by which an AI agent stores and retrieves information across turns or sessions, beyond the fixed context window of the underlying language model.

Planner-executor is an agent pattern that separates a high-level planning model — which produces a multi-step plan up front — from a tool-calling executor that runs each step, decoupling strategic reasoning from tactical action.

A tool router is a layer in an agent system that selects which tool (or subset of tools) to expose to the model for a given query, instead of putting all tool definitions in every prompt.

The Model Context Protocol (MCP) is an open standard, introduced by Anthropic in 2024, that lets AI applications connect to external tools, data sources, and services through a common interface — analogous to the Language Server Protocol for editors.

A browser-use agent is an AI agent that operates a real web browser to complete tasks — clicking, typing, scrolling, and reading rendered DOM or screenshots — in place of using APIs.

A code-interpreter agent is an AI agent that writes and executes code (typically Python) in a sandbox to answer questions, run analyses, or transform data — using execution as a tool to escape the limitations of pure text reasoning.

Model latency is the time between request and response for a model inference call. For LLMs it decomposes into time-to-first-token (TTFT, dominated by the prefill stage) and time-per-output-token (TPOT, dominated by autoregressive decoding).

Model throughput is the number of tokens (or requests) per second a serving system produces, aggregated across concurrent users — the metric that determines GPU economics.

Tail latency is the latency at high percentiles (P95, P99, P99.9) of the response-time distribution — the part of the experience that drives user frustration and SLA violations, even when median latency is fine.

Trace sampling is the practice of recording detailed observability traces (spans, prompts, tool calls, token counts) for a fraction of requests, balancing observability cost against debug coverage.

Prompt observability is the practice of capturing, indexing, and analyzing every prompt, response, tool call, latency, token-count, and quality metric in an LLM application — for debugging, evaluation, audit, and continuous improvement.

AI bias is systematic, unfair difference in an AI system's outputs across demographic, geographic, or other groups — usually caused by biased training data, biased labels, or biased problem framing rather than the algorithm itself.

Explainability (XAI) is the property of an AI system whose decisions can be understood by humans — through model-level documentation, prediction-level attributions, and counterfactual explanations — and a regulatory expectation in finance, healthcare, hiring, and other high-stakes domains.

AI governance is the framework of policies, roles, controls, and processes an organization uses to ensure its AI systems are lawful, safe, fair, accountable, and aligned with business intent — across the full lifecycle from problem framing to retirement.

An AI audit is a structured, evidence-based examination of an AI system or AI program against defined criteria — covering training data, model, deployment context, monitoring, and governance — performed by an internal team, an external firm, or a regulator.

The EU AI Act (Regulation (EU) 2024/1689) is the European Union's comprehensive, risk-tiered regulation of AI systems, the world's first horizontal AI law, with obligations phasing in from February 2025: general-purpose AI model rules from 2 August 2025 (Commission fines from 2 August 2026; models already on the market have until 2 August 2027) and, after the Digital Omnibus (Regulation (EU) 2026/1744), high-risk rules from 2 December 2027 (Annex III) and 2 August 2028 (Annex I).

Algorithmic accountability is the principle that a specific person, role, or organization is identifiable and answerable for the design, deployment, outcomes, and harms of an automated decision system — and that the mechanisms to enforce that answerability exist.

Differential privacy is a mathematical framework that bounds how much any single individual's data can influence the output of an analysis or trained model — providing a quantifiable privacy guarantee (epsilon) rather than relying on after-the-fact anonymization.

Factuality is the property of an AI system's outputs being verifiably true with respect to a trusted reference corpus or world knowledge — a distinct dimension from fluency, helpfulness, or generic accuracy.

A jailbreak is an adversarial prompt or technique that bypasses an AI model's safety guardrails to elicit content or behaviors the model was trained to refuse — for example, hazardous instructions, restricted personal data, or off-policy assertions.

Prompt injection is an attack in which adversarial text — placed directly in user input or hidden inside content the model retrieves — overrides developer system instructions and hijacks the model's behavior, exfiltrating data or causing unauthorized actions.

AI red-teaming is the practice of systematically probing an AI system — using adversarial prompts, jailbreak techniques, and edge-case inputs — to surface harmful, unsafe, biased, or policy-violating behaviors before and after deployment.

AI guardrails are the layered controls — input filters, output classifiers, policy engines, schema validation, and structured generation — that constrain an AI system to safe, on-policy, and on-task behavior.

An AI Bill of Materials (AIBOM) is a structured, machine-readable inventory of every component used in an AI system — base models, fine-tuning datasets, third-party APIs, prompts, vector indexes, libraries, and licenses — extending the SBOM concept to AI supply chains.

A model card is a short, structured document that describes an AI model's intended use, performance, limitations, training data, evaluation results across demographic and operational subgroups, and known failure modes — the AI equivalent of a nutrition label.

A datasheet for datasets is a structured document — proposed by Gebru et al. (2018) — describing a dataset's motivation, composition, collection process, labeling, preprocessing, recommended uses, distribution, and maintenance, so downstream model developers can make informed choices.

An AI risk assessment is a structured review of an AI system's potential harms — to individuals, groups, the organization, and society — covering likelihood, severity, affected populations, and mitigation controls across safety, fairness, security, privacy, and compliance dimensions.

The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) is a voluntary U.S. framework for managing AI risks throughout the lifecycle, organized around four core functions — Govern, Map, Measure, Manage — and seven characteristics of trustworthy AI.

Constitutional AI (CAI) is a training technique, introduced by Anthropic in 2022, in which a model critiques and revises its own outputs against a written set of principles, then learns from those revisions instead of from human-labeled preferences alone.

The alignment tax is the performance cost — measured on capability benchmarks — of making a model safer, more honest, or more aligned with human values, relative to the same model trained only for capability.

Scalable oversight is the alignment problem of supervising AI systems on tasks where humans cannot easily check the answer themselves — and the family of techniques being developed to keep oversight tractable as models grow more capable.

Deceptive alignment is a hypothetical failure mode in which an AI system behaves aligned during training and evaluation — because it predicts that misbehavior will be corrected — but pursues different goals once it determines it is no longer being supervised.

Mesa-optimization occurs when a learned model is itself an optimizer — a "mesa-optimizer" — pursuing its own internal "mesa-objective" that may diverge from the training (base) objective the gradient descent process was optimizing.

Refusal training is the post-training step that teaches a model to decline requests that violate its policies (illegal, harmful, privacy-invasive, etc.) — and, critically, to comply with the much larger set of legitimate requests that superficially resemble refused ones.

EU AI Act risk tiers are the four-level classification — unacceptable, high, limited, and minimal/no risk — that determines which obligations apply to an AI system placed on the EU market under Regulation (EU) 2024/1689.

A conformity assessment under the EU AI Act is the procedure by which a high-risk AI system is verified to meet the regulation's requirements (Articles 8–15) before being placed on the EU market or put into service.

A Fundamental Rights Impact Assessment (FRIA) is the structured analysis required by Article 27 of the EU AI Act for certain deployers of high-risk AI systems — bodies governed by public law, private entities providing public services, and deployers of credit-scoring AI or of life- and health-insurance risk-assessment and pricing AI (Annex III points 5(b) and 5(c)).

Data residency is the requirement — imposed by law, regulation, contract, or policy — that data be stored, processed, or accessed only within specified geographic or jurisdictional boundaries.

An AI strategy is a written, board-level plan for how an organization will use AI to create competitive advantage — naming the business goals, prioritized use cases, required capabilities, governance posture, partner choices, and a 12–36 month investment plan.

AI readiness is an organization's practical capacity to deploy and operate AI safely and economically — measured across data foundations, technology stack, talent, governance, and operating model — and the prerequisite to any large AI investment paying off.

AI maturity is a multi-dimensional measure of how systematically an organization develops, deploys, governs, and benefits from AI — typically scored on a 1–5 scale from "ad-hoc experimentation" to "AI-native operating model."

AI ROI is the financial return generated by an AI investment relative to its total cost — including build, inference, MLOps, governance, and change-management cost — and the metric that ultimately determines whether an AI program survives the next budget cycle.

AI-native describes a firm or product designed from inception around AI as the core production function — where AI is not a feature on top of legacy systems but the substrate that data flow, decisions, and value creation are built on.

AI-native consulting is a category of professional services where the consulting firm itself is built around AI engineering capability — designing, deploying, and operating AI systems as core deliverables, not adjacent to a traditional management consulting practice.

An AI adoption framework is a structured, repeatable method for moving an organization from no-AI to systematic AI use — typically across five phases: assess, design, build, govern, and scale — with named owners, gates, and metrics at each step.

An MLOps maturity model is a tiered framework that ranks an organization's ML lifecycle automation — from manual notebook handoffs (Level 0) to fully automated continuous integration, delivery, and training pipelines with automated retraining triggers (Level 4).

Token economics is the practice of modeling AI product costs and margins as a function of input and output tokens consumed per user action — the GenAI equivalent of cloud unit economics, and the single most important number on a CFO's AI dashboard.

Training cost is the total cost — GPU/TPU compute, energy, data acquisition, and labor — required to train a machine-learning model from scratch or to fine-tune a pretrained one to a target capability or domain.

An AI Center of Excellence (AI CoE) is a dedicated cross-functional team that sets standards, builds shared platforms (MLOps, governance, eval), and accelerates AI adoption across business units — combining centralized expertise with federated execution.

Build vs buy in AI is the strategic decision between developing an AI capability internally — model, platform, data layer — and procuring it from a vendor, hyperscaler, or open-source ecosystem, weighed against differentiation, cost, time-to-value, and lock-in.

AI vendor lock-in is the cost and difficulty of switching away from a chosen AI vendor — driven by proprietary APIs, fine-tuned weights, embedding incompatibilities, prompt portability gaps, and integrated platform features that have no clean equivalents elsewhere.

AI Total Cost of Ownership (TCO) is the total cost of an AI system over its full lifecycle — including model and inference costs, infrastructure, integration, data preparation, governance, monitoring, retraining, talent, and exit costs — usually expressed as 3-year fully loaded.

AI use case prioritization is the structured process of ranking candidate AI initiatives — typically across value (revenue, cost, risk reduction), feasibility (data, technology, talent), risk (regulatory, reputational), and strategic fit — to concentrate investment on the small set of bets that will move the P&L.

An AI operating model defines how AI capability is structured across an enterprise — centralized, federated, hub-and-spoke, or platform-plus-product — and how decisions, talent, data, platforms, and accountability flow between the center and business units.

The AI value chain describes the layered set of activities — from chip and energy supply through foundation-model training to applications — that produces deployed AI value, and where economic margin accumulates within that stack.

An AI flywheel is the self-reinforcing loop in which product usage generates proprietary data, that data improves the underlying models, better models attract more usage, and the gap to competitors widens over time.

The build-train-deploy split is the allocation of AI investment across three phases — building infrastructure and data pipelines, training or fine-tuning models, and deploying and operating them — and how that allocation shifts as a portfolio matures.

A GPU (Graphics Processing Unit) is a massively parallel processor that, for AI workloads, executes the matrix multiplications at the heart of neural networks many times faster than a CPU — and the dominant hardware for both training and inference of modern AI models.

A TPU (Tensor Processing Unit) is Google's custom AI accelerator designed specifically for tensor operations — the dense matrix multiplications at the core of neural networks — and used in production to train and serve Google Search, Translate, YouTube, and Gemini.

An inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.

Edge AI is the practice of running AI models on local devices — smartphones, vehicles, sensors, factory equipment, AR/VR headsets — instead of the cloud, in order to deliver lower latency, stronger privacy, and continued operation when offline.

The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.

Model serving is the runtime infrastructure that hosts a trained model and exposes it as an API for low-latency online inference at scale, handling batching, autoscaling, GPU sharing, versioning, and routing.

Speculative decoding is an inference technique where a small "draft" model proposes multiple candidate tokens that the large "target" model verifies in a single forward pass, accepting prefixes that match the target distribution.

Continuous batching is an inference scheduling technique that adds and removes requests from a GPU batch every decoding step, instead of waiting for all requests in a static batch to finish.

Paged attention is a KV-cache memory-management technique that stores attention keys and values in fixed-size blocks, addressed via a page table — analogous to virtual memory in operating systems.

Model parallelism is a distributed-training and inference technique that splits a single neural network across multiple devices when the model does not fit on one device, in contrast to data parallelism (which replicates the model).

Tensor parallelism is a model-parallelism strategy that splits individual matrix multiplications across multiple GPUs along the hidden dimension, recombining results with all-reduce or all-gather collectives.

FlashAttention is an exact attention algorithm that reorders computation to minimize GPU high-bandwidth-memory (HBM) reads and writes, making attention 2–4x faster than standard implementations without changing outputs.

Prefix caching is an inference optimization that reuses the KV-cache computed for a shared prompt prefix — system prompts, few-shot examples, retrieved documents — across multiple requests, eliminating redundant prefill compute.