Tensor Parallelism

Full definition

In a transformer, the QKV projection and feed-forward weight matrices are sharded column-wise across GPUs in the tensor-parallel group; activations are broadcast and partial outputs are summed. Tensor parallelism is bandwidth-bound: every layer requires two all-reduces, so it only scales efficiently within a single NVLink-connected node (typically 8 GPUs). Beyond that, pipeline or data parallelism takes over. TP=8 is the standard configuration for serving Llama 3 70B on a single H100/A100 node.

Why it matters

Tensor parallelism is the reason 8-GPU NVLink nodes (DGX, HGX) command a premium over loose GPU servers. For inference at scale, NVLink interconnect bandwidth — not GPU FLOPS — is often the binding constraint.

Example

Serving Llama 3.1 70B FP16 with TP=4 on 4xH100 fits the 140GB weights with room for KV-cache; TP=8 halves per-GPU memory but adds communication overhead — vLLM benchmarks show TP=4 wins on throughput for typical workloads.

Related terms

• Model Parallelism
  Model parallelism is a distributed-training and inference technique that splits a single neural network across multiple devices when the model does not fit on one device, in contrast to data parallelism (which replicates the model).
• GPU (in AI context)
  A GPU (Graphics Processing Unit) is a massively parallel processor that, for AI workloads, executes the matrix multiplications at the heart of neural networks 10-100× faster than a CPU — and the dominant hardware for both training and inference of modern AI models.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Inference Server
  An inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.