Encoder-Decoder

Full definition

Encoder-decoder transformers (T5, BART, the original 2017 transformer) excel at sequence-to-sequence tasks where input and output have different shapes — translation, summarization, structured extraction. Encoder-only models (BERT) are best for classification and embedding. Decoder-only models (GPT, Llama) are best for open-ended generation. The choice of stack determines latency, training objective, and the kind of fine-tuning that works.

Why it matters

Vendor pitches often blur architectural details. Knowing whether a model is encoder-only, decoder-only, or encoder-decoder tells procurement which workloads it suits — encoder-decoder for translation pipelines, decoder-only for chat, encoder-only for retrieval embeddings.

Example

Google T5 is an encoder-decoder transformer that frames every NLP task as text-to-text — translation, classification, summarization all use the same model with different prompts.

Related terms

• Transformer Architecture
  The transformer architecture is a neural-network design built on stacked self-attention and feed-forward layers, with no recurrence or convolution, that processes sequences in parallel.
• Autoregressive Model
  An autoregressive model generates output one token at a time, where each new token is conditioned on every previous token in the sequence, producing text by repeated next-token prediction.
• Masked Language Model
  A masked language model (MLM) is a model trained to predict tokens that have been hidden in the input, learning bidirectional context rather than left-to-right next-token prediction.
• Embedding Model
  An embedding model is a neural network that maps text, images, or other inputs into fixed-dimensional vectors where semantic similarity corresponds to geometric closeness (cosine or dot-product distance).