Masked Language Model

Full definition

BERT (Devlin et al., 2019) introduced the MLM objective: randomly mask 15% of tokens and train the model to recover them from surrounding context. Because attention is bidirectional, MLMs build representations informed by both left and right context — ideal for classification, named-entity recognition, and embedding generation. RoBERTa, DeBERTa, and modern embedding models use MLM-style pretraining. MLMs are not naturally generative; for generation, decoder-only autoregressive models dominate.

Why it matters

Most production embedding models in vector search are MLM-style encoders. Buyers selecting a retrieval stack should know whether the embedding model is MLM-pretrained (typical) or contrastively fine-tuned on top, since this drives semantic-search quality.

Example

BERT-base, fine-tuned on a regulatory-text dataset, classifies clauses as "data-processing" vs "liability-cap" with 94% F1 — a task where bidirectional context matters more than generation.

Related terms

• Encoder-Decoder
  An encoder-decoder model is a neural architecture with two stacks: an encoder that compresses input into a representation and a decoder that generates output from it, with cross-attention linking the two.
• Embedding Model
  An embedding model is a neural network that maps text, images, or other inputs into fixed-dimensional vectors where semantic similarity corresponds to geometric closeness (cosine or dot-product distance).
• Autoregressive Model
  An autoregressive model generates output one token at a time, where each new token is conditioned on every previous token in the sequence, producing text by repeated next-token prediction.
• Pretraining
  Pretraining is the first, most compute-intensive stage of training a foundation model — typically self-supervised next-token prediction over trillions of tokens of text, code, and other modalities — that produces the base model later fine-tuned for specific tasks.