Masked Language Model

Full definition

BERT (Devlin et al., 2019) introduced the MLM objective: randomly mask 15% of tokens and train the model to recover them from surrounding context. Because attention is bidirectional, MLMs build representations informed by both left and right context — ideal for classification, named-entity recognition, and embedding generation. RoBERTa, DeBERTa, and modern embedding models use MLM-style pretraining. MLMs are not naturally generative; for generation, decoder-only autoregressive models dominate.

Why it matters

Most production embedding models in vector search are MLM-style encoders. Buyers selecting a retrieval stack should know whether the embedding model is MLM-pretrained (typical) or contrastively fine-tuned on top, since this drives semantic-search quality.

Example

BERT-base, fine-tuned on a regulatory-text dataset, classifies clauses as "data-processing" vs "liability-cap" with 94% F1 — a task where bidirectional context matters more than generation.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Encoder-Decoder
  An encoder-decoder model is a neural architecture with two stacks: an encoder that compresses input into a representation and a decoder that generates output from it, with cross-attention linking the two.
• Embedding Model
  An embedding model is a neural network that maps text, images, or other inputs into fixed-dimensional vectors where semantic similarity corresponds to geometric closeness (cosine or dot-product distance).
• Autoregressive Model
  An autoregressive model generates output one token at a time, where each new token is conditioned on every previous token in the sequence, producing text by repeated next-token prediction.
• Pretraining
  Pretraining is the first, most compute-intensive stage of training a foundation model — typically self-supervised next-token prediction over trillions of tokens of text, code, and other modalities — that produces the base model later fine-tuned for specific tasks.

More models terms

• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.
• Generative AI
  Generative AI is a class of AI systems that produce new content — text, images, code, audio, or video — by learning the distribution of their training data and sampling from it, rather than classifying or predicting from existing inputs.