🔤BERT

NEW

Also known as: Bidirectional Encoder Representations from Transformers

Transformer-based language model pre-trained bidirectionally using masked language modeling and next sentence prediction.

DESCRIPTION

BERT is a bidirectional Transformer encoder pre-trained on large corpora with masked language modeling and next sentence prediction objectives. Its contextual embeddings enable strong performance on diverse NLP tasks via fine-tuning. BERT’s architecture focuses on self-attention without autoregressive decoding. Derivatives like RoBERTa refine training objectives, while DistilBERT reduces size for efficiency. BERT popularized transfer learning in NLP and remains a baseline for many benchmarks.

ALTERNATIVE DEFINITIONS

Devlin et al. (2018)

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.

Original introduction and naming of the model.

Source

Google AI Blog

BERT is a method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks.

Emphasizes state-of-the-art results across many NLP tasks.

Source

Hugging Face

BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion.

Tooling-focused definition highlighting self-supervised pretraining.

Source

RELATED TERMS

Transformer Model Self-Attention Masked Language Modeling Next Sentence Prediction DistilBERT Sentence-BERT Fine-Tuning

COMPETING TERMS

RoBERTa GPT XLNet

RELATIONS

• Broader: Transformer Model
• Narrower: ALBERT

KNOWLEDGE AREAS

AI Data Science

TARGET AUDIENCE

AI Engineer Data Scientist MLOps Engineer AI Product Manager

EXTERNAL SOURCES

• Wikipedia - BERT (language model)
• arXiv - Original BERT paper
• Google Research - Official BERT code and checkpoints
• Hugging Face - BERT model usage in Transformers

TOP QUESTIONS

What makes BERT bidirectional?

It conditions on both left and right context simultaneously via self-attention, unlike left-to-right or right-to-left models.

Category: Concept

How is BERT pre-trained?

With masked language modeling and next sentence prediction on large corpora like Wikipedia and BooksCorpus.

Category: Definition

How do you fine-tune BERT?

Add a task-specific head and train on labeled data with modest learning rates and sequence lengths.

Category: Practice

How does BERT compare to GPT?

BERT is an encoder-only bidirectional model; GPT is decoder-only, autoregressive, optimized for generation.

Category: Comparison

What are common BERT variants?

RoBERTa (improved pretraining), DistilBERT (compressed), ALBERT (parameter sharing), and multilingual BERT.

Category: Overview

What tasks benefit from BERT?

Text classification, QA, NER, sentence similarity, and many other discriminative NLP tasks.

Category: Application

What are BERT’s limitations?

Quadratic attention limits long sequences; it’s not optimized for open-ended generation.

Category: Risk

How large should the fine-tuning dataset be?

BERT adapts well with thousands of examples, though more data typically improves robustness.

Category: Requirements

How do you deploy BERT efficiently?

Use distillation, pruning, quantization, or smaller variants, and batch inference to improve throughput.

Category: Operations