2018Model

BERT: Bidirectional Language Understanding

Google proposes BERT, achieving breakthrough language understanding through bidirectional Transformers.

#nlp #bert #transformer #pre-training #google

Overview

In October 2018, Jacob Devlin and colleagues from Google AI Language released BERT (Bidirectional Encoder Representations from Transformers). BERT uses bidirectional Transformer encoders, learning deep bidirectional language representations through two pre-training tasks: "Masked Language Model" and "Next Sentence Prediction." BERT achieved state-of-the-art results on 11 NLP tasks, pioneering the paradigm of pre-trained language models. Its "pre-training + fine-tuning" approach became the standard pipeline in NLP, profoundly influencing later language model development.

Study Notes

Background, mechanics, impact, and key takeaways in one read

About 2 min read

Background & Context

In 2018, pretrained language models became mainstream: ELMo used bidirectional LSTMs for context-aware embeddings, and ULMFiT showed language models could be fine-tuned. But OpenAI's GPT-1 read only left-to-right, unable to use both sides of a word's context at once. At Google AI Language, Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova submitted BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding in October 2018 (NAACL 2019 Best Long Paper). BERT pretrains a deep bidirectional Transformer encoder so every token sees both left and right context. It pushed the pretrain-then-fine-tune paradigm to the top and quickly became the default NLP baseline.

Key Points

1It uses an encoder-only Transformer to build genuinely deep bidirectional context representations.
2Masked language modeling masks tokens and predicts them from context, beating the unidirectional ceiling.
3Next sentence prediction learns sentence-pair relations for QA, inference, and similar tasks.
4After pretraining, only a small output layer is added and fine-tuned—no per-task architecture redesign.
5It set new state of the art on 11 NLP tasks including GLUE and SQuAD.
6Released as a universal baseline, it spawned RoBERTa, ALBERT, DeBERTa, and many variants.

How It Works

BERT keeps only the Transformer's encoder stack and pretrains on two self-supervised tasks. The first is masked language modeling (MLM): about 15% of WordPiece tokens are randomly masked, and the model predicts them from both sides of the context—so every word absorbs bidirectional information. The second is next sentence prediction (NSP): given two sentences, the model judges whether they are adjacent in the source text, learning inter-sentence relations. Inputs begin with [CLS], use [SEP] to separate sentences, and sum token, segment, and position embeddings. After pretraining, only a task-specific output layer is added and fine-tuned; the body stays fixed. BERT-Base has 12 layers and 110M parameters; BERT-Large has 24 layers and 340M, both trained on BooksCorpus and English Wikipedia.

Impact & Legacy

BERT made pretrain-then-fine-tune the standard NLP pipeline and entered production fast: Google Search began using it for English queries in October 2019 and covered 70+ languages by year end. Its bidirectional idea inspired many successors such as RoBERTa, which dropped NSP. Debates note that NSP proved to add little, and that large models raise compute barriers and can be unstable on small data. Still, BERT defined the direction of the foundation-model era.

Key Takeaways

Bidirectional context beats unidirectional; MLM is the trick to obtain it.
Pretraining learns general language ability once; downstream needs only light adaptation.
Interface design like [CLS]/[SEP] decides how many tasks a model can serve.
Even later-discarded designs such as NSP carry research value.

Glossary

Masked Language Model (MLM)掩码语言模型

A pretraining objective that masks some input tokens and has the model predict them from context.

Next Sentence Prediction (NSP)下一句预测

A binary pretraining task judging whether two sentences are adjacent in source text, to learn inter-sentence relations.

Foundation Model基础模型

A large model pretrained on massive unlabeled data that can be adapted to many downstream tasks.

Related Events

2017