2015Model

ResNet: Residual Networks Breakthrough

Microsoft Research proposes ResNet, solving deep network training difficulties through residual connections.

#cnn #deep-learning #resnet #residual-learning #microsoft

Overview

In 2015, Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun at Microsoft Research proposed Residual Networks (ResNet). ResNet introduced "skip connections", allowing network layers to learn residual mappings instead of direct mappings, effectively solving the degradation problem in deep neural networks. ResNet-152 reached 152 layers deep, winning the ImageNet competition with 3.57% error rate, first surpassing human-level performance (approximately 5%). Residual connections became a standard component in deep learning architectures, enabling networks to easily scale to hundreds or even thousands of layers, profoundly influencing subsequent neural network design.

Study Notes

Background, mechanics, impact, and key takeaways in one read

About 2 min read

Background & Context

As AlexNet, VGG, and GoogLeNet pushed networks to 19–22 layers, researchers hit a counterintuitive wall: making them deeper raised training error instead of lowering it — not overfitting but the "degradation problem" (an optimization difficulty). In 2015, Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun at Microsoft Research Asia proposed residual learning: using skip connections so the network learns residuals directly, successfully training 152-layer and deeper nets. It won ILSVRC 2015 classification (3.57% top-5 error) and earned the CVPR 2016 Best Paper Award.

Key Points

1He et al. (2015) proposed ResNet, using residual connections to solve deep-net degradation.
2Core formula y = F(x) + x: a block learns the residual from input to output, not the full map.
3Identity skip connections let gradients flow back through shortcuts, easing optimization.
4It trained 152-layer nets and won ILSVRC 2015 with 3.57% top-5 error.
5The residual connection became a universal building block in CNNs and Transformers alike.
6The paper is regarded as among the most-cited research articles of the 21st century.

How It Works

The key ResNet idea is to replace learning a full mapping H(x) with learning the residual F(x) = H(x) − x, then add the input x back via an identity skip connection: y = F(x) + x. Why does it work? If the ideal mapping is near identity, driving F(x) toward zero is far easier than forcing an ordinary layer to approximate identity, so a deeper net is at least no worse than a shallow one. The skip connection also provides a "highway": gradients can flow back to shallow layers almost intact, avoiding the gradient decay of deep backpropagation. The block itself is ordinary stacked convolutions/fully-connected layers plus this additive shortcut. The idea echoes the intuition that identity mapping is a good starting point — the network only needs to learn "how much to correct the identity." ResNet spawned standard depths (18/34/50/101/152) and became a backbone for detection and segmentation.

Impact & Legacy

ResNet cleared the optimization barrier to depth, proving nets can reach hundreds or even thousands of layers, directly enabling deeper vision models and sweeping ILSVRC/COCO 2015 tasks. Residual/skip connections became standard in almost all later architectures, including Transformers (around multi-head attention and feed-forward sub-layers), far beyond vision. Limitations: extremely deep nets (1000+ layers) show diminishing returns and higher cost; but the principle of "preserving gradients via identity shortcuts" is now deep-network design common sense.

Key Takeaways

Deeper-is-worse is often an optimization problem, not a capacity or fitting problem.
Learning a residual is easier to optimize than learning a full mapping.
Identity skip connections preserve gradients and are a universal modern trick.
ResNet proved depth itself is a powerful resource once optimization barriers are removed.

Glossary

Degradation Problem退化问题

The phenomenon where adding depth raises training error, caused by optimization difficulty rather than overfitting or vanishing gradients.

Skip / Shortcut Connection跳跃连接

A connection adding a layer's input directly to a later layer's output, letting gradients flow via an identity path; the core of residual learning.

Related Events

2012