Identity Mappings in Deep Residual Networks

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

2016 Computer vision arXiv ArXivIsual PDF

Status:

Importance:

Question

What propagation formulations behind residual blocks enable training of extremely deep networks?

Key Idea

Identity mappings as skip connections + after-addition activation enable direct signal propagation between any two blocks, enabling 1001-layer ResNets.

Concepts

residual-networks skip-connections identity-mappings deep-learning

Prerequisites

Deep Residual Learning for Image Recognition 2015 · Kaiming He

Related Papers

Deep Residual Learning for Image Recognition 2015 · Kaiming He Residual Networks Behave Like Ensembles of Relatively Shallow Networks 2016 · Andreas Veit

cs.CV cs.LG computer-vision deep-learning residual-networks skip-connections

Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet.