LSTM Neural Network
Hochreiter and Schmidhuber propose LSTM, solving the long-term dependency problem of recurrent neural networks.
Overview
In 1997, Sepp Hochreiter and Jürgen Schmidhuber published their groundbreaking paper on Long Short-Term Memory networks (LSTM). LSTM solved the vanishing gradient problem of standard recurrent neural networks by introducing gating mechanisms (input gate, forget gate, output gate) and memory cells, enabling the network to learn long-term dependencies. This innovation became a core technology for sequence modeling over the following two decades, widely applied in speech recognition, machine translation, and time series prediction. LSTM was one of the most important neural network innovations before the deep learning era, only gradually replaced by Transformers.
Study Notes
Background, mechanics, impact, and key takeaways in one read
Background & Context
Recurrent neural networks (RNNs) can in principle remember past inputs through feedback connections, but when trained by backpropagation through time the error signal decays or explodes exponentially with the number of steps, preventing learning of long-range dependencies. Sepp Hochreiter proved this problem theoretically in his 1991 thesis. In 1997 Hochreiter, then at the Technical University of Munich, and Jürgen Schmidhuber at IDSIA published 'Long Short-Term Memory' in Neural Computation, solving vanishing gradients with gated memory cells and a constant error carousel, so the network could remember information from more than a thousand steps earlier.
Key Points
- 1LSTM was designed to overcome the vanishing and exploding gradients that block long-range learning in RNNs.
- 2Its core is a memory cell with a constant error carousel, letting gradients flow unchanged over long spans.
- 3The original design had input and output gates controlling write and read; the forget gate came later in 2000.
- 4It succeeded for the first time on artificial tasks with gaps exceeding a thousand time steps.
- 5It became the dominant architecture for speech recognition and machine translation for two decades.
How It Works
In an ordinary RNN, backpropagating the error through time requires multiplying the weight Jacobian repeatedly, so the gradient either vanishes toward zero or explodes. LSTM’s fix is a constant error carousel: the memory cell’s state is updated mostly by addition, c_t = f·c_{t-1} + ..., and addition does not shrink the gradient exponentially, so error flows almost unchanged across many steps. Gating units, sigmoid switches outputting values between 0 and 1, control this carousel. The input gate decides what new information may enter the cell state, and the output gate decides how much of that state reaches the hidden layer; together with the forget gate added in 2000, the network learns when to remember, forget, and read out. The memory cell is like a leak-free pipe with valves, letting the error travel a long distance safely.
Impact & Legacy
LSTM let recurrent networks learn long-range dependencies stably for the first time, becoming the workhorse of speech recognition, machine translation, handwriting recognition, and language modeling through the 2000s and 2010s, and was deployed at scale by Google, Apple, and others. It has limits: serial computation resists parallelism, the gating structure is intricate, and very long sequences remain hard. After 2017 the Transformer displaced LSTM on many tasks with self-attention, yet the idea of gated memory endures.
Key Takeaways
- Vanishing and exploding gradients are the core obstacle in recurrent training; LSTM defeats them with additive state.
- Gating is essentially a differentiable memory switch that learns when to remember and forget.
- Architectural breakthroughs often follow a precise mathematical analysis of why things fail.
- Even a strong sequence model can be overtaken by a simpler parallel paradigm such as attention.
Glossary
The tendency of gradients to shrink to zero or blow up exponentially during backprop, making deep or recurrent nets hard to train.
The LSTM component that holds long-term state via a constant error carousel, letting gradients flow unchanged.
A sigmoid switch with output between 0 and 1 that controls information flow, such as input, forget, and output gates.
Related Links
Related Events
Transformer: The Attention Revolution
The paper "Attention Is All You Need" is published, introducing the Transformer architecture and revolutionizing NLP.
Backpropagation Algorithm Revival
Rumelbart, Hinton, and Williams popularize the backpropagation algorithm, reviving neural network research.