Recurrent Neural Network (RNN) & Long Short-Term Memory (LSTM)

Introduction

Deep Learning/Machine Learning technologies have gained traction over the last few years, with significant impacts being seen in real-world applications like image/speech recognition, Natural Language Processing (NLP), classification, extraction, and prediction. These are being made possible by artificial neural networks. Among the advanced artificial networks, Recurrent Neural Networks (RNNs) offer a tremendous amount of versatility as users can operate over sequences. If you want to make sense of patterns in your data that changes with time, your best bet is a Recurrent Neural Network. RNNs remember the inputs and the context as they have internal memory, enabling users to have more flexibility in the types of data that models or networks can process. RNNs are a powerful tool when data is sequential, and the next data point depends on the previous data point. This can include sequences at the input, sequences at the output, or both simultaneously. Since they understand the context in a sequence, RNNs can produce better results/predictions. Thus, RNNs provide avenues to experiment with various types of input and output data.

RNNs are versatile in their applications

Feedforward neural networks depend on a fixed size data; they do not remember the input received previously, and are thus not very good at tasks involving sequences and time series data. RNNs, on the other hand, are designed to capture information from sequences or time series data as they account for memory and remember the input received previously. The ability to work with sequences is what makes RNN applications so versatile as they accept variable sized inputs/values and can also produce variable size outputs/values. They have a built-in feedback loop, which enables them to remember current and past inputs/information while arriving at a decision. This feature is what makes them good at forecasting too.

The uses of RNNs are diverse, including speech or voice recognition, image captioning/description, stock market prediction, image classification, video classification frame by frame, machine/language translation, sentiment analysis, sequence generation, and time series. For example, we can do “one to many” models in the case of image captioning, where the input (image) is of a fixed size and the output (a sentence or a string of words describing the image) is of variable size. It could be a “many to one” model in the case of sentiment analysis, where the input is a sentence or many words, and the output – which could be a positive or negative sentiment that the sentence is trying to evoke – is of a fixed length. The model could also be “many to many”, for example, in the case of language translation.

LSTM solves unstable gradients in RNNs

While training artificial neural networks with gradient-based learning methods, a common problem that one encounters is an unstable or vanishing gradient, which impairs the ability of the network to learn/perform. Hence, though RNNs allow a lot of flexibility, they are prone to vanishing gradients as they can remember information for a short duration of time only. This can be addressed with a better architecture like a Long Short-Term Memory (LSTM). The latter, which is an RNN variant, is a more complicated architecture that resolves issues of vanishing gradients.

Since the output is based on previous calculations/computations, it is vital to remember previous inputs. A standard RNN can look back a few time steps only, which is not enough to produce the most accurate results when more parameters are added. An LSTM, on the other hand, can look back many time steps and are considered best suited for applications like sequence predictions, language translation, speech to text and for any information rooted in time like video. An LSTM can forget and remember information/patterns selectively for a long duration of time. It understands which data/information is important and should be retained in the network, and this is what gives LSTM an edge over simple RNNs and feedforward neural networks.