Reinforcement Learning

Introduction

From Deepmind’s success with AlphaGo to breakthroughs in robotic arm manipulation and computers automatically playing Atari video games – the field of Reinforcement Learning (RL) has gained significant momentum over the last few years. It is a branch of Machine Learning (ML), involving an agent and an environment. This Machine Learning algorithm enables agents to establish, by themselves, what is the ideal behavior/action in a specific context to maximize its performance/goals accordingly. It is used by various software and machines to ascertain the best possible path it should take in a given situation.

What’s Supervised Learning & the Need for Reinforcement Learning

Supervised learning in ML implies that the user gives an input to the neural network (computing systems inspired by the human brain) model and also knows the output that the model should produce. In this case, users compute gradients to train the network to provide the desired output. For example, in supervised learning, if we want to train a neural network to play a game of chess, we have to create a dataset to train on, which is not always an easy task. Again, if we teach the neural network to imitate the actions of the human chess player, the agent will never be better at playing the game than the human gamer. To resolve these issues and ensure that the agent can play the game on its own, one can deploy Reinforcement Learning, which has an agent that can take action in its environment and it is rewarded for its action. In RL, users do not need to specify the rules to cover all the possibilities to determine the best moves and win the game. Thus, the agent learns by actually playing the game in RL.

How does RL work?

• Just like humans learn through interactions, the goal of the agent in RL is to learn how to take appropriate actions by interacting with its environment so that it can maximize a numerical reward signal.
• A lot of Reinforcement Learning takes place as a conversation between the agent and the environment, in which the latter reveals itself to the agent in the form of a state.
• The agent, in turn, gets to influence the environment by taking action, which is a set of possible moves (moving up, down, left or right, for example). The environment will give back a reward as well as the next state. In RL, policies determine what action needs to be taken.
• Since the agent is penalized if it makes an incorrect move, it learns through trial and error, by using feedback from its own experiences, till it can maximize its rewards and minimize the penalties. This will keep going on in this loop until the environment gives back a terminal state, which then ends the episode.
• For example, in the ancient Chinese game Go, the agent’s objective is to win the game. The state will be the position of all the pieces on the board, whereas the action will be where to place the next piece during the game. The reward will be ‘one’ if the agent wins at the end of the game and ‘zero’ if the agent does not. Again, in the field of robotics, the aim is to move from point A to the goal point, and the software agent will have to try all possible permutations and combinations until it finds the best path (the one with least hurdles) to reach the goal point.
• Each time, until the agent can achieve the desired result, the output will depend on the state of the current input and the next input will depend on the output of the previous input.

Applications & Future of RL

Diverse Applications: Reinforcement Learning has been tested and applied across industries, which includes gaming, robotics, industrial automation, vehicle navigation and traffic, industrial logistics, finance, and healthcare, among others.

Industrial Robot: An industrial robot made by a Japanese company, Fanuc, for example, uses deep reinforcement learning to train itself and learn new tasks. “It tries picking up objects while capturing video footage of the process. Each time it succeeds or fails, it remembers how the object looked, knowledge that is used to refine a deep learning model, or a large neural network, that controls its action,” says an article in MIT Technology Review. According to Fanuc’s official website, its applications include preventive maintenance support and minimizing downtime, among others.

Research: Google’s AI blog talks of a Tensorflow-based framework that “aims to provide flexibility, stability, and reproducibility” for RL researchers. “Inspired by one of the main components in reward-motivated behavior in the brain and reflecting the strong historical connection between neuroscience and reinforcement learning research, this platform aims to enable the kind of speculative research that can drive radical discoveries,” says the August 2018 Google AI blog.

Gaming: A paper published by Deepmind in the journal ‘Nature’, talks about AlphaGo Zero, the latest evolution of the computer program AlphaGo, the first such program that defeated a world champion in Go, thanks to RL.

Traffic: To minimize average delay, congestion, and intersection cross-blocking, researchers from The University of Tennessee, Knoxville, tested an RL-based multi-agent system. According to the results of the study, performance improvement was seen concerning both average vehicle delay as well as “cross-blocking likelihood, particularly in the context of high traffic scenarios”. Their findings, published in “IET Intelligent Transport Systems” describes the multi-agent system and RL-based framework for “scheduling traffic signals at intersection networks” for achieving “efficient traffic signal control policy, aimed at minimizing the average delay, congestion, and likelihood of intersection cross-blocking”.

Immense Future Potential: The field of RL has seen significant advances over the years. It is touted to have tremendous potential in the field of AI and ML. Since RL learns from its behavior and experiences, it has the potential to make this Machine Learning algorithm a go-to technology in areas where the there are no datasets for training the network, and the only way to collect information about the environment is to interact with it. The field of RL is evolving and is expected to witness developments which will impact inventory, delivery management, manufacturing, resource management, besides assisting in e-commerce personalization, improving chatbots that can learn via user interactions to make, simplifying factory jobs, developing intelligent automated robots, creating advanced self-driving vehicles, stock trading, and optimizing financial objectives.