Rohan Gangakhedkar.
← All projects

Inverting a pendulum with model-free reinforcement learning

Reinforcement Learning & Optimal Control for Robotics (ROB-GY 6323) · Fall 2022

Rohan Gangakhedkar  ·  New York University

Code All projects
TL;DR

Learning a swing-up policy for an inverted pendulum with no model of the system. Torque is limited, so the pendulum has to rock back and forth to build enough energy to invert.

The learned policy inverting the pendulum.

The Q-learning algorithm took 6000 episodes to learn how to invert the pendulum. In the video above there is one back-and-forth swing before the pendulum lifts up and stays inverted.

Approach

The pendulum model.
The pendulum model.

The inverted pendulum has a limit on the maximum torque it can apply, so it must perform a few back-and-forth motions to reach the inverted position (\(\theta=\pi\)) from standing still (\(\theta=0\)).

The state is written as \(x = \begin{pmatrix} \theta \\ \omega \end{pmatrix}\). The dynamics are time-discretised, with \(x_n\) the state at time \(t = n \Delta t\) for a discretisation step \(\Delta t\). The following discounted cost is minimised, \(\sum_{i=0}^{\infty} \alpha^i g(x_i, u_i)\), where:

\begin{equation} g(x_i, u_i) = (\theta-\pi)^2 + 0.01 \cdot \dot{\theta}_i^2 + 0.0001 \cdot u_i^2 \qquad \textrm{and} \qquad \alpha=0.99 \end{equation}

This cost mostly penalises deviation from the inverted position, but also encourages small velocities and small control effort.

Control of \(u \in \{-4,0,4\}\)

Learned policy for \(u \in \{-4,0,4\}\).

One back-and-forth swing precedes the lift. The graph below shows cost per episode in blue and average cost in orange, falling as learning proceeds. This control scheme took 6000 episodes to converge.

Learning progress — cost per episode and running mean.
Learning progress — cost per episode and running mean.
Value function and policy. The policy is the optimal action at each state; the value function is the associated value when acting optimally.

Control of \(u \in \{-5,0,5\}\)

Learned policy for \(u \in \{-5,0,5\}\).

The extra control effort allows quicker learning in the Q-table, visible as a lower final cost of around 200. The policy inverts the pendulum faster than the \(u \in \{-4,0,4\}\) case — a single swing, with no back-and-forth needed. It again took 6000 episodes, learning slightly more in that time.

Learning progress for the higher-torque control.
Learning progress for the higher-torque control.
Value function and policy for the higher-torque control.

Acknowledgements

I would like to specially thank Professor Ludovic Righetti for allowing me to showcase this project on my website.