Getting a simulation to behave exactly like real-world hardware is notoriously tough. This project demonstrates the successful zero-shot sim-to-real transfer of a deep reinforcement learning policy to control a highly non-linear, 2-DoF underactuated Furuta pendulum. Executing swing-up maneuvers and maintaining continuous balance for over one hour on physical hardware without a single line of hardware-specific tuning.

GitHub Repository

2-DoF Underactuated System
100 Hz Embedded Control Loop
>1 Hour Continuous Balance
Zero-Shot No Hardware Tuning

Mechanical Design

The physical system was designed entirely from scratch in Fusion 360 and manufactured via FDM 3D printing. The Furuta pendulum configuration consists of a motorized horizontal arm (rotor) and a free-swinging vertical pendulum link. All mechanical components were designed to be lightweight while maintaining the rigid joint geometry required for accurate dynamic modeling.

Exploded CAD assembly view of the Furuta pendulum

Exploded assembly view — the full pendulum mechanism with motor mount, rotor arm, and pendulum link.

Simulation Development & Calibration

A high-fidelity MuJoCo physics simulation was the foundation of the entire project. Getting the sim-to-real transfer to work required the simulation to be meticulously calibrated to match the physical system.

Two isolated calibration experiments were designed to independently characterize and tune the key physical parameters before any RL training began.

Pendulum Drop Test

The pendulum link was released from a fixed angle and the free-swing decay was recorded. The simulation’s inertia and damping parameters were iteratively adjusted until the simulated drop response matched the physical hardware response.

Before Calibration Initial pendulum drop test — simulation vs hardware mismatch
After Calibration Tuned pendulum drop test — simulation matches hardware

Rotor Friction Test

The rotor arm was driven at a fixed command and released, isolating the motor’s friction and back-EMF characteristics. This independently constrained the motor torque model in the simulation.

Before Calibration Initial rotor isolation test — friction model mismatch
After Calibration Tuned rotor isolation test — friction model matched

Reinforcement Learning

Training Methodology

A Proximal Policy Optimization (PPO) agent was trained in a custom Gymnasium environment wrapping the calibrated MuJoCo simulation. Several techniques were combined to maximize real-world robustness:

Curriculum Learning — A multi-phase reward structure progressively taught the agent distinct sub-behaviors in sequence: stabilizing near the top, catching the pendulum from near-upright positions, and finally executing full swing-up maneuvers from rest. Training was gated so the agent could not advance to the next phase until the current one was reliably solved.

Domain Randomization — On every episode reset, physical parameters were sampled randomly within calibrated bounds: pendulum mass, motor torque limits, and joint damping coefficients. This forced the policy to generalize across the range of hardware variation rather than overfit to a single nominal model.

Noise & Latency Injection — Manual velocity noise was added to all observations, and action latency buffers were used to simulate communication delays between the inference engine and the motor controller, directly mimicking the timing characteristics of the embedded system.

TensorBoard training curves — PPO policy reward over training steps

TensorBoard training curves showing episode reward progression across curriculum phases. The distinct reward steps correspond to each phase unlock.

Embedded Deployment

Exporting the Policy

After training, the final PyTorch model weights were exported directly as a flat array of floating-point values representing the neural network in a C++ header file.

Dual-Core Architecture on ESP32

The ESP32’s dual-core architecture was leveraged to strictly isolate compute-intensive inference from real-time motor control:

Core 0
RL Inference & State Estimation
  • Reads encoder positions via SPI at 100 Hz
  • Runs neural network forward pass
  • Calculates target torque command
Core 1
Real-Time Motor Control
  • Executes Field Oriented Control (FOC)
  • Drives BLDC gimbal motor via SimpleFOC
  • Handles current sensing & torque execution

Torque Field Oriented Control was used for actuation, providing precise, linear torque commands rather than velocity or position targets. This gave the RL policy direct, low-latency authority over the motor output, which was essential for the fast dynamics of the swing-up maneuver.

Hardware Debugging & Observation Logging

The first deployment onto physical hardware did not work. The pendulum would immediately diverge rather than swing up and balance. Rather than retraining or adjusting reward functions, observation logging was added to both the simulation and the physical hardware to directly compare the two state streams side by side.

Simulation Simulated observation space logging
Physical Hardware Physical hardware observation space logging

The logs immediately revealed the issue: the SPI encoder readings from the physical AS5047P were sign-inverted relative to the simulation’s coordinate convention. The motor was receiving the correct torque magnitude but driving the pendulum in the wrong direction on every step. A single sign flip in the firmware resolved the issue entirely, no retraining, no reward shaping, no parameter adjustment. The policy worked reliably from that point forward.

Results

The trained policy achieved continuous balance for over one hour before powering off the system. I have yet to find the upper bound for the inverted pendulum balance time. The zero-shot transfer was validated not just by the duration of balance, but by the quality of the swing-up: the policy consistently executed the correct energy-pumping maneuver learned entirely in simulation.