car-racing-ppo
PPO RL agent for CarRacing-v3
2 min readupdated
Built on the PPO algorithm, this reinforcement learning (RL) agent drives a car in the Gymnasium CarRacing-v3 simulator directly from raw screen pixels. Created by TheOnlyGoatsan using Stable-Baselines3 (CnnPolicy architecture), this release is the fourth iteration in a series of reward-function tuning experiments. Instead of relying solely on default game points, it adds penalties for driving onto the grass, drifting off the track centerline, and jerky steering, alongside a speed bonus for staying on the asphalt. It is ideal for developers and students exploring reinforcement learning and reward shaping in classic vision-based environments.
Key features
- Higher scores: Across 100 fixed tracks, the agent averages 742 ± 31 points compared to 540 ± 34 for a baseline model trained on standard game rewards, outperforming it on 87 out of 100 tracks.
- Pixel-based control: Takes a stack of the last 2 grayscale 64×64 frames as input and makes a steering decision every 2 frames.
- Thoughtful reward shaping: Rewards staying near the track centerline and maintaining speed on asphalt, while penalizing grass off-tracks (-0.1) and abrupt steering changes.
- Ready-to-use artifacts: The repository includes the model weights (
final.zip), normalization stats (final_vecnormalize.pkl), and a per-track benchmark table (bench.csv).
What you can do
- Run the pre-trained agent in Gymnasium
CarRacing-v3to evaluate its driving behavior across different tracks. - Use
bench.csvand the custom reward formula as a baseline for your own experiments aimed at breaking the 900-point mark.
Example
To run the agent, wrap the environment with standard frame preprocessing (downscaling to 64×64, grayscale conversion, skipping 2 frames, and stacking 2 frames) and load final.zip into Stable-Baselines3. Evaluated deterministically across 100 test seeds (1000–1099), the agent hits a median score of 783 and scores 900 or higher on 15% of the tracks.
Read more
DetailsBenchmark (100 tracks): mean 742 ± 31, median 783, 15% of tracks ≥ 900 (the threshold for solving the environment is a 900 average).
Limitation: The agent lacks top speed — on 85% of tracks, it hits the 1,000-frame episode limit before completing a full lap.
Training: 4M decision steps, PPO (SB3 2.9.2) with gSDE, batch size 128, 10 epochs, linear learning rate decay from 1e-4 to 0, and 256-unit actor/critic heads (GELU).










Comments