AIEA Lab

The goal of this comparative analysis was to see what were the outputs of training an autonomous agent on different reinforcement learning algorithms. This agent was required to navigate a car racing simulation using only visual data. I used the CarRacing-v2 environment, which was challenging to use as a beginner. After researching and reading documentation, I figured out that this environment rewards the agent for visiting new track tiles and penalizes it for straying off-road or doing nothing. The challenge about this environment is that the agent receives no underlying variables about its speed, coordinates, or heading. It must operate only on the raw pixels rendered on the screen.

The main problem is training this agent under the technical specifications of the environment. It has to figure out where it is and develop “spatial awareness” from a grid of pixels. It has to independently learn that a cluster of gray pixels means road, green pixels mean grass, and the red blob in the center is its own vehicle. It then has to learn how to map those visual patterns into physical controls like steering, turning and braking.

I started by exploring pre-existing algorithms from the stable-baselines library in Task 5. I decided to choose the Policy Proximity Optimization algorithm to run. Here was my result from the resulting job that I had deployed. I realized that training for 50,000 timesteps is a short run for a pixel-based environment like CarRacing-v2. It probably explains why my TensorBoard graphs showed that the exploration phase had generated negative rewards.

I was starting to become familiar with basic functions like how to create an environment with different parameters like episodes and steps. I realized what the metrics meant for loss, reward, and exploration. I still was curious about what these algorithms meant, and also what they could actually do in the real world. That is when I started reading about the 2013 Deepmind Research Paper about the Deep Q-Network algorithm for task 6.

What is Deep Q-Network (DQN)? That was the question that I had asked when task 6 instructed me to implement a Reinforcement Learning Algorithm from scratch. I learned from the paper that their DQN algorithm used a convolutional neural network whose input was raw pixels. I had initially tried forcing CarRacing through a linear network without using a CNN. This actually resulted in me having to flatten the image into a 1D line of pixels. This caused the agent to lose the “spatial awareness”, and cause no reward curve, so the agent would never learn how to drive. So initially my task 6 had failed. I then watched some youtube videos (mainly 3blue1brown) about what I should do when implementing a convolutional neural network. I also took some help from AI tools to understand what I was doing. I also realized that trying to process full sized RGB images was going to be time consuming and compute inefficient so I implemented a preprocessing step that converts images to grayscale and crops them to 84 by 84 pixels. I also learned about gradients and how they can show issues about the loss function and how to take care of it. In my DQN algorithm, I implemented the Mean Squared Error (MSE) loss function and Root Mean Square Propagation as my optimizer (RMS).

Results of Task 6: I ran this custom DQN for around 100 training episodes, and then evaluated the model’s performance afterwards. The episode reward curve started around -50, so the car was driving off the track. I noticed that by the 100th episode, the rewards were around 200 and the training loss stayed near zero. There were a few spikes up to 2.6, but the PyTorch gradients were relatively stable. While the agent learned basic driving skills, the learning curve was noisy. My exploration rate (epsilon) decayed linearly from 1.0 to 0.1. Because the agent was still taking completely random actions 10% of the time, it kept throwing itself off the track even as its policy improved. During testing, my DQN agent surprisingly managed to get around 400 - 500 score, which was not at all bad for a 100 episode training session. Overall this algorithm implementation was a success because it meant that the agent could learn driving, but it still did not manage to finish the car racing track.

Refinements of Task 8: Task 8 asked me to figure out what the drawbacks were to the DQN algorithm that I implemented. After looking at my reward graphs from Task 6, I started researching why standard Q-learning struggles to maintain a stable policy. The Deepmind paper had mentioned DQN performs poorly on games with complex actions, and also is prone to overestimation bias. I learned that overestimation bias could be present in my DQN algorithm because the same neural network that was used to select an action was also used to evaluate how many points that action was worth. I found out that if my agent coincidentally found a lucky state, it would artificially inflate the value of that state, which eventually corrupts the entire policy.

To fix this, I rewrote the algorithm into a Double Deep Q-Network (DDQN), which the Deepmind research paper also mentions. This splits the math: I kept my primary network to choose the actions, but I added a secondary, frozen "target" network to evaluate the actual point value of those actions. I also had to address the gradient spikes I saw in my earlier loss graphs. I read through PyTorch documentation and realized that using MSE was causing problems. When the agent made a substantial error, squaring that loss value created a huge mathematical spike that ruined the learned weights. I used a more refined loss function called Huber Loss instead, which combines the best of MSE and Mean Absolute Error (MAE) by being quadratic for small errors and linear for large errors. I had also upgraded my optimizer from RMSprop to Adam, which combines RMSprop’s optimization algorithm with bias correction. I also implemented a warmup phase, which was basically just more training episodes to fill the agents replay buffer with diverse driving data before the neural network was allowed to start learning.

Results of Task 8: These functional changes made a big difference in my data. I ran the DDQN model for 1,000 episodes. The difference from the DQN algorithm was evident. The moving average of the rewards climbed smoothly, and it avoided the erratic collapses DQN had struggled with earlier. By the end of the run, the moving average was around 800 points, where some individual episodes also broke past the 900-point threshold. This means the agent was very close to solving the CarRacing-v2 environment. The training loss scaled upward as the Q-values grew, but the Huber loss kept it bound and prevented the network from crashing.

While the DQN implementation worked, the authors of the Deepmind research paper warned that it relied on a memory buffer which used a massive amount of physical RAM. I was curious if there was a way to train a network without a memory buffer. I started reading the 2016 DeepMind paper on Asynchronous Advantage Actor-Critic (A3C) methods. I learned that you could remove the memory buffer by running multiple independent agents in parallel across different CPU threads.

I decided to try writing an asynchronous Q-learning agent. I mainly used the same environment setup and image preprocessing, CNN, and frame skipping wrapper from Task 8, so the only main thing I had to implement was the actual agent logic itself. After watching some example implementations of this online. I used Python's multiprocessing library, which set up a shared global neural network and spawned eight independent worker threads. Each worker ran its own copy of the racing environment, calculated its gradients locally, and updated the global network asynchronously. The theory was that because eight different cars were driving on eight different parts of the track at the same time, the incoming data would naturally be randomized without needing a storage buffer.

Results of Task 7: I let this algorithm run on my Nautilus desktop and let it train for around 48 hours at 6,000 global episodes. Once the training was complete I noticed right away from the graphs that something was off. The moving average reward never went into the positive, it flattened out at -30, and the training loss spiked erratically. However, when I looked into my code; I realized that I made a logic bug by setting my exploration rate (epsilon) to slowly decay from 1.0 to 0.05 over 5,000 episodes. But because I divided the 6,000 global episodes among eight parallel workers, each individual thread only played about 750 episodes. Because of this the epsilon value only dropped to 0.86. For the entire training session my workers were choosing random controls 86% of the time, throwing their cars off the track.

Even with this error, the algorithm actually succeeded. When I turned off the randomness and ran an evaluation phase, the agent scored over 240 points, which meant that the agent could at least move and use controls, and slightly follow the track.

Implementing the one click solution: To actually understand which approach was better, I needed to compare them fairly. In Task 9, I built a benchmarking script to run my DDQN agent and my asynchronous agent together in the same environment. I set both algorithms to a 250-episode limit. I wanted to see who could learn the fastest with the least amount of data, my hypothesis said DQN would probably learn faster. The script was easy to implement since I had implemented all the code earlier, I just had to combine it all. Both algorithms shared the same image preprocessing, frame wrapping, and action space. I configured the script to track the raw rewards, calculate rolling averages, and output comparative graphs using matplotlib as well as put a final log into tensorboard.

Results of Task 9: The benchmarking plots proved why Experience Replay buffers are better for this kind of problem, I also wondered if anyone else had the same kind of issue when running an A2C or A3C algorithm on CarRacing, and it turns out that Asynchronous Algorithm just performs poorly in this environment (RadoSlaw). The DDQN's moving average was up to 600 within 150 episodes because it reused its past data to quickly learn the track. In contrast, the asynchronous agent's score stayed completely flat near -50. Parallel agents throw their data away the second they calculate a gradient, 250 episodes was basically zero data for the asynchronous network.

Conclusion: The results of my comparison between the memory-based DDQN against the asynchronous agent proved my hypothesis right. I found out that A2C algorithms are better for more complex games with large action/state spaces and its ability to handle continuous actions. DQN outperformed asynchronous algorithms in this scenario for more simpler discrete, pixel-based games.