A3C And A2C

Описание: The speaker provides a prototypical implementation of an actor-critic method, with the example of A3C (Asynchronous Advantage Actor Critic) algorithm. In this method, you have a network consisting of an actor and a critic, and multiple workers (processes in your computer) that can each have a copy of the environment you're trying to solve. The number of workers you can have is typically limited by the number of CPUs in your computer.

In A3C, each worker collects samples of experiences, calculates gradients, and then asynchronously updates the network. There are no synchronization points in this method. The speaker recommends reading the A3C paper for more details and justification of this approach.

The speaker also introduces A2C (Advantage Actor Critic), which is similar to A3C but with a synchronization point. In A2C, gradients are collected from all workers, and then the network updates all workers at once. This method is useful when you have access to a GPU. There is no paper on A2C, but OpenAI has a blog post that explains the method and its advantages when using a GPU.

In conclusion, if you don't have access to a GPU, A3C can be a good choice for solving multi-agent problems in reinforcement learning.

Papers mentioned: https://docs.google.com/spreadsheets/...