ADAP

A Space of Specialized Agents

---

The Niche Specialization Experiment (Section 4)

Recall from the paper that in this experiment, we enforce that agents can attack only one of chickens or towers. Agents can initially choose either one, but once they make a choice, they are locked in. Lastly, agents do not see which object they are locked into. This means that a "single" policy paradigm would result in either a "jack-of-all-trades" agent, or an agent that only attacks one particular unit type. But by instead learning a policy space, we can find individually specialized policies that, in aggregate, harness all resources.

Visualization

Agents that locked into towers are boxed in green, and agents that are locked into chickens are boxed in red. If a chicken-specialized agent attacks a tower, or vice versa, we flash the agent in purple, and increment the counter of blunders. A high number of blunders indicates that the general agent population does not specialize very well.

We are looking at this for a qualitative demonstration of behavior, and are not concerned too much with the numbers. However, it is clear to see that ADAP keeps the number of blunders relatively low. Additionally, in ADAP, there is a small subset of agents that blunder, indicating it has learned a multi-modal policy space with specialized agents. This means that we can optimize out these erroneous agents. In Vanilla and DIAYN*, all agents tend to blunder with equal chance, and cannot be optimized out.

Adapting to Ablations

---

Now that we have visualized how ADAP learns a generator with a specialized and multi-modal policy space, we can look at how to use this policy space to adapt to different environmental ablations.

These ADAP policies were not trained to solve these ablations specifically! It turns out that by learning a diverse policy space on the original, open-ended train environment, ADAP learned a policy "species" that was implicitly well-suited for most ablations. For example, ADAP learned species that "prefer" to go to the bottom right, or wait until their health is low before farming food.

Again, here we are primarily concerned with qualitative results - we quantify variance in the paper. We show results after latent space optimization onto each ablated environment.

• Far Corner
• Wall Barrier
• Patience
• Speed
• Poison Chickens

Multi-Goal Evaluation

---

We present a discrete multi-goal environment: 40 agents start out in the center of the map (gold diamond), and must navigate to one of four gold stars at the corners. There is one small twist: each star only has 10 units of food, and agents consume 1 unit of food each.

• Agent Observation: one hot encoding of agent's (x, y) position
• Episode length: 20
• Reward: 0 at timesteps 0-19. 1 at timestep 20 if agent found food, 0 otherwise.

Visualization

We visualize two seeds each of the results of ADAP, DIAYN*, and Vanilla, using both concatenation and multiplicative models. Each agent trajectory is shown as a series of connected line segments, representing the series of steps in its trajectory. The RGB color of the agent trajectory is directly proportional to its latent z.

Summary of Results

• ADAP: Mult. model learned to reach all 4 goals consistently.
• DIAYN*: Both models reached all four goals. Suprisingly, the concat. model was better at finding consistently diverse policy space.
• Vanilla: Mult. model generally failed to reach more than 2 goals, resulting in low overall rewards. Concatenation model fared even more poorly.
• Vanilla (higher entropy): High entropy policies do not imply diverse policies, and this method generally only reached 1 goal.

• ADAP
• DIAYN*
• Vanilla
• Vanilla (Higher Entropy)

CartPole

---

Here, we have qualitative results for CartPole, under normal conditions, and various ablations.

• Normal: the normal CartPole conditions
• Noisy Observations: Random uniform noise added to observations.
• Reward for left and right sides: Ablated reward function, where reward for a timestep is equal to the negative or positive absolute value of the x-position for left and right respectively.

Summary of Results

As in prior experiments, we optimize for the ablations entirely in the latent space distribution. ADAP, Vanilla, and DIAYN* all performed nearly perfectly under Normal and Noisy conditions. On the other hand, only ADAP and DIAYN* were able to produce policies that explored the left and right half of the x-axis.

• Normal
• Noisy Observations
• Reward for Left Side
• Reward for Right Side