# PPO One Step

## Environment Requirements

* `env`: Must be registered in  `sdriving.environments.REGISTRY`. The `step` function must return the Reward Tensor of size `N x 1`. By design it will assume the horizon is of size 1. This model will most likely **never** converge for any other horizon size.
* `env_params`: These are passed to `env` as `env(**env_params)`. So, ensure compatibility with the corresponding environment.

## Logging / Checkpointing

* `log_dir`: Path to the directory for storing the logs, and checkpoints
* `wandb_id`: Id with which to log in wandb. If the id is same as one before, it will append the logs to it
* `load_path`: Checkpoint from which to load a previously trained model
* `save_freq`: The frequency with which to checkpoint models

## Configurable HyperParameters

* `actor_kwargs`: Arguments passed to `PPOWaypointCategoricalActor/PPOWaypointGaussianActor`. `observation_space` and `action_space` are automatically passed from the `env`, so no need to pass those
* `seed`: Random Seed
* `steps_per_epoch`: Number of observation and action pairs to be collected before training the model. This is split equally across all the processes
* `epochs`: Total epochs
* `gamma`: Discount Factor
* `clip_ratio`: Clip Ratio in the PPO Algorithm
* `pi_lr`: Learning Rate for the Actor
* `train_pi_iters`: Total number of times the same set of data is passed for training
* `entropy_coeff`: Coefficient for entropy regularization
* `lam`: Lambda for GAE-Lambda. (Always between 0 and 1, close to 1.)
* `target_kl`: Roughly what KL divergence we think is appropriate between new and old policies after an update. This will get used for early stopping. (Usually small, 0.01 or 0.05.)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://avikpal.gitbook.io/social-driving/agents/algorithms/ppo-one-step.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.