> For the complete documentation index, see [llms.txt](https://avikpal.gitbook.io/social-driving/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://avikpal.gitbook.io/social-driving/agents/algorithms/ppo-one-step.md).

# PPO One Step

## Environment Requirements

* `env`: Must be registered in  `sdriving.environments.REGISTRY`. The `step` function must return the Reward Tensor of size `N x 1`. By design it will assume the horizon is of size 1. This model will most likely **never** converge for any other horizon size.
* `env_params`: These are passed to `env` as `env(**env_params)`. So, ensure compatibility with the corresponding environment.

## Logging / Checkpointing

* `log_dir`: Path to the directory for storing the logs, and checkpoints
* `wandb_id`: Id with which to log in wandb. If the id is same as one before, it will append the logs to it
* `load_path`: Checkpoint from which to load a previously trained model
* `save_freq`: The frequency with which to checkpoint models

## Configurable HyperParameters

* `actor_kwargs`: Arguments passed to `PPOWaypointCategoricalActor/PPOWaypointGaussianActor`. `observation_space` and `action_space` are automatically passed from the `env`, so no need to pass those
* `seed`: Random Seed
* `steps_per_epoch`: Number of observation and action pairs to be collected before training the model. This is split equally across all the processes
* `epochs`: Total epochs
* `gamma`: Discount Factor
* `clip_ratio`: Clip Ratio in the PPO Algorithm
* `pi_lr`: Learning Rate for the Actor
* `train_pi_iters`: Total number of times the same set of data is passed for training
* `entropy_coeff`: Coefficient for entropy regularization
* `lam`: Lambda for GAE-Lambda. (Always between 0 and 1, close to 1.)
* `target_kl`: Roughly what KL divergence we think is appropriate between new and old policies after an update. This will get used for early stopping. (Usually small, 0.01 or 0.05.)